恢复策略引发的错误：鲁棒GUI智能体的基准测试与轨迹合成

AI 导读

针对GUI智能体缺乏从自身错误中恢复能力的问题，本研究提出了GUI-RobustEval基准测试和RoTS轨迹合成框架。GUI-RobustEval包含1216个可执行测试用例，系统评估智能体在多种错误模式下的恢复能力。RoTS框架通过基于树的流程合成了80万条高质量数据。在此基础上训练的RoTS-7B和RoTS-32B模型，在GUI-RobustEval及传统基准上均获得显著提升。其中RoTS-32B在OSWorld上取得了47.4%的成功率和33.8%的All-Pass@4分数，表明长时程错误恢复能力的增强对鲁棒性和整体性能均有贡献。

推荐理由

GUI Agent 能不能从自己犯的错里爬起来，才是落地的关键，阿里这篇论文给出了一个不错的基准和训练方案，做 Agent 的可以看看。

正文

View PDF HTML (experimental)

Abstract:While GUI agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduce GUI-RobustEval and propose Robustness-driven Trajectory Synthesis. GUI-RobustEval contains $1,216$ executable test cases that systematically measure error recovery capabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates $800k$ high-quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our dataset, both demonstrate significant gains on GUI-RobustEval and traditional GUI benchmarks. Notably, RoTS-32B achieves state-of-the-art performance on OSWorld, with a $47.4\%$ success rate and a $33.8\%$ All-Pass@4 score, suggesting that improved long-horizon error recovery ability contributes to both robustness and overall performance. Our code is available at this https URL.

Comments:	ICML 2026 Spotlight. 36 pages, 19 figures, includes appendix
Subjects:	Computer Vision and Pattern Recognition (cs.CV); Computation and Language (cs.CL)
Cite as:	arXiv:2605.29447 [cs.CV]
	(or arXiv:2605.29447v1 [cs.CV] for this version)
	https://doi.org/10.48550/arXiv.2605.29447 arXiv-issued DOI via DataCite

Submission history

From: Tianpeng Bu [view email]
[v1] Thu, 28 May 2026 06:40:24 UTC (11,767 KB)