





























Abstract:Autonomous Graphical User Interface (GUI) agents often struggle with multi-step tasks due to constrained context windows and static policies that fail to adapt to dynamic environments. To address these limitations, this work proposes the Self-Evolving GUI Agent (SE-GA), a novel framework that integrates hierarchical memory structures with an iterative self-improvement mechanism. At the core of our approach is Test-Time Memory Extension (TTME), which facilitates long-term planning by dynamically retrieving episodic, semantic, and experiential memories to provide salient contexts during inference. To ensure continuous learning, we introduce Memory-Augmented Self-Evolution (MASE), which is a training pipeline that adopts the data collected by TTME to stabilize and enhance the agent's foundational policy. Extensive evaluations across both offline and online benchmarks demonstrate SE-GA achieves state-of-the-art performance, reaching success rates of 89.0\% on ScreenSpot and 75.8\% on the challenging AndroidControl-High dataset. Furthermore, significant improvements on the AndroidWorld benchmark highlight the superior generalization to dynamic environments. Open source code: this https URL
| Comments: | Accepted by ICML 2026 |
| Subjects: | Machine Learning (cs.LG) |
| Cite as: | arXiv:2605.16883 [cs.LG] |
| (or arXiv:2605.16883v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16883 arXiv-issued DOI via DataCite (pending registration) |
From: Shilong Jin [view email]
[v1]
Sat, 16 May 2026 08:51:57 UTC (7,432 KB)
此内容由惯性聚合(RSS阅读器)自动聚合整理,仅供阅读参考。 原文来自 — 版权归原作者所有。