TL;DR
Over the past few weeks, we’ve been working tirelessly on making GUI agents more open, accessible and easy to integrate. Along the way, we created the largest benchmarking suite for GUI agents performances 👉 let us introduce ScreenSuite.
We are very excited to share it with you today: ScreenSuite is the most comprehensive and easiest way to evaluate Vision Language Models (VLMs)across many agentic capabilities!
GUI Agents in action - courtesy of OSWorld
In short, an AI Agent is a robot that acts in the virtual world. (more thorough definition here)
In particular, a “GUI Agent” is an agent that lives in a GUI. Think “an agent that can do clicks and navigate on my desktop or my phone”, à la Claude Computer Use.
This means in essence that the AI model powering the agent will be given a task like “Fill the rest of this Excel column”, along with screen captures of the GUI. Using this information, it will then decide to take action on the system : click(x=130, y=540) to open a web browser, type(”Value for XYZ in 2025"), scroll(down=2) to read further… To see a GUI agent in action, you can try our Open Computer Agent, powered by Qwen2.5-VL-72B.
A good GUI agent will be able to navigate a computer just like we would, thus unlocking all computer tasks : scrolling through Google Maps, editing a file, buying an item online. This involves a variety of capabilities that can be hard to evaluate.
The literature, for instance Xu et al. (2025) or Qin et al. (2025), generally splits GUI agent abilities amongst several categories:
So our first contribution is to gather and unify a comprehensive suite of 13 benchmarks spanning the full range of these GUI agent capabilities.
If you look at the last category listed above, evaluating Multi-step agentic capabilities is especially challenging as it requires virtual machines to run the agent’s environment, be it Windows, Android, Ubuntu... To address this, we provide support both for E2B desktop remote sandboxes, and we created from scratch a new option to easily launch Ubuntu or Android virtual machines in Docker!
Implementation details
We’ve carefully designed our benchmark suite with modularity and consistency in mind, ensuring strong alignment across tasks and environments. When required, especially for online benchmarks, we leverage smolagents as framework layer to streamline agent execution and orchestration.
To support reproducibility and ease of use, we’ve built custom Dockerized containers that allow local deployment of full Ubuntu Desktop or Android environments.
Unlike many existing GUI benchmarks that rely on accessibility trees or other metadata alongside visual input, our stack is intentionally vision-only. While this can result in different scores on some established leaderboards, we deem that it creates a more realistic and challenging setup, one that better reflects how humans perceive and interact with graphical interfaces.
– All agentic frameworks (Android World, OSWorld, GAIAWeb, Mind2Web) use smolagents and rely solely on vision, without any accessibility tree or DOM added (in contrast with evaluation settings reported in other sources). – Mind2Web (Multimodal) originally used element-name-based multi-choice selection based on the accessibility tree and screenshots, but was later adapted to click precision within bounding boxes using vision only, which significantly increases task difficulty.
We’ve evaluated leading VLMs on the benchmark
Our scores are in general agreement with the scores reported in various sources! With the caveat that we evaluate on vision only, causing some differences, see implementation details above.
💡 Note that ScreenSuite does not intend to exactly reproduce benchmarks published in the industry: we evaluate models on GUI agentic capabilities based on vision. As a result, on benchmarks like Mind2Web where other benchmarks gave the agent a view of information rich context like DOM or accessibility tree, our evaluation setting is much harder, thus ScreenSuite does not match other sources.
Head to the repository.
git clone --recurse-submodules git@github.com:huggingface/screensuite.gituv sync --extra submodules --python 3.11python run.py python examples/run_benchmarks.py for more fine-grained control, like running evaluations for several models in parallel.The multistep benchmarks requires a bare-metal machine to run and deploy desktop/mobile* environment *emulators (see README.md)
Running consistent and meaningful evaluations easily allows the community to quickly iterate and make progress in this field, as we’ve seen with Eleuther LM evaluation harness, the Open LLM Leaderboard and the Chatbot Arena.
We hope to see much more capable open models in the coming month that can run a wide range of tasks reliably and even run locally!
To support this effort:
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。