Model Context Protocol solved a real problem. Before MCP, every tool integration required custom glue code. After MCP, agents can talk to databases, APIs, and services through a standard interface. The protocol gave us a common language for agent-tool communication, and the ecosystem responded with thousands of MCP servers.

But there’s a category of interaction that MCP doesn’t address: graphical user interfaces. Agents still struggle to operate desktop applications, web interfaces, and mobile apps with the same fluency they show when calling APIs. The problem isn’t intelligence. It’s the lack of a standard way for agents to perceive and manipulate GUIs.

Three approaches have emerged to close this gap. Each makes different engineering tradeoffs, and none has achieved the kind of adoption that MCP has seen. Understanding these tradeoffs matters if you’re building agents that need to interact with the visual world.

The GUI Agent Problem

APIs are structured. They accept typed parameters, return predictable responses, and document their behavior. GUIs are none of these things. A button might be labeled “Submit,” “Save,” or “OK.” It might be positioned differently on different screen sizes. Its availability might depend on the state of three other form fields. The visual representation is an abstraction layer over underlying state, and that abstraction was designed for humans, not machines.

An agent operating a GUI needs to do three things: perceive the current interface state, decide what action to take, and execute that action. The perception step is where approaches diverge. How does the agent “see” the interface?

Approach 1: API Integration

The most straightforward approach is to bypass the GUI entirely and use APIs when they exist. Many desktop applications expose scripting interfaces. Web applications often have REST APIs or GraphQL endpoints. Mobile apps sometimes support URL schemes or accessibility hooks that can be invoked programmatically.

This approach works well when APIs are available and well-documented. An agent can call create_document(), pass structured parameters, and receive a confirmation response. The interaction is fast, reliable, and doesn’t require visual processing.

The limitation is coverage. Most consumer applications don’t expose comprehensive APIs. Even when APIs exist, they often don’t cover all the functionality available through the GUI. A photo editing app might have an API for basic operations but not for advanced filters. A web application might have a public API for reading data but not for complex workflows that require multi-step form submissions.

API integration also doesn’t help with legacy systems, proprietary software, or applications where the API has been deprecated or never built. The agent becomes dependent on the goodwill of application developers to expose the functionality it needs.

Some frameworks try to extend API coverage by wrapping GUI automation libraries. Python’s pyautogui, Apple’s AppleScript, and Windows UI Automation all provide programmatic access to GUI elements. These tools work, but they’re fragile. They depend on element identifiers, window titles, and UI hierarchies that change between application versions. A script that works today might break after a software update.

Approach 2: Accessibility Tree

Operating systems maintain accessibility trees: structured representations of UI elements designed for screen readers and assistive technologies. These trees contain information about buttons, text fields, menus, and their current states. An agent can query the accessibility tree to understand what elements are present and what actions they support.

This approach is more robust than raw API integration because accessibility trees are standardized at the OS level. macOS uses the Accessibility API, Windows uses UI Automation, and web browsers expose the DOM through accessibility interfaces. An agent that understands these APIs can operate across many applications without custom integration for each one.

Google’s approach with Gemini 3.5 Flash Computer Use leans heavily on accessibility trees. The model can query the accessibility tree of a web page or application, identify interactive elements, and generate actions to manipulate them. This works well for structured interfaces like web forms, file managers, and settings panels.

The accessibility tree approach has limits. Not all applications expose complete accessibility information. Custom-drawn interfaces, games, and applications with non-standard UI frameworks often have sparse or inaccurate accessibility trees. Canvas-based rendering, WebGL content, and video players present particular challenges because their visual content doesn’t map cleanly to accessibility nodes.

Accessibility trees also miss visual information that matters for human-like interaction. The tree might tell you a button exists and is labeled “Save,” but it won’t tell you that the button appears grayed out, that a tooltip is hovering over it, or that a dialog box has appeared in the foreground. These visual cues inform human decision-making, and agents operating purely on accessibility data miss them.

Approach 3: Pure Vision

The third approach treats the screen as an image and uses computer vision to understand it. The agent takes screenshots, processes them through a vision-language model, and generates mouse and keyboard actions based on what it sees. This is the most general approach because it works with any GUI, regardless of whether APIs or accessibility trees exist.
Pure vision agents don’t depend on application-specific integrations. They see the screen the way a human sees it: pixels arranged in patterns that represent buttons, text, images, and layouts. A vision-language model can identify a “Submit” button by its visual appearance, even if the button has no accessibility label or API endpoint.

The tradeoff is computational cost and latency. Processing a screenshot through a vision model takes time. A single inference might require 200-500 milliseconds, and complex interfaces might need multiple inference steps to parse correctly. This makes pure vision agents slower than API-based approaches, where actions execute in milliseconds.

Memory and compute requirements are also higher. Vision-language models need to process high-resolution images, which consume significant VRAM or RAM. Running these models on edge devices—laptops, desktops, mobile phones—requires careful optimization.

Mano-P takes the pure vision approach and optimizes it for edge deployment. The model is a GUI-VLA (Visual-Language-Action) agent designed to run locally on consumer hardware. The 72B parameter version achieves 58.2% success rate on OSWorld benchmarks, ranking first among specialized GUI agents. But the 72B model is primarily for research and benchmarking. The 4B quantized version is what users can actually deploy: it runs with 4.3GB of peak memory usage, achieving 476 tokens per second during prefill and 76 tokens per second during decoding on Apple M4 hardware with 32GB RAM.

The performance characteristics matter for practical deployment. An agent that takes two seconds to process each screenshot creates a sluggish user experience. An agent that processes screenshots in under 200 milliseconds feels responsive. The 4B model’s throughput makes real-time GUI interaction feasible on commodity hardware, which changes where GUI agents can be deployed.

Pure vision also handles edge cases that other approaches struggle with. Custom-drawn interfaces, games, video content, and applications with non-standard UI frameworks all render to pixels. A vision agent can operate them without requiring special integration work.

Engineering Tradeoffs

Choosing between these approaches depends on your deployment context.
API integration is fastest and most reliable when APIs exist. It’s the right choice for agents that operate within well-defined software ecosystems where API coverage is comprehensive. If you’re building an agent that automates workflows in Salesforce, Jira, and Slack, API integration makes sense. The agent will be fast, predictable, and easy to debug.

Accessibility tree approaches offer broader coverage with reasonable performance. They work well for agents that need to operate across many applications but don’t require pixel-perfect visual understanding. Web automation, form filling, and menu navigation are good fits. The approach breaks down when applications have poor accessibility support or when visual context matters for decision-making.

Pure vision is the most general but most expensive. It’s appropriate when agents need to operate arbitrary GUIs, handle visual complexity, or work with applications that lack APIs and accessibility support. The computational cost has historically limited this approach to cloud deployments, but model compression and edge optimization are changing that equation.

Hybrid approaches are also viable. An agent might use API integration for structured operations, fall back to accessibility trees for standard UI elements, and use vision only when the other approaches fail. This layered strategy balances performance and coverage, though it increases implementation complexity.

The Standardization Question

MCP succeeded because it defined a minimal viable interface: tools expose functions with typed parameters, agents call those functions, and results flow back. The protocol is simple enough to implement quickly and flexible enough to cover many use cases.

A GUI agent protocol would need to address different concerns. It would need to standardize how agents perceive interface state (screenshots, accessibility trees, or both), how they specify actions (coordinates, element references, or semantic descriptions), and how they receive feedback (visual confirmation, state changes, or error messages).

No such standard exists yet. Each GUI agent framework defines its own perception and action primitives. Mano-P uses screenshot-based perception with coordinate-based actions. Other frameworks use accessibility tree queries with element ID-based actions. The lack of standardization means that GUI agents are not portable across frameworks, and application developers have no clear target to optimize for.

The Octo workspace takes a different approach to this problem. Rather than standardizing the GUI interaction layer directly, it focuses on agent coordination and task orchestration. Individual agents within Octo can use different interaction strategies—API integration for some tasks, GUI automation for others—while the workspace manages context, state, and collaboration between them. This sidesteps the standardization question by treating GUI interaction as an implementation detail rather than a protocol concern.

Whether a GUI-specific protocol will emerge remains unclear. The diversity of approaches suggests the problem space is not yet well understood. MCP took years to gain traction, and GUI agent interaction may require a similar maturation period before a standard emerges.

Open Problems

Several technical challenges remain unsolved regardless of which approach dominates.

State tracking across actions. GUIs are stateful. Clicking a button might open a dialog, change a menu, or trigger a loading spinner. Agents need to track these state changes and adjust their behavior accordingly. Current approaches handle this through repeated perception cycles—take a screenshot, detect the state change, decide the next action—but this is inefficient and error-prone.

Error recovery. GUIs fail in unpredictable ways. A dialog might appear unexpectedly. A network request might timeout, leaving the interface in an inconsistent state. An element might be obscured by a popup. Agents need robust error detection and recovery strategies, but defining what constitutes an “error” in a GUI context is difficult.

Multi-step workflows. Many GUI tasks require sequences of actions with intermediate verification. Filling out a multi-page form, configuring application settings, or navigating a complex menu hierarchy all require maintaining context across multiple perception-action cycles. Current agents struggle with long-horizon tasks that span dozens of steps.
Cross-platform consistency. An agent that works on macOS might fail on Windows or Linux because the same application renders differently on each platform. Building agents that generalize across operating systems requires handling platform-specific UI conventions, keyboard shortcuts, and interaction patterns.

What Comes Next

The GUI agent space is where API integration was before MCP: fragmented, with each implementation defining its own conventions. The success of MCP suggests that standardization is possible, but the problem spaces are fundamentally different. API integration deals with structured data and typed interfaces. GUI interaction deals with pixels, visual layouts, and human-designed abstractions.

Pure vision approaches like Mano-P demonstrate that general GUI agents are technically feasible on edge hardware. The 4B model’s performance characteristics—sub-200ms inference, 4.3GB memory footprint—show that the computational barriers are falling. The question is whether the community will converge on a standard interface for GUI agent interaction, or whether the diversity of approaches will persist.

For developers building agents today, the pragmatic answer is to use the approach that fits your deployment context. API integration when coverage is sufficient. Accessibility trees for broader reach with acceptable performance. Pure vision when generality matters more than speed. And keep watching for standards to emerge, the way MCP emerged for tool integration.

The code for Mano-P https://github.com/Mininglamp-AI/Mano-Pis available on GitHub under Apache 2.0 license. The 4B quantized model is ready for local deployment on Apple M4 hardware. If you’re working on GUI agent problems, the repository includes benchmark scripts, model weights, and deployment documentation. Worth a star if the problem space interests you.