Model Context Protocol solved a real problem. Before MCP, every tool integration required custom glue code. After MCP, agents can talk to databases, APIs, and services through a standard interface. The protocol gave us a common language for agent-tool communication, and the ecosystem responded with thousands of MCP servers.
But there’s a category of interaction that MCP doesn’t address: graphical user interfaces. Agents still struggle to operate desktop applications, web interfaces, and mobile apps with the same fluency they show when calling APIs. The problem isn’t intelligence. It’s the lack of a standard way for agents to perceive and manipulate GUIs.
Three approaches have emerged to close this gap. Each makes different engineering tradeoffs, and none has achieved the kind of adoption that MCP has seen. Understanding these tradeoffs matters if you’re building agents that need to interact with the visual world.
The GUI Agent Problem
APIs are structured. They accept typed parameters, return predictable responses, and document their behavior. GUIs are none of these things. A button might be labeled “Submit,” “Save,” or “OK.” It might be positioned differently on different screen sizes. Its availability might depend on the state of three other form fields. The visual representation is an abstraction layer over underlying state, and that abstraction was designed for humans, not machines.
