Agent-Desktop: How AI Agents Operate Desktop Apps Without Screenshots

In short: Agent-desktop is an open-source tool from Vercel Labs that lets an AI agent read and operate desktop applications on macOS, Windows and Linux. Instead of looking at screenshots and guessing where to click, it reads the operating system's accessibility tree, the structured list of buttons, fields and menus that screen readers use, and acts on those elements directly. It is released under the Apache-2.0 licence and installs with npm. For businesses, it is a way to automate software that has no API, with real security questions attached.

Key Takeaways

• The README describes it as computer use for AI agents through the operating system's accessibility tree, as a command-line tool and a C library for macOS, Windows and Linux.
• An agent takes a snapshot, which returns JSON listing every element it can act on, each with a reference. It acts on a reference, then snapshots again to check the result. No pixel matching is involved.
• By default, actions go through the accessibility interface without moving the pointer or taking focus. A flag allows physical input when needed.
• It needs permission on each system: macOS requires an Accessibility grant, and Linux needs the AT-SPI service and a graphical session.
• The licence is Apache-2.0. At the time we looked, the repository showed about 10,400 GitHub stars. We could not confirm a release date or an announcement from Vercel.
• This is an early open-source tool. We have read its documentation but have not run it in production.

How It Works

Most "computer use" agents work from screenshots. A vision model looks at the screen, guesses coordinates and clicks. That is flexible but slow, and a small layout change can break it.

Agent-desktop takes a different route. Operating systems already expose a tree of interface elements for accessibility. The tool reads that tree, gives each actionable element a reference and returns the list as JSON. The loop is:

1. The agent asks for a snapshot of the app.
2. It chooses an element by reference and an action, such as click or type.
3. The tool performs the action through the accessibility interface.
4. The agent takes another snapshot to check what changed.

Because this is text rather than pixels, it is easier for a language model to reason about, and it does not depend on where something sits on the screen. Each command prints a single JSON result, which makes it easy to wrap in an agent framework.

Where It Could Help

• Software with no API. Older finance, practice-management, government or industry-specific desktop apps often cannot be integrated any other way.
• Repetitive back-office work. Copying data between a desktop app and a web system, or running the same steps across many records.
• Testing. Walking through an application the way a user would.

Risks and Limits

• Broad access. An agent that can operate your desktop can, in principle, do what you can do: read messages, open files, submit forms. The accessibility permission is powerful, so grant it narrowly.
• Prompt injection. Text on screen, in an email or in a document can contain instructions aimed at the agent. An agent that reads and acts on everything it sees is exposed to this. Our posts on OWASP's MCP Top 10 and Microsoft's Execution Containers cover related defences.
• Accessibility trees vary. Some applications expose a poor tree, especially custom-drawn interfaces, so coverage will be uneven.
• Early-stage software. Expect rough edges, change and limited support.
• Identity. Run such an agent under its own limited account, not yours. See our post on IBM's agent identity announcement.

Our View

Treat desktop control as a last resort, not a first choice. If a system has an API or an MCP server, use that: it is faster, cheaper and easier to audit. Where nothing else exists, an accessibility-based approach is likely sturdier than screenshot guessing, but run it in an isolated machine or virtual desktop, with a dedicated account and logging, and keep a person approving anything that moves money or sends messages.

The Short Version

Agent-desktop is a sign of where agent tooling is heading: reading structured interfaces rather than looking at pictures. It is open source, simple to try and potentially useful for legacy software. The same power is why it needs containment, narrow permissions and human approval for consequential actions.

If you are weighing automation options for legacy tools, our team can help you choose between integrations, agents and custom software: see custom AI agent development and custom software development.

Related Insights

• AI Coding Agents in 2026: How Developers Actually Work Now
• Google's Gemini Agent: What a Universal Work Agent Actually Changes
• NVIDIA OpenShell and Vera Rubin: What GTC Berlin 2026 Means for AI Agents

Sources

• Vercel Labs, agent-desktop README and repository, read November 2026