agent-desktop
Built on xa11y

A reference desktop
agent tool

A working CLI and MCP server for controlling native desktop apps through the accessibility tree — on macOS, Windows, and Linux. Use it as-is for automation and testing, or read the source to see how easy it is to build your own with xa11y.

Install xa11y — the library behind this →
macOS · Linux · Windows MIT licensed Rust · Python · JavaScript

What it looks like

# list running apps
agent-desktop apps

# read the accessibility tree
agent-desktop tree --app Safari

# find elements with CSS-like selectors
agent-desktop find "button[name='New Tab']" --app Safari

# interact
agent-desktop action press "button[name='New Tab']" --app Safari
agent-desktop action type-text "text_field[name='Address']" --app Safari --value "xa11y.dev"

# screenshot any app or element
agent-desktop screenshot --app Safari --output screen.png
MCP All commands are also exposed as MCP tools — drop agent-desktop into any MCP-compatible agent framework with one config line.

Built on xa11y

agent-desktop is a thin layer over xa11y — a Playwright-style library for driving native apps via OS accessibility APIs. One unified API for AXUIElement (macOS), UI Automation (Windows), and AT-SPI2 (Linux). The same agent-desktop logic fits in ~50 lines in any supported language. 

// cargo add xa11y
use xa11y::{App, InputSim};

async fn run() -> xa11y::Result<()> {
    // connect to any running app by name
    let app = App::by_name("Safari")?;

    // CSS-like selectors, auto-waits for visible + enabled
    let locator = app.locator("button[name='New Tab']");
    locator.press().await?;

    // screenshot any element or region
    let shot = app.screenshot(None).await?;
    shot.save_png("screen.png")?;
    Ok(())
}
# pip install xa11y
from xa11y import App

# connect to any running app by name
app = App.by_name("Safari")

# CSS-like selectors, auto-waits for visible + enabled
locator = app.locator("button[name='New Tab']")
locator.press()

# screenshot any element or region
shot = app.screenshot()
shot.save_png("screen.png")
// npm install @crowecawcaw/xa11y
import { App } from '@crowecawcaw/xa11y';

// connect to any running app by name
const app = await App.byName('Safari');

// CSS-like selectors, auto-waits for visible + enabled
const locator = app.locator("button[name='New Tab']");
await locator.press();

// screenshot any element or region
const shot = await app.screenshot();
await shot.savePng('screen.png');
Quick start Build your own tool with xa11y → Source Read the agent-desktop implementation → Testing Desktop testing guide on xa11y.dev →

Where to go from here

↓

Use agent-desktop

Install the CLI or add the MCP server to your agent stack. Works out of the box on macOS, Windows, and Linux.

Install →
⬡

Build your own with xa11y

Use agent-desktop as a reference and xa11y as your foundation. Rust, Python, or JavaScript bindings available.

xa11y.dev →
✓

Desktop testing

Looking for a cross-platform desktop testing framework? xa11y has a dedicated testing guide with CI setup for all three platforms.

Testing guide →

Install

# via cargo (Rust)
cargo install agent-desktop

# verify
agent-desktop --version

Pre-built binaries for macOS, Linux, and Windows via GitHub Releases. MCP server config: see README.