claude-workman
Computer use for Linux, as an MCP server. It gives an AI assistant human-mode control of a desktop — screenshot, read the accessibility tree, then move the mouse, click, type, and switch windows. Works with Claude, Claude Code, and any MCP client.
GitHub repository Model Context Protocol MIT license
Install
sudo apt install xdotool ffmpeg gir1.2-atspi-2.0 python3-gi
pip install claude-workman
claude mcp add --scope user workman -- python -m workman.serverWhy accessibility-tree clicking beats pixel clicking
Most desktop automation guesses pixel coordinates and breaks when a layout shifts. claude-workman fuses a screenshot with the AT-SPI2 accessibility tree, so an action can be addressed semantically — “click the push button named Continue” — and land on that button wherever it moved. It is the same principle as macOS AXUIElement-based computer use, ported to Linux.
| Approach | Addressing | Survives layout change |
|---|---|---|
| Pixel-only automation | guessed x, y | No — brittle |
| Accessibility-tree automation | role + name → element | Yes — robust |
Tools
screenshot, screenshot_region | Capture the display or a sub-rectangle as PNG |
accessibility_tree | Actionable elements with role, name and screen coordinates |
click_element | Find an element by role and name, click its center |
enable_accessibility | Enable AT-SPI export and silence the screen reader |
list_windows, focus_window | Enumerate and raise windows |
click, move, drag, scroll | Pointer actions |
type_text, press_key | Keyboard input (xdotool key syntax) |
show_cursor | Visual click cursor overlay for human oversight |
Frequently asked questions
What is claude-workman?
An open-source MCP server that lets an AI assistant control a Linux desktop the way a person does — screenshot, locate, click, type, verify — with accessibility-tree targeting so clicks hit real UI elements instead of guessed coordinates.
How is this different from browser automation?
Playwright, Puppeteer and CDP drive a web page. claude-workman drives the whole desktop: native apps, terminals, file managers, settings dialogs, and browsers alike.
Does it work with Claude Code?
Yes — claude mcp add --scope user workman -- python -m workman.server. It works
with any MCP client, not only Claude.
Does it support Wayland or macOS?
Not yet — X11 today. A Wayland backend and a macOS AXUIElement backend are planned.
Is it safe to let an agent control my desktop?
Treat it like handing over mouse and keyboard. Use show_cursor to see every
action, prefer a dedicated display or VM for unattended runs, and verify by screenshot before
anything irreversible.
Does it solve CAPTCHAs?
No. It is a general desktop-control server, not an anti-bot bypass tool.