Computer use
Overdeck agents can read and operate a local desktop app (the accessibility tree, clicks, typing, keys, scrolling, dragging, and screenshots) throughpan computer. It is
off by default and works on Linux, macOS (see macOS), or Windows through WSL2.
What it does
pan computer runs a vendored copy of Orca’s Linux computer-use helper (native/computer-use-linux/runtime.py,
MIT, see its README).
The helper reads the AT-SPI accessibility tree, takes window screenshots, and sends input
through AT-SPI actions, XTest and xdotool. Agents use it to read a window’s state, find
elements by their accessibility tree index, and act on them the way a person would, when no
programmatic path (shell, files, git, HTTP, an existing CLI) can reach the same app.
When Overdeck runs inside WSL2, pan computer drives the Windows desktop instead, with Orca’s
Windows helper; see Windows (WSL2).
Turn it on
From Settings
Open Settings → Computer Use in the dashboard (or run Open Computer Use settings from the command palette) and press Set up and turn on. One click does the whole setup on the machine running Overdeck:-
Installs missing system packages. Overdeck runs your package manager through
pkexec, so your desktop asks for your password once. Overdeck never sees the password and never runsapt-get update. If apt reports “Unable to locate package”, runsudo apt-get update, then press Try again. The package names are fixed on the server:When the desktop has no password-prompt agent, Overdeck opens a terminal window that runs thesudoinstall command instead; enter your password there, then press Try again. Withzypper,apkor no detected package manager, setup shows the command to run yourself and stops before turning anything on. -
Turns on GNOME accessibility (
toolkit-accessibility), without a password. Apps opened before this need a restart to show their contents. -
Turns
computer_use.enabledon in~/.overdeck/config.yaml. - Tests the helper with a handshake.
In config.yaml
Add acomputer_use: block to ~/.overdeck/config.yaml:
.pan.yaml can never set it, so a work
agent committing to its own feature branch can’t switch on desktop control for the operator. With
it off, every pan computer command except capabilities and release refuses with disabled.
Dashboard routes
The Settings page uses five admin-only routes:GET /api/computer-use/status,
POST /api/computer-use/setup, PUT /api/computer-use/settings,
POST /api/computer-use/release and, on macOS only, POST /api/computer-use/open-permission
(opens one System Settings permission pane). No scoped access token can call them, so an agent can’t turn
computer use on for itself.
macOS
Computer use works on macOS 14 (Sonoma) or newer when you run Overdeck fromnpx @overdeck/core
or a global npm install. It drives native apps through Overdeck Computer Use, a small helper
app adapted from Orca’s macOS helper (MIT, see
its README).
The helper ships inside the npm package, so you don’t need Xcode. The Overdeck desktop app does
not bundle the helper yet.
Why macOS asks for two permissions
macOS never lets an app grant itself these permissions, so you switch them on yourself, once:- Accessibility lets the helper read app windows’ accessibility trees and click, type and scroll in them.
- Screen Recording lets the helper take window screenshots. Without it the tree still works, but screenshots fail.
Setup
Open Settings → Computer Use and press Set up and turn on:- Installs the helper to
~/.overdeck/computer-use/macos/Overdeck Computer Use.app. No password is needed. The app must stay at that path, because macOS ties the permissions to it. - Opens the two System Settings pages (Privacy & Security → Accessibility and Screen Recording). Switch on Overdeck Computer Use in each, then press Continue in Overdeck.
- Turns
computer_use.enabledon and tests the helper with a handshake.
pan computer call. It
listens on /tmp/overdeck-cu-<uid>/agent.sock, only accepts callers running as your user with
the token in ~/.overdeck/computer-use/macos/agent.token, and exits after 10 minutes without a
request. On macOS, actions run in the background by default: the target app is not brought
forward, your frontmost window keeps focus and the pointer does not move. An agent that passes
--restore-window brings the app forward and uses your screen, keyboard focus and pointer for
that action; the skill tells agents to do this only when a background action can’t work, and to
tell you first.
Updates and signing
Overdeck builds the helper in CI. Without an Apple Developer ID certificate it is ad-hoc signed, and macOS ties the permissions to that exact build. So after an Overdeck update that changes the helper, macOS asks you to allow the updated helper again: the card shows Needs attention, and Fix setup reinstalls the helper and reopens the two System Settings pages, where you switch Overdeck Computer Use off and on again. A helper signed with a Developer ID certificate keeps its permissions across updates. For maintainers: the release workflow signs with a Developer ID certificate whenMACOS_CERTIFICATE exists, and signs ad-hoc otherwise. It notarizes the helper only when it is
Developer ID signed and the App Store Connect API key secrets exist; an ad-hoc build skips
notarization, because Apple notarizes only Developer ID-signed apps:
A source checkout without the bundled zip builds the helper on first setup with
npm run build:computer-macos, which needs the Xcode Command Line Tools
(xcode-select --install).
Requirements
On macOS, see macOS. On Linux:- A Linux desktop with a logged-in session (X11 or Wayland). A headless host or a linked machine
with no desktop session gets
no_desktop_session. - System packages:
python3-gi,gir1.2-atspi-2.0,xdotool(X11 hotkeys/clicks), optionallyxcliporxsel(clipboard paste;wl-copyon Wayland), andgrimon Wayland (screenshots). - GNOME accessibility on:
gsettings set org.gnome.desktop.interface toolkit-accessibility true. - Under WSL2, none of the above applies: see Windows (WSL2).
Windows (WSL2)
Overdeck on Windows runs inside WSL2. There,pan computer and Settings → Computer Use drive
the Windows desktop, never WSLg windows. They use a vendored copy of Orca’s Windows UI
Automation helper (native/computer-use-windows/runtime.ps1, MIT, see
its README).
The commands, flags and output are the same as on Linux.
Requirements
- WSL2 with Windows interop on (the default). WSL1 is not supported.
- Windows 10 or 11.
- Windows PowerShell 5.1, which ships with Windows. Nothing is installed on Windows, so the Settings button reads Turn on Computer Use and asks for no password.
What runs where
- Before each command, Overdeck copies the helper to
%LOCALAPPDATA%\Overdeck\computer-use\on the Windows side, when the copy is missing or differs. Operation files go to itsops\folder and are deleted after each call. The WSL path of that folder is cached in~/.overdeck/computer-use/windows-helper-dir.txt. - Each command starts one
powershell.exewith-ExecutionPolicy Bypass. The bypass applies to that one process only and needs no administrator rights. Each command takes 1 to 3 s, because the helper compiles its Windows API bindings on every start. - A command that runs longer than 30 s is killed with
helper_timeout. Killing it from WSL may leave the Windowspowershell.exerunning; the helper’s own steps are short, so this is rare.
Known limits
list-windowsreturns only an app’s main window (index 0). Wait for a dialog withpan computer wait --textinstead.- The helper is DPI-aware only while it captures screenshots, so clicks can land in the wrong place on setups that mix display scales.
- An app running as administrator refuses input with a
target_elevated:message. - A WSL terminal started as administrator runs the helper elevated too.
- A Group Policy that enforces signed scripts overrides
-ExecutionPolicy; the helper then fails with a message asking the PC’s administrator to allow local scripts. hotkeyandpress-keyrefuse Alt+F4, Win+L and Ctrl+Shift+Escape unless--allow-system-shortcutis passed.
Commands
Every subcommand takes--json. Every action and get-app-state also take --window-index <n>,
--restore-window and --no-screenshot.
See the
pan-computer-use skill for the full field reference and error-handling guidance.
One agent at a time
A desktop lease, held in a file under Where files go, ensures only one holder drives the desktop’s input at a time. An input command claims the lease for 120 seconds, renewed on every call; a holder without it getsdesktop_busy naming the current holder and when the
lease expires. Read-only commands (capabilities, list-apps, list-windows, wait, and
get-app-state without --restore-window) never need the lease, so other agents can still look.
Run pan computer release when a GUI task is finished, or pan computer release --force to
clear a stuck lease held by someone else.
Where files go
Everything lives under~/.overdeck/computer-use/:
Safety
hotkeyandpress-keyrefuse a chord that would quit, lock, log out or switch virtual terminals (alt+F4,ctrl+alt+F2,super+l,ctrl+q, …) withsystem_shortcut_blocked, unless--allow-system-shortcutis passed. On macOS the refused chords arecmd+q,cmd+shift+q,cmd+ctrl+q,cmd+alt+escapeandcmd+shift+backspace.--text-stdinand--value-stdinread sensitive text from stdin so it never lands in argv or shell history.- The shipped skill tells agents never to push, submit, send, buy, delete, or change account settings through computer use unless the user explicitly asked for that action.
Limits
- Wayland can’t do hotkeys (
xdotoolneeds X11); screenshots there needgrim. - No remote hosts: a linked machine with no logged-in desktop session always refuses.
- On some multi-monitor setups, AT-SPI can report a window’s bounds at
(0,0)instead of its real screen position. When that happens, its screenshot can show the wrong area of the desktop, andclick/scroll/dragcoordinates computed from that screenshot can miss the window entirely. Prefer--element-indexactions over--x/--ycoordinates when a window’s reported bounds look suspicious, and verify a coordinate action’s effect against a freshget-app-staterather than trusting the screenshot alone. - The helper is a verbatim vendored port of Orca’s Linux computer-use bridge; see
native/computer-use-linux/README.mdfor how it’s re-vendored.
Troubleshooting
On macOS, Settings → Computer Use shows every check (macOS version, helper, Accessibility, Screen Recording, handshake);pan doctor points there instead of probing. A
permission_denied error means a permission is off: open Settings → Computer Use and finish
setup.
On Linux, run pan doctor. With the feature off it prints no Computer use row at all. With it on, it
checks the AT-SPI Python bindings, the desktop session, xdotool on X11, a clipboard tool,
grim on Wayland, and GNOME accessibility, ending in a single Ready (<session>, display <d>)
row once the helper’s handshake succeeds.
Under WSL2, pan doctor and the What Overdeck checked list show four checks:
If the CLI works but the dashboard’s helper test fails with “Exec format error” or a vsock error,
the dashboard service may be missing the WSL interop socket. Overdeck sets
WSL_INTEROP from
/run/WSL/ when it’s missing; report the error with your WSL version if it persists.
Still stuck? Ask on the Overdeck Discord or file a GitHub Issue.