Skip to main content

Computer use

Overdeck agents can read and operate a local desktop app (the accessibility tree, clicks, typing, keys, scrolling, dragging, and screenshots) through pan computer. It is off by default and works on Linux, macOS (see macOS), or Windows through WSL2.

What it does

pan computer runs a vendored copy of Orca’s Linux computer-use helper (native/computer-use-linux/runtime.py, MIT, see its README). The helper reads the AT-SPI accessibility tree, takes window screenshots, and sends input through AT-SPI actions, XTest and xdotool. Agents use it to read a window’s state, find elements by their accessibility tree index, and act on them the way a person would, when no programmatic path (shell, files, git, HTTP, an existing CLI) can reach the same app. When Overdeck runs inside WSL2, pan computer drives the Windows desktop instead, with Orca’s Windows helper; see Windows (WSL2).

Turn it on

From Settings

Open Settings → Computer Use in the dashboard (or run Open Computer Use settings from the command palette) and press Set up and turn on. One click does the whole setup on the machine running Overdeck:
  1. Installs missing system packages. Overdeck runs your package manager through pkexec, so your desktop asks for your password once. Overdeck never sees the password and never runs apt-get update. If apt reports “Unable to locate package”, run sudo apt-get update, then press Try again. The package names are fixed on the server: When the desktop has no password-prompt agent, Overdeck opens a terminal window that runs the sudo install command instead; enter your password there, then press Try again. With zypper, apk or no detected package manager, setup shows the command to run yourself and stops before turning anything on.
  2. Turns on GNOME accessibility (toolkit-accessibility), without a password. Apps opened before this need a restart to show their contents.
  3. Turns computer_use.enabled on in ~/.overdeck/config.yaml.
  4. Tests the helper with a handshake.
When nothing is missing, the button reads Turn on Computer Use and needs no password. Once it is on, the Allow agents to use the desktop switch turns it off again without uninstalling anything. Take back control releases the desktop from the agent that holds it. The collapsed What Overdeck checked list shows every readiness check, whether or not the feature is on.

In config.yaml

Add a computer_use: block to ~/.overdeck/config.yaml:
This is a global-layer-only setting: a project’s .pan.yaml can never set it, so a work agent committing to its own feature branch can’t switch on desktop control for the operator. With it off, every pan computer command except capabilities and release refuses with disabled.

Dashboard routes

The Settings page uses five admin-only routes: GET /api/computer-use/status, POST /api/computer-use/setup, PUT /api/computer-use/settings, POST /api/computer-use/release and, on macOS only, POST /api/computer-use/open-permission (opens one System Settings permission pane). No scoped access token can call them, so an agent can’t turn computer use on for itself.

macOS

Computer use works on macOS 14 (Sonoma) or newer when you run Overdeck from npx @overdeck/core or a global npm install. It drives native apps through Overdeck Computer Use, a small helper app adapted from Orca’s macOS helper (MIT, see its README). The helper ships inside the npm package, so you don’t need Xcode. The Overdeck desktop app does not bundle the helper yet.

Why macOS asks for two permissions

macOS never lets an app grant itself these permissions, so you switch them on yourself, once:
  • Accessibility lets the helper read app windows’ accessibility trees and click, type and scroll in them.
  • Screen Recording lets the helper take window screenshots. Without it the tree still works, but screenshots fail.
macOS ties both permissions to the helper app itself, not to Terminal or Overdeck, which is why the helper is installed as its own app.

Setup

Open Settings → Computer Use and press Set up and turn on:
  1. Installs the helper to ~/.overdeck/computer-use/macos/Overdeck Computer Use.app. No password is needed. The app must stay at that path, because macOS ties the permissions to it.
  2. Opens the two System Settings pages (Privacy & Security → Accessibility and Screen Recording). Switch on Overdeck Computer Use in each, then press Continue in Overdeck.
  3. Turns computer_use.enabled on and tests the helper with a handshake.
Once it is on, the card lists Accessibility and Screen Recording with their status. A permission that is not allowed has an Open settings button that opens its System Settings page again. Overdeck re-checks when you come back to the dashboard window. The helper runs as one background process per user, shared by every pan computer call. It listens on /tmp/overdeck-cu-<uid>/agent.sock, only accepts callers running as your user with the token in ~/.overdeck/computer-use/macos/agent.token, and exits after 10 minutes without a request. On macOS, actions run in the background by default: the target app is not brought forward, your frontmost window keeps focus and the pointer does not move. An agent that passes --restore-window brings the app forward and uses your screen, keyboard focus and pointer for that action; the skill tells agents to do this only when a background action can’t work, and to tell you first.

Updates and signing

Overdeck builds the helper in CI. Without an Apple Developer ID certificate it is ad-hoc signed, and macOS ties the permissions to that exact build. So after an Overdeck update that changes the helper, macOS asks you to allow the updated helper again: the card shows Needs attention, and Fix setup reinstalls the helper and reopens the two System Settings pages, where you switch Overdeck Computer Use off and on again. A helper signed with a Developer ID certificate keeps its permissions across updates. For maintainers: the release workflow signs with a Developer ID certificate when MACOS_CERTIFICATE exists, and signs ad-hoc otherwise. It notarizes the helper only when it is Developer ID signed and the App Store Connect API key secrets exist; an ad-hoc build skips notarization, because Apple notarizes only Developer ID-signed apps: A source checkout without the bundled zip builds the helper on first setup with npm run build:computer-macos, which needs the Xcode Command Line Tools (xcode-select --install).

Requirements

On macOS, see macOS. On Linux:
  • A Linux desktop with a logged-in session (X11 or Wayland). A headless host or a linked machine with no desktop session gets no_desktop_session.
  • System packages: python3-gi, gir1.2-atspi-2.0, xdotool (X11 hotkeys/clicks), optionally xclip or xsel (clipboard paste; wl-copy on Wayland), and grim on Wayland (screenshots).
  • GNOME accessibility on: gsettings set org.gnome.desktop.interface toolkit-accessibility true.
  • Under WSL2, none of the above applies: see Windows (WSL2).

Windows (WSL2)

Overdeck on Windows runs inside WSL2. There, pan computer and Settings → Computer Use drive the Windows desktop, never WSLg windows. They use a vendored copy of Orca’s Windows UI Automation helper (native/computer-use-windows/runtime.ps1, MIT, see its README). The commands, flags and output are the same as on Linux.

Requirements

  • WSL2 with Windows interop on (the default). WSL1 is not supported.
  • Windows 10 or 11.
  • Windows PowerShell 5.1, which ships with Windows. Nothing is installed on Windows, so the Settings button reads Turn on Computer Use and asks for no password.

What runs where

  • Before each command, Overdeck copies the helper to %LOCALAPPDATA%\Overdeck\computer-use\ on the Windows side, when the copy is missing or differs. Operation files go to its ops\ folder and are deleted after each call. The WSL path of that folder is cached in ~/.overdeck/computer-use/windows-helper-dir.txt.
  • Each command starts one powershell.exe with -ExecutionPolicy Bypass. The bypass applies to that one process only and needs no administrator rights. Each command takes 1 to 3 s, because the helper compiles its Windows API bindings on every start.
  • A command that runs longer than 30 s is killed with helper_timeout. Killing it from WSL may leave the Windows powershell.exe running; the helper’s own steps are short, so this is rare.

Known limits

  • list-windows returns only an app’s main window (index 0). Wait for a dialog with pan computer wait --text instead.
  • The helper is DPI-aware only while it captures screenshots, so clicks can land in the wrong place on setups that mix display scales.
  • An app running as administrator refuses input with a target_elevated: message.
  • A WSL terminal started as administrator runs the helper elevated too.
  • A Group Policy that enforces signed scripts overrides -ExecutionPolicy; the helper then fails with a message asking the PC’s administrator to allow local scripts.
  • hotkey and press-key refuse Alt+F4, Win+L and Ctrl+Shift+Escape unless --allow-system-shortcut is passed.

Commands

Every subcommand takes --json. Every action and get-app-state also take --window-index <n>, --restore-window and --no-screenshot. See the pan-computer-use skill for the full field reference and error-handling guidance.

One agent at a time

A desktop lease, held in a file under Where files go, ensures only one holder drives the desktop’s input at a time. An input command claims the lease for 120 seconds, renewed on every call; a holder without it gets desktop_busy naming the current holder and when the lease expires. Read-only commands (capabilities, list-apps, list-windows, wait, and get-app-state without --restore-window) never need the lease, so other agents can still look. Run pan computer release when a GUI task is finished, or pan computer release --force to clear a stuck lease held by someone else.

Where files go

Everything lives under ~/.overdeck/computer-use/:

Safety

  • hotkey and press-key refuse a chord that would quit, lock, log out or switch virtual terminals (alt+F4, ctrl+alt+F2, super+l, ctrl+q, …) with system_shortcut_blocked, unless --allow-system-shortcut is passed. On macOS the refused chords are cmd+q, cmd+shift+q, cmd+ctrl+q, cmd+alt+escape and cmd+shift+backspace.
  • --text-stdin and --value-stdin read sensitive text from stdin so it never lands in argv or shell history.
  • The shipped skill tells agents never to push, submit, send, buy, delete, or change account settings through computer use unless the user explicitly asked for that action.

Limits

  • Wayland can’t do hotkeys (xdotool needs X11); screenshots there need grim.
  • No remote hosts: a linked machine with no logged-in desktop session always refuses.
  • On some multi-monitor setups, AT-SPI can report a window’s bounds at (0,0) instead of its real screen position. When that happens, its screenshot can show the wrong area of the desktop, and click/scroll/drag coordinates computed from that screenshot can miss the window entirely. Prefer --element-index actions over --x/--y coordinates when a window’s reported bounds look suspicious, and verify a coordinate action’s effect against a fresh get-app-state rather than trusting the screenshot alone.
  • The helper is a verbatim vendored port of Orca’s Linux computer-use bridge; see native/computer-use-linux/README.md for how it’s re-vendored.

Troubleshooting

On macOS, Settings → Computer Use shows every check (macOS version, helper, Accessibility, Screen Recording, handshake); pan doctor points there instead of probing. A permission_denied error means a permission is off: open Settings → Computer Use and finish setup. On Linux, run pan doctor. With the feature off it prints no Computer use row at all. With it on, it checks the AT-SPI Python bindings, the desktop session, xdotool on X11, a clipboard tool, grim on Wayland, and GNOME accessibility, ending in a single Ready (<session>, display <d>) row once the helper’s handshake succeeds. Under WSL2, pan doctor and the What Overdeck checked list show four checks: If the CLI works but the dashboard’s helper test fails with “Exec format error” or a vsock error, the dashboard service may be missing the WSL interop socket. Overdeck sets WSL_INTEROP from /run/WSL/ when it’s missing; report the error with your WSL version if it persists. Still stuck? Ask on the Overdeck Discord or file a GitHub Issue.