Computer use

Drive local desktop apps from an agent via accessibility trees, screenshots, and safe UI actions.

The devcore computer CLI lets an agent inspect and control native desktop apps — list running apps, read accessibility trees, click controls, set values, type text, scroll, and take screenshots. Use it when a task needs to operate the OS or a third-party app rather than a terminal or the built-in browser.

Beta

Computer use ships native helpers per platform and requires Accessibility (and on macOS, Screen Recording) permission. The command surface is stable enough for skills to build against, but flag names may still shift.

First-time setup

Check the runtime and permissions:

devcore status --json
devcore computer permissions --json
devcore computer capabilities --json

If permissions reports anything missing, grant Accessibility (and Screen Recording on macOS) to DevAgent Computer Use in System Settings, then re-run permissions --json to confirm.

Snapshot → act → snapshot

Every interaction follows the same loop: read the app's current state, act on a specific element, then re-read state to verify the result.

devcore computer list-apps --json
devcore computer get-app-state --app com.spotify.client --json
devcore computer click --app com.spotify.client --element-index 42 --json

Element indexes are scoped to the latest get-app-state result and may be sparse. In --json output, read the tree from result.snapshot.treeText. Do not invent indexes from elementCount. Refresh state after navigation, focus changes, scrolling, or any app re-render before reusing an index.

Multi-window apps

devcore computer list-windows --app com.microsoft.edgemac --json
devcore computer get-app-state --app com.microsoft.edgemac --window-id <id> --json
devcore computer click --app com.microsoft.edgemac --window-id <id> --element-index 12 --json

Prefer stable --window-id when the listed id is not none; otherwise use --window-index.

Selecting an app

Prefer bundle IDs returned by list-apps:

devcore computer get-app-state --app com.microsoft.edgemac --json

App names work when unambiguous (--app Spotify). Use --app pid:<number> only when bundle ID and name both collide.

Available actions

devcore computer click --app <app> --element-index <i> --json
devcore computer set-value --app <app> --element-index <i> --value "text" --json
devcore computer type-text --app <app> --text "text" --json
devcore computer press-key --app <app> --key Return --json
devcore computer hotkey --app <app> --key CmdOrCtrl+A --json
devcore computer paste-text --app <app> --text "text" --json
devcore computer scroll --app <app> --element-index <i> --direction down --json
devcore computer drag --app <app> --from-x 100 --from-y 100 --to-x 300 --to-y 300 --json
devcore computer perform-secondary-action --app <app> --element-index <i> --action <name> --json

Prefer semantic actions (click, set-value, perform-secondary-action) over raw type-text or press-key — they target accessibility elements directly and survive focus changes that keyboard input doesn't.

When accessibility targeting fails, fall back carefully to coordinates:

devcore computer click --app com.apple.Safari --x 120 --y 340 --json
devcore computer drag --app <app> --from-element-index 3 --to-element-index 9 --json

Sensitive input

Pass secrets through stdin so they don't land in shell history:

printf '%s' "$TEXT" | devcore computer set-value \
  --app com.apple.Safari --element-index 7 --value-stdin --json

--text-stdin works the same way for type-text and paste-text.

Screenshots

get-app-state returns an accessibility tree and, by default, a screenshot. With --json, the image bytes are written to disk and the path is returned in screenshot.path rather than embedded in the response. Pass --no-screenshot when pixels aren't needed (faster, smaller payload). Pass --restore-window to bring a hidden or minimized window into view before capture.

Use it from an agent

The shipped computer-use skill packages the same command surface with safety guidance. Install it into the agent's skill directory:

npx skills add https://github.com/Clouditera/devagent-skills --skill computer-use

See Skills registry & MCP for how skills are picked up.

Next steps

DevCore CLI overview — the rest of the CLI surface (worktrees, terminal, browser).
Skills registry & MCP — distributing this CLI to agents.

Computer use ​

First-time setup ​

Snapshot → act → snapshot ​

Multi-window apps ​

Selecting an app ​

Available actions ​

Sensitive input ​

Screenshots ​

Use it from an agent ​

Next steps ​