dsh-computer-use

一句话定位

让 DeepSeek Harness Agent 像一个会看屏幕的本地用户一样，操作 macOS 上的其他桌面 App：先读取无障碍 (Accessibility) 元素树，再点击、输入、滚动或拖拽，全程不抢前台、不动你真正的鼠标光标。

核心能力

• 列出当前可见 macOS App 并显示进程 ID、Bundle ID、是否最前
• 抓取目标 App 的无障碍元素树和当前状态，附带可选截图
• 通过无障碍语义（按角色、值、可执行动作）发起点击、设值、键入、按键、滚动、拖拽
• 对截图中的视觉问题，自动把产物交给 vision-tools Skill 处理（OCR、定位、裁剪）
• 按 App 维度授予"只读"或"可控制"权限，高风险动作需一次性确认令牌
• 在 Web 客户端提供一个 /computer-use/settings 页面，展示健康度、权限状态和应用授权

技术实现

• 语言: TypeScript（宿主插件）+ Swift（原生辅助进程）+ 一段 JSON 配置
• 关键依赖: @deepseek-ai/cordis、@deepseek-ai/dsh-tools、@deepseek-ai/schemastery、zod
• 架构模式: Cordis Service 启动 macOS Provider → 初始化阶段通过 installComputerUseConsumer 注册 computer-use Skill 和一个 computer_use_activate 引导工具；Agent 在会话里调用该工具加载 Skill 后，再由 exposure.ts 把 11 个 computer_* 工具按 Agent 粒度暴露给模型
• 入口文件: src/index.ts（导出 ComputerUseBundle），native/macos/manifest.json（固化原生 Helper 的 SHA-256、架构、最低 macOS 版本）

适用场景

需要让 Agent 操作一个没有专用连接器、没有 CLI、也没有 API 的 macOS 原生 App 时使用，例如填一份只能在桌面 App 里打开的表单、从一段 GUI 流程里抓数据、或在 Safari 之外的 App 里完成一个浏览器自动化也做不了的步骤。浏览器任务请用浏览器自动化能力，API/CLI 能解决的请用 API/CLI。

前置依赖与兼容性

安装方式

dsh plugin --profile web add github:Anionex/dsh-computer-use

该命令只对 Web Profile 生效；Headless Profile 需把末尾的 web 改成 headless 单独再装一次。npm 上的官方包名是 @anionex/dsh-computer-use，按本市场收录的命令形式通过 GitHub 源安装。

配置项

Bundle 启动时通过 cordis.patch.yml 把 focusPolicy 设为 preserve、keyboardPolicy 设为 activate，与上面裸 Schema 的默认值不一致；如需修改，请在 Settings 里覆盖。

常见问题

Q: 我装好了，但是在 Agent 里看不到 computer_click 等工具，怎么办？

A: 安装后默认只有 computer_use_activate 一个引导工具。让 Agent 在当前会话里执行 /computer-use 加载 Skill，或者直接调用 computer_use_activate，加载完成后 11 个执行类 computer_* 工具才会出现在工具列表里。重启 dsh web 主机并新建一个 Session 也是常见原因。

Q: 截图里看到的内容比较模糊，我想让 Agent 读图，怎么办？

A: 这个插件本身不做 OCR。computer_observe 返回截图时，会同时把 Artifact 路径交给当前 Agent；Agent 应当加载 vision-tools Skill，再把这段路径传给 vision_glance、vision_ground、vision_detect、vision_crop、vision_long_screenshot_ocr。请不要让它去用 tesseract、screencapture 或临时 Swift 脚本替代。

Q: 为什么 Agent 操作完之后，我的"最前面 App"被换了？

A: 默认策略下，语义无障碍动作和带目标进程的指针输入都不会抢前台。如果你看到前台被切走了，多半是 focusPolicy: activate 或 keyboardPolicy: activate 被打开，或者目标 App 自己的副作用引起的。把策略改回 preserve 即可恢复默认行为。

Q: 某些 App 明明没禁用辅助功能，但 Agent 还是失败，怎么办？

A: 这是设计上的"失败关闭"。custom canvas、游戏、走自定义输入处理的 App、以及未来 macOS 版本都可能拒绝通过 SkyLight 投递的指针或键盘事件。能用语义无障碍（按角色、值、advertised actions）解决就尽量用语义；坐标点击是兜底，不是首选。

Q: 我能给某个 App 开"只读"但不开"控制"权限吗？

A: 可以。在 Web Settings 的「Application access」里加一条精确的 Bundle ID 授权，把 read 勾上、control 不勾即可。control: true 会自动包含 read，反过来不会。

Q: 删除插件之后，截图和授权记录会一起没吗？

A: 不会。dsh plugin remove 只注销 Skill、工具、进程级观察和确认令牌；截图文件（在 artifactRoot 下）和 computer_use_state 侧车文件会保留，需要你手动清理。

上手难度

进阶 — 需要先在 macOS 系统设置里授予辅助功能/屏幕录制权限，再在 Web Settings 里按 Bundle ID 配置应用授权；同时要理解"观察 → 锁定目标 → 输入"的协议才能写出稳定的 Agent 工作流，普通用户首次使用会有一段适应期。

已知问题与限制

• 仅支持 macOS；Windows UI Automation 和 Linux 提供者尚未实现
• 状态处于早期 0.1.0，模型面（Skill 文案、工具签名）和提供者行为在稳定版之前可能变化
• 指针投递依赖动态解析的 SkyLight SPI；若该 SPI 在当前 macOS 上不可用，回退到坐标点击时会失败关闭，而不会切换到全局鼠标
• 点击坐标必须落在目标 App 的某个可见窗口里；Helper 会自动解析点击点下最顶层的匹配窗口，但最小化、隐藏或无窗口的目标会被拒绝
• 自定义画布、游戏、加固输入面、未来 macOS 版本可能拒绝目标进程级别的指针或键盘事件；推荐尽可能走语义无障碍
• focusPolicy: activate 和 keyboardPolicy: activate 是为兼容性预留的破坏性策略，仅在操作员明确要求时使用
• 插件只捕获"按需"的观察，不维护实时桌面流；如果需要持续画面，请使用屏幕录制或专门的视觉方案
• 浏览器任务请继续用浏览器自动化能力，因为 DOM/CDP 状态更窄、更准确
• DSH danger-full-access 内置授权策略是 approval/policy: never，会让未授权的 App 在弹窗前就被策略阻断；这种情况下插件会报 COMPUTER_PERMISSION_REQUIRED，并不视为用户拒绝

Native macOS control for DeepSeek Harness that keeps your real cursor and foreground application alone by default; the Bundle may bring the target app forward before keyboard input for reliable typing.

DSH Computer Use gives an Agent fresh Accessibility observations, exact process/window targeting, stale-state rejection, scoped application access, and verified post-action state. Semantic Accessibility comes first; mouse, drag, wheel, and keyboard fallback are routed to the selected process instead of the global desktop.

English | 中文

Why it is different

Accessibility permission lets a process inspect and operate macOS UI elements, but the permission itself does not prevent focus stealing or cursor movement. Those behaviors depend on the input route.

The default DSH Computer Use route is deliberately non-interfering:

• No system-cursor movement: the helper contains no cursor-warp path.
• No global pointer injection: click, scroll, and drag fallback use a pid/window-targeted SkyLight route, not the global HID event stream.
• No pointer-triggered activation: semantic Accessibility, process-targeted pointer input, and keyboardPolicy: preserve run without activation; keyboardPolicy: activate (Bundle default) brings the target app forward before keyboard fallback, matching Codex Computer Use.
• A separate Agent cursor: click, scroll, and drag actions animate a click-through, nonactivating software cursor while the macOS system cursor remains untouched. It is visible by default and stays at the action position until the bound window changes or a hide command; cursorAutoHideMs can opt into timed auto-hide.
• No blind replay: every action is tied to an exact, unexpired observation and returns fresh state.

The result is a native action layer that can operate many background applications while the user continues working in the current foreground application.

What it adds

• Observe before acting. Return a bounded Accessibility tree, indexed elements, exact app/process/window metadata, permission state, and an optional screenshot Artifact.
• Bind actions to state. Every element exposes an observation-local index and opaque targetHandle; exact lookup remains compatible, while explicitly allowed rebinding accepts only a unique native or semantic identity inside the same process and window.
• Prefer semantic input. Use AXPress, editable values, selected-text assignment, and advertised Accessibility actions before pointer fallback.
• Route fallback to the target. Keyboard input goes to the selected pid; pointer input goes to the selected pid and CGWindowID with window-local coordinates, resolving the app window under the point so arbitrary screen coordinates work.
• Return fresh evidence. Every successful action settles for a bounded interval and returns a new full or diff observation.
• Scope application access. Read and control leases are separated by Agent, Session, turn, and exact bundle id; high-impact actions require one-use confirmation.
• Keep the model surface focused. Execution Tools appear only after the current Agent loads the Computer Use Skill.

Proof: a never-active background fixture

The repository includes a deterministic AppKit fixture and a universal native helper. Release tests start the fixture with open -g in background mode, then use the same protocol exposed to the Agent.

observe exact bundle id + pid
-> element: "Targeted pointer probe", no AXPress action
-> computer_click with observationId + element index + allowCoordinateFallback
-> fresh observation
-> activation "not-requested"; pointerRouting "target-process"
-> status "Status: pointer click"

The fixture records every applicationDidBecomeActive callback. An independent native monitor also samples the system cursor and frontmost pid every millisecond throughout click, scroll, and drag. The default release path must not increase activationCount; it also requires unchanged cursor coordinates, an unchanged frontmost pid, exact click/scroll counts, and one complete down/up drag gesture.

See Foreground-safe input policy for the requirements, architecture, decisions, evidence, and compatibility limits.

Scope

dsh-computer-use is the native action layer. It does not replace narrower interfaces:

• browser tasks should use browser automation and DOM/CDP state;
• APIs, CLIs, and purpose-built application plugins remain preferable when available;
• OCR, visual grounding, and pixel interpretation should use the separately installed dsh-vision-toolkit: load the vision-tools Skill and pass the exact screenshot Artifact path to vision_glance, vision_ground, vision_detect, vision_crop, or vision_long_screenshot_ocr; do not replace those tools with shell-driven tesseract, screencapture, or ad hoc Swift/Python OCR;
• domain bundles such as dsh-design can compose Computer Use when a workflow crosses into a native application.

Quick start

Prerequisites

• macOS 14 or newer.
• DeepSeek Harness with a Web or Headless Profile and the Skill Tool mounted.
• macOS Accessibility permission for observation and native actions.
• macOS Screen Recording permission only when a screenshot is requested.
• Node.js ^22.19.0 or >=24.0.0 when building this repository.

Install the Web and Headless bundles directly from npm:

dsh plugin --profile web add @anionex/dsh-computer-use
dsh plugin --profile headless add @anionex/dsh-computer-use

dsh --profile web --dump-config | grep computer-use
dsh --profile headless --dump-config | grep computer-use

For local development, replace the package name with an absolute checkout path.

Restart a running dsh web host after changing the installed plugin, then start a new Session so the host reloads the Bundle and Skill catalog.

Load the Skill in that Session:

/computer-use

Then try:

Use Computer Use to inspect the running DSH Computer Use Fixture, enable its deterministic option, and report the fresh status. Prefer Accessibility elements and do not reuse an old observation.

How it works

flowchart LR
    A["Select exact bundle id and pid"] --> B["Acquire scoped read access"]
    B --> C["Observe AX tree and optional screenshot"]
    C --> D["Choose target handle, index, or window-relative point"]
    D --> E["Acquire control and optional one-use confirmation"]
    E --> F["Re-observe and validate exact target"]
    F --> G{"Input route"}
    G -->|"Semantic"| H["Accessibility action or value"]
    G -->|"Keyboard"| I["Post to target pid"]
    G -->|"Pointer"| J["Post to target pid + window"]
    H --> K["Wait for settlement"]
    I --> K
    J --> K
    K --> L["Return fresh full or diff observation"]

Every observed element has an observation-local compatibility index and an opaque targetHandle. Index-only actions retain exact locator behavior. A low-risk element action may pass targetHandle with allowRebind: true; immediately before input, the provider-independent resolver obtains fresh Accessibility state and checks, in order, the original locator, a unique provider-native identifier such as macOS AXIdentifier, then one unique semantic match over role, accessible name, advertised actions, and stable ancestor fingerprint. The resolver keeps the exact bundle id, pid, and selected-window identity, and fails closed with COMPUTER_TARGET_AMBIGUOUS or COMPUTER_TARGET_LOW_CONFIDENCE instead of guessing. Coordinate actions still require the complete referenced window state to remain current.

Successful element actions report resolution.mode, confidence, candidateCount, and targetChanged. A sensitive target that needs rebinding invalidates the prior one-use confirmation and returns COMPUTER_TARGET_REBIND_REQUIRES_CONFIRMATION; the caller must observe the current UI and confirm the newly selected handle. Visual coordinates are not target handles and never authorize sensitive rebinding. Provider-native visual hit-testing is not part of this foundation release and remains follow-up work.

The default interaction policy is:

interaction:
  focusPolicy: preserve
  keyboardPolicy: activate
  pointerInputPolicy: targeted
  cursorVisualization: visible
  cursorMotionMs: 180
  cursorAutoHideMs: 0

cursorVisualization: visible displays the Agent's own non-interactive cursor for click, scroll, and drag. It never replaces or moves the macOS system cursor. Set it to hidden when visual feedback is unwanted. pointerInputPolicy: deny disables coordinate click/fallback, scroll, and drag. keyboardPolicy: activate (Bundle default) makes type-text keyboard fallback and press-key reliable by activating the target app first; focusPolicy: activate is the broader compatibility mode that also activates before pointer input. After activation, the helper re-observes and revalidates the exact target before input.

The cursor is a 28x28 transparent whole-image cursor (Cursor arrow plus DeepSeek whale, assets/cursor.png) with the hotspot at the image's top-left corner. It is a separate process, click-through, nonactivating, and bound to the exact observed pid, window, and frame so it disappears if the target window closes, moves, resizes, or is minimized.

The helper executable is an internal DSH transport rather than a public authorization API. It requires an isolated process group plus parent-owned standard transports, so ordinary shell redirection fails closed before command parsing. This is defense in depth, not authentication against arbitrary code running as the same macOS user: a deliberately constructed detached parent can reproduce that transport topology. Use the registered Tools so application leases, sensitive-action confirmation, and host policy checks remain in force; danger-full-access must not be treated as protection against direct native invocation.

Successful action results include:

activation: 'not-requested' | 'already-frontmost' | 'activated'
pointerInput: boolean
pointerRouting: 'none' | 'target-process'
resolution?: {
  mode: 'exact-locator' | 'native-identifier' | 'semantic-rebind'
  confidence: number
  candidateCount: number
  targetChanged: boolean
}

The model cannot override these host policies through Tool arguments.

Model Tools

The Bundle initially contributes only computer_use_activate. Loading the Skill exposes the focused execution vocabulary for that Agent.

Show the complete Tool vocabulary

Tool	Purpose
computer_list_apps	List bounded user-facing applications with bundle id, pid, frontmost state, and permission diagnostics
computer_observe	Return a fresh full/diff Accessibility observation and optional screenshot Artifact
computer_click	Prefer AXPress; accept an exact index or opaque target handle, with optional safe rebinding, before target-process coordinate fallback
computer_set_value	Set or clear an editable Accessibility value through an exact index or opaque target handle without using the clipboard
computer_type_text	Insert Unicode through Accessibility when supported, with a process-targeted keyboard fallback
computer_press_key	Send one key from a finite vocabulary to the selected process, with optional modifiers
computer_scroll	Send bounded directional scrolling to the selected process and window at a resolved element or window/screen coordinate
computer_drag	Drag between two window/screen points in the referenced observation
computer_perform_action	Execute one Accessibility action advertised by an exact or safely rebound selected element
computer_wait	Poll one bounded text/role/title condition and return fresh state without modifying the app
computer_confirm	Obtain a one-use token bound to one exact sensitive action

No Tool accepts AppleScript, JXA, shell, Swift, Objective-C, native selectors, arbitrary Accessibility constants, or source code.

Observation, permissions, and sensitive actions

An observation contains an opaque id and expiry, exact app identity, frontmost/window metadata, bounded tree text, current elements with opaque target handles, optional screenshot metadata, and permission state. Target handles expose no provider object reference or native identifier. Secure text values are emitted as [secure]; they do not enter target descriptors, tree text, Tool results, screenshot metadata, or native errors. A screenshot can still contain other visible application data and should be treated as sensitive.

The technical access model has two exact-bundle-id leases:

• read: inspect Accessibility state and a requested screenshot;
• control: send UI input to the selected application.

Without a configured grant, DSH asks for approval. Read approval lasts for the Session; control approval lasts for the current turn. A user rejection is final for that app and scope for the rest of the Session.

The Bundle keeps Session-wide read grants and rejected app/scope decisions in its own computer_use_state storage-domain sidecar, fenced by the Session header's createdAt and cwd. It does not add Computer Use events to the official Session log or modify DSH Core. The Web Profile already composes @deepseek-ai/dsh-storage-domain; a custom Profile must compose it before this Bundle if interactive read grants or durable rejections are needed. Exact grants configured in Settings remain available without storage-domain, and an allowed control decision remains process-local for the current turn. When a durable interactive decision cannot be stored, the operation fails clearly instead of silently weakening its lifetime.

The DSH danger-full-access preset uses approval/policy: never, so an ungranted app is policy-blocked before any prompt. The plugin reports an actionable COMPUTER_PERMISSION_REQUIRED error and does not record that outcome as a user rejection. Add the exact bundle id in Computer Use Settings or use a preset whose approval policy is ask.

High-impact communication, sensitive-data transmission, irreversible deletion, account/security/privacy changes, unrequested installation, legal acceptance, and financial completion beyond explicit authorization require computer_confirm immediately before execution. The token is short-lived, one-use, and bound to the exact app, process, observation, target handle, and action. Grants do not bypass it. If resolution moves beyond the exact locator, the token is invalidated and a fresh observation plus confirmation is required.

macOS permissions and native integrity

The Web Settings section reports helper integrity, Accessibility and Screen Recording status, active generation, interaction policy, limits, and exact application grants. Its buttons can open the relevant macOS privacy pane after a user click; the plugin cannot grant TCC permission itself.

Accessibility and Screen Recording are UI permissions, not filesystem permissions. Normal use stays under DSH workspace-write: screenshots remain in the Session workspace, transient files use Session-private temporary storage, and the Bundle does not require danger-full-access.

The committed helper is an ad-hoc-signed universal arm64 + x86_64 binary targeting macOS 14 or newer. native/macos/manifest.json pins its SHA-256, source digest, architectures, and deployment target. pnpm run check:native also checks the target-process-only pointer route and rejects system-cursor warp or global pointer-post symbols.

Configuration

Show Bundle configuration fields

Field	Purpose
observationTtlMs	Lifetime of an observation before reuse is rejected; default 0 disables expiry, or set any value up to 86400000 ms (24 hours)
confirmationTtlMs	Lifetime of a one-use sensitive-action confirmation
actionTimeoutMs	Hard native action timeout from 1000 to 120000 ms
settleMs	Interval between post-action state checks from 0 to 10000 ms
maxSettleMs	Maximum post-action settlement budget from 100 to 60000 ms
maxNodes / maxDepth / maxTextBytes	Accessibility traversal and model-visible text bounds
maxScreenshotBytes	Maximum PNG Artifact size
artifactRoot	Workspace-relative screenshot directory
helper.path	Optional explicit external helper executable
helper.allowSourceBuild	Permit an explicit managed-source rebuild when the committed helper is absent; default false
interaction.focusPolicy	preserve (default) avoids target-app activation; activate explicitly permits it and requires re-observation/revalidation
interaction.keyboardPolicy	preserve keeps keyboard events routed without activation; activate (Bundle default) activates the target app before keyboard fallback
interaction.pointerInputPolicy	targeted (default) permits pid/window-targeted pointer input; deny disables click fallback, scroll, and drag
interaction.cursorVisualization	visible (default) shows the separate Agent cursor; hidden disables only the overlay
interaction.cursorMotionMs	Animated Agent-cursor travel duration, default 180 ms
interaction.cursorAutoHideMs	Idle time before the Agent cursor hides; default 0 keeps it visible until the bound window changes or a hide command, or set a finite value up to 30000 ms
allowAllApps	Grant read and control to every running app; default false. When enabled, exact grants are ignored
grants	Exact non-wildcard bundle-id read/control policy; control: true implies read

Settings updates replace the active provider generation only after validation and health checks pass. Replacement invalidates existing observations and pending confirmations.

Status and limitations

• Status: early 0.1.0; model-facing and provider behavior may change before a stable release.
• The current provider is macOS-only. Windows UI Automation and Linux providers are not implemented.
• Target-process pointer delivery uses dynamically resolved SkyLight SPI. If it is unavailable, pointer fallback fails closed rather than switching to global input.
• The clicked point must fall inside an on-screen window of the selected app; the helper resolves the topmost matching window so ambiguous frame/title matches no longer block coordinate actions. Minimized, hidden, or windowless targets fail closed.
• Custom canvases, games, hardened input surfaces, and future macOS releases may reject target-process pointer or keyboard events. Prefer semantic Accessibility whenever possible.
• focusPolicy: activate and keyboardPolicy: activate are intentionally disruptive and exist as operator-selected compatibility modes.
• A target application may change its own activation or focus as a side effect of an accepted action.
• The package captures requested discrete observations, not a live desktop feed.
• Browser work should continue to use browser automation because DOM/CDP state is narrower and more precise.
• The public npm package installs into both Web and Headless profiles as @anionex/dsh-computer-use.

Development and release verification

Place this repository beside a DeepSeek Harness checkout so TypeScript and Vitest resolve the exact DSH peer declarations and runtime modules:

workspace/
├── packages/
├── vendor/
└── dsh-computer-use/

Then run:

pnpm install --frozen-lockfile
pnpm run build
DSH_COMPUTER_USE_REQUIRE_TCC=1 pnpm test
pnpm run check:native
pnpm pack --dry-run
pnpm run validate

pnpm run validate runs the keyless local and clean Web/Headless Profile lanes. The real-model release lane needs DEEPSEEK_API_KEY and accepts an optional DEEPSEEK_BASE_URL:

pnpm run validate:model
# or keyless validation followed by the real-model lane
pnpm run validate:release

Removal

dsh plugin --profile web remove @anionex/dsh-computer-use
dsh plugin --profile headless remove @anionex/dsh-computer-use

Removing or disabling the Bundle unregisters the Skill and Tools, cancels helper work, releases process-local Agent observations, turn control grants, and confirmations, closes its storage-domain handle, and removes Web contributions. Existing screenshot files and the plugin-owned computer_use_state sidecar remain for explicit user cleanup.

Security, community, and support

• Report suspected vulnerabilities privately through SECURITY.md.
• Read CONTRIBUTING.md before changing code or documentation.
• Use SUPPORT.md for installation, permission, configuration, and workflow questions.
• Follow the Code of Conduct in project spaces.
• See CHANGELOG.md for release history.
• See FUNDING.md to support maintenance without purchasing roadmap control or private support.

About

DSH Computer Use is maintained by anionex. If you would like to follow my future work, follow me on X or GitHub.

License

MIT © 2026 anionex.