一句话定位

为 DeepSeek Harness 中只能看文字的纯文本模型补上眼睛，让它们能直接看懂粘贴的截图并回答关于图像内容的问题。在此基础上还提供元素定位、长图 OCR、像素对比、UI 还原等本地视觉处理能力。

核心能力

• 粘贴即用：把截图粘到 DSH Web，文本模型自动切到 (Vision Toolkit) 变体并理解画面，回答、OCR、多图比较都可用
• 视觉定位：按名称找出图中的特定元素（如"登录按钮"），返回原图像素坐标并可生成带框预览图
• 元素清点：按类别（图标、按钮等）枚举图中所有元素，返回编号清单和坐标
• 长截图 OCR：自动把很长的截图切成多块并合并成 Markdown，支持断点续传和拆分预览
• 像素级对比：拿参考图与实现图对比，输出差异比例、最差区域、热力图和 JSON 报告
• 本地图像处理：在不联网的情况下完成裁剪、SVG 描摹、前景抠图、主色提取和本地 HTML 截图

技术实现

• 语言: TypeScript（同时打包出 React 客户端与 Node 后端）
• 关键依赖: @deepseek-ai/cordis（DSH 插件运行时）、@deepseek-ai/dsh-tools（原生 Tool 注册）、@deepseek-ai/dsh-skill（Skill 注入）、saxes（SVG 解析）
• 架构模式: 通过 cordis.patch.yml 以 vision-toolkit 身份挂入 DSH Profile 层；启动后注册一个 vision-tools Skill 与 10 个原生 Tool；Tool 调用通过 DSH Subprocess 在打包好的 agent-vision-toolkit Python 快照中执行，运行在隔离的 Python 虚拟环境（$DSH_HOME/cache/dsh-vision-toolkit）
• 入口文件: src/index.ts（服务端 apply 钩子），前端扩展由 src/client/index.tsx 注册到 Web Profile

适用场景

在 DSH 里用 DeepSeek 等纯文本模型做 UI 还原、截图排障、视觉回归测试、长截图转写时最有用。当你需要把"截图"变成"模型可以读取的证据"，或想从截图里抠图标、提取品牌主色、做局部裁剪时，本插件可以避免反复切换工具链。

前置依赖与兼容性

安装方式

dsh plugin --profile web add github:Anionex/dsh-vision-toolkit

配置项

常见问题

Q: 安装后还需要自己申请视觉模型的 API Key 吗？

A: 默认不需要。插件内置免费的 Gemini 3.7 Flash 视觉服务（vision.anionex.me），开箱即可粘贴图片提问；如果需要更高额度或私有端点，可在 Settings → 视觉工具 中替换为自备的 OpenAI/Anthropic 兼容服务。

Q: 在 Web 里粘贴图片，模型还是提示不支持图像输入怎么办？

A: 通常是因为页面缓存或 Profile 没有切到带 (Vision Toolkit) 后缀的变体。重启 Web Profile 并刷新页面，确认当前模型已切换到 (Vision Toolkit) 变体；也可以把图片放进会话工作区，再通过 /vision-tools Skill 触发。

Q: 免费视觉服务返回 429 该如何处理？

A: 这是共享容量临时耗尽。按错误响应里的 Retry-After 秒数等待后重试即可；如果经常触发，建议在 Settings 中替换为自备端点（Groq、自建 OpenAI 兼容网关均可）。

Q: 报错说图片过大或像素超限，如何处理？

A: 错误会明确指出是字节限制（默认 4 MiB）还是像素限制（默认 2000 万）。先用 vision_crop 或外部工具裁剪/缩放图片后再调用视觉工具。

Q: vision_html_screenshot 找不到 Chrome 怎么办？

A: 安装 Chrome、Chromium 或 Edge 任一浏览器即可。只有 HTML 截图这一个工具受影响，其他 9 个视觉工具仍可正常使用。

Q: 是否支持视频、音频或摄像头输入？

A: 当前版本不支持。本插件只处理静态图片（PNG/JPEG/GIF/WebP）和本地 HTML 截图，不会自动点击 GUI，也不做视频流、音频或摄像头采集。

Q: 插件的运行数据存放在哪里？

A: 粘贴的图片存到会话工作区下的 .dsh-vision-toolkit/ 目录；插件隔离的 Python 虚拟环境与运行时缓存放在 $DSH_HOME/cache/dsh-vision-toolkit（DSH_HOME 未设置时回退到 ~/.dsh/cache/dsh-vision-toolkit）。

Q: 如何卸载或临时禁用这个插件？

A: 卸载命令为 dsh plugin --profile web remove @anionex/dsh-vision-toolkit；如需临时关闭，可在 Profile patch 中设置 disabled: true，重启 Profile 即可生效。

上手难度

进阶 — 安装命令即用，但要让 Python 隔离环境、可选 Chrome、API Key 凭据都能按预期工作，需要理解 DSH Profile patch、Credential 引用与 Settings 的关系。普通用户按默认配置粘贴图片就能体验核心能力，无需编程。

已知问题与限制

• 不支持视频、音频或摄像头输入，也不会自动点击 GUI（README.zh.md:393-395）
• 交互式标注编辑、远程服务集群、模型投票、跨会话视觉缓存不在当前范围（README.zh.md:394）
• vision_html_screenshot 必须依赖 Chrome/Chromium/Edge 中的一个；缺失时仅该工具不可用
• 自动 Python 探测失败时需要手动指定 runtime.python；Windows 启动器请填 py，不要填 py -3
• 共享免费视觉服务有容量上限，繁忙时段会返回 429，按 Retry-After 重试即可
• 单次请求最多 5 张图片、单张最大 4 MiB、单张最多 2000 万像素、单次输出最多 4096 tokens（来自 README 配置章节）

If you use DeepSeek or another text-only model in DeepSeek Harness (DSH), you may have run into the same problems: the model cannot see a screenshot, generic descriptions miss the point, buttons have no usable coordinates, and a rebuilt page can look “close enough” without a way to measure the remaining difference.

🏆 This project is the first comprehensive vision-tool plugin in the DeepSeek Harness ecosystem: it was initiated before internal beta and built during the beta with reference to agent-vision-toolkit.

Original work: The system and division of responsibilities behind these visual tools, together with the vision-skills Skill, were personally created and continuously refined by the author through long-term real-world use and repeated iteration.

Highlights

• Paste and use it immediately. Paste an image in DSH Web and the text-only route switches to its (Vision Toolkit) variant without manual path copying or model changes.
• A seamless image workflow. Native thumbnails, session history, and workspace paths stay intact; Web can preview artifacts and Headless can continue using the same structured results.
• One command to install. The built-in free Gemini 3.7 Flash vision service is ready after installation, with no API key required.
• Built-in free vision. The shared service works immediately after installation with a quota of 300 images per machine per day.
• Vision guided by intent. The agent extracts evidence for the task at hand, such as “Where is the error?” or “Where is the button?”, instead of returning a generic caption.
• A complete screenshot-to-verification loop. Reference images, HTML screenshots, difference regions, and pixel comparison work together for UI restoration.

agent-vision-toolkit gives an agent more than image captions: it can read, locate, crop, trace, rebuild, and verify visual work. DSH Vision Toolkit is its native DeepSeek Harness integration, bringing that workflow into Web and Headless Profiles.

This project has two layers:

1. Visual tools and a Skill: the agent learns when to inspect, ground, OCR, crop, trace, or compare pixels.
2. Native DSH integration: those capabilities live inside Profiles, sessions, Settings, Artifacts, and the Web UI, with a free Gemini 3.7 Flash vision service ready after installation.

Install and use it immediately. The default setup includes a free Gemini 3.7 Flash vision service and requires no API key. Cropping, pixel diffing, color analysis, foreground extraction, SVG tracing, and HTML screenshots run locally without spending vision API requests.

dsh plugin --profile web add @anionex/dsh-vision-toolkit

Upstream toolkit: Anionex/agent-vision-toolkit · Project website: agent-vision.anionex.me

Recent updates

• 2026-08-16 · Windows Python: Added Microsoft Store Python support, fixing first-time isolated-runtime setup failures for affected Windows users.
• 2026-08-17 · Free vision upgrade: Switched the built-in no-key service to Gemini 3.7 Flash and fixed Qwen/Gemini bounding-box coordinate order.
• 2026-08-16 · Better free vision: Switched the built-in no-key service to Groq Qwen3.6, improving image understanding without adding setup steps.
• 2026-08-16 · Image paste: Text-only routes now switch to a (Vision Toolkit) variant and keep a workspace path, fixing blocked pastes and images that could not be reused later.
• 2026-08-16 · More shared capacity: Expanded the free service capacity to reduce peak-time 429 responses.
• 2026-08-16 · Real model test: Added a full image-request test in Settings, fixing the false confidence caused by a successful /models request to a model that still cannot process images.

Who it is for

See it in action

Paste an image directly into DSH

Paste an image into the conversation. A text-only model can switch to its Vision Toolkit variant and inspect the image in the context of the user's question.

Screenshot to editable page

Left: the reference screenshot. Right: an editable HTML/CSS result. The result can continue into screenshot rendering and pixel comparison instead of ending as an image description.

Sketch to working interface

Left: a hand-drawn reference. Right: the working interface reconstructed from it.

Turn “looks close” into a verifiable result

The repository includes a reproducible UI-restoration example: the agent renders the reference and implementation, then uses difference regions, a heatmap, and a JSON report to guide the next correction.

Quick start: three steps

1. Install

dsh plugin --profile web add @anionex/dsh-vision-toolkit

You can install it into a Headless Profile too:

dsh plugin --profile headless add @anionex/dsh-vision-toolkit

2. Restart and check it

Restart a running Web Profile, then open Settings → Vision Toolkit. The free provider is already configured; run Test vision model to confirm it is reachable.

The first start prepares an isolated runtime: the plugin prefers a system Python 3.11+; when none is found, it downloads a hash-verified standalone Python (about 35 MB) from a pinned release source on first use. A normal installation does not require an agent-vision-toolkit source checkout or a local path setting.

3. Paste an image and describe the outcome you want

Paste a screenshot into the conversation or place an image in the session workspace, then invoke /vision-skills. For example:

Inspect this screenshot. Explain the error and tell me what to fix first.
Find the login button in the top-right corner, return original pixel coordinates, and make a boxed preview.
Crop this icon and convert it to SVG.
Rebuild the page from reference.png. After each pass, render it and run a pixel diff until the major differences are gone.

Common workflows

Toolbox

The plugin provides 10 tools that can be called independently or composed into a workflow:

Coordinates always use original-image pixels in x1,y1,x2,y2 form, so grounding output can feed directly into cropping, tracing, or later automation.

For a long HTML document, pass fullPage=true. The requested width and height remain the layout viewport, while the resulting PNG covers the complete document and reports pageHeight in CSS pixels.

How it works

The plugin keeps image understanding and deterministic local image processing in one Agent workflow. Expand the flow below for the implementation boundary.

flowchart LR
    Image["Screenshot or local HTML"] --> Skill["vision-skills Skill"]
    Skill --> Agent["Text agent selects a task"]
    Agent --> Vision["Use a vision model when image understanding is needed"]
    Agent --> Local["Run crop, SVG, and pixel work locally"]
    Vision --> Result["Answer, OCR, coordinates"]
    Local --> Artifact["PNG, SVG, heatmap, JSON"]
    Result --> Session["Continue reasoning and acting"]
    Artifact --> Session

Configuration and limits

Built-in free service

The default setup uses:

Base URL: https://vision.anionex.me/v1
Model:    gemini-3.7-flash
API Key:  https://agent-vision.anionex.me (filled automatically)

Requests that still use the previous qwen/qwen3.6-27b model name remain compatible and are routed to the Qwen backend.

This is a shared zero-configuration entry point, not an unlimited private endpoint. Request safeguards include:

These safeguards prevent unusually large requests from monopolizing memory or request time. When shared capacity is reached, the service returns a readable 429 response with Retry-After instead of collapsing into an unexplained model failure.

Existing clients that still send api_key="free" remain compatible.

Bring your own vision model

For higher quotas, private endpoints, or another model, change the provider in Settings → Vision Toolkit and store the API key as a DSH Credential. Settings stores the Credential reference and never reads the saved secret back into the browser.

Step-by-step Groq tutorial: Get a free Groq API key and use Qwen3.6-27B for image understanding. It includes screenshots for account/API-key setup, the exact Vision Toolkit settings, and working cURL and Python examples.

You can also configure a Profile patch:

- id: vision-toolkit
  config:
    provider:
      baseUrl: https://api.example.com/v1
      credential: MY_VISION_KEY
      model: your-vision-model
      protocol: openai

OpenAI Chat Completions-compatible endpoints and Anthropic Messages are supported. The Web Settings panel exposes the full provider, runtime, timeout, image-limit, and image-input-variant configuration.

For a trusted internal endpoint that uses a self-signed certificate or MITM proxy, start the DSH process with VISION_SSL_VERIFY=0. The plugin forwards that value to the isolated Python runtime; certificate verification remains enabled when the variable is unset or has any other value. The false values false, off, no, none, and disabled are also accepted, case-insensitively.

Requirements

• A DeepSeek Harness Web or Headless Profile.
• Node.js ^22.19.0 or >=24.0.0.
• Python 3.11+ is usually not needed in advance: the plugin prefers a system Python and otherwise downloads a pinned standalone Python 3.13 automatically, preparing its own isolated environment. Only that first automatic download needs network access.
• Only vision_html_screenshot requires Chrome, Chromium, or Edge.
• Inputs must be PNG, JPEG, GIF, or WebP files in the session workspace, the platform temporary directory, or an explicitly allowed directory.

Configure the Python runtime

By default the plugin picks a system Python 3.11+, or downloads a standalone Python when none is found; most users never need to configure this section. The rest is for advanced setups where automatic discovery fails, a specific interpreter is required, or an external runtime is used.

The packaged managed runtime creates its own isolated virtual environment. runtime.python selects the Python executable used to bootstrap or refresh that environment; it does not replace the managed environment with the interpreter's global site-packages. Set it when automatic discovery fails or when several Python installations exist. The override is also used by runtime.mode: external.

Python 3.11 or newer is required; the automatically downloaded standalone Python is 3.13.15 and, like a system interpreter, is only used to bootstrap the isolated environment. Without an override, the plugin tries python3 then python on macOS/Linux, and python, py -3, then python3 on Windows, before falling back to the standalone download. A configured value is passed as one executable name or path, not as a shell command with arguments, so use py (not py -3) for the Windows launcher; use an absolute path when you need a specific version.

Configure it in the Profile patch:

- id: vision-toolkit
  config:
    runtime:
      # macOS/Linux system Python
      python: python3
      # Or a project-local environment:
      # python: /absolute/path/to/project/.venv/bin/python
      # Windows venv (forward slashes also work in YAML):
      # python: C:/Users/you/project/.venv/Scripts/python.exe
      # Windows launcher, when its default Python is 3.11+:
      # python: py

For a managed runtime, create the project-local interpreter and point runtime.python at it. The plugin installs the locked dependencies into its own managed cache, so installing the lockfile into this bootstrap environment is optional:

python3 --version                         # must report 3.11 or newer
uv venv .venv --python 3.13

For runtime.mode: external, install the locked dependencies using the runtime/requirements.lock from the DSH Vision Toolkit plugin checkout, then point runtime.agentVisionToolkitPath at a separate exact agent-vision-toolkit snapshot. The packaged vendor/agent-vision-toolkit directory is such a snapshot when it has not been modified:

uv pip install --python .venv/bin/python \
  -r /absolute/path/to/dsh-vision-toolkit/runtime/requirements.lock

- id: vision-toolkit
  config:
    runtime:
      mode: external
      python: /absolute/path/to/dsh-vision-toolkit/.venv/bin/python
      agentVisionToolkitPath: /absolute/path/to/dsh-vision-toolkit/vendor/agent-vision-toolkit

On Windows, use py -3 --version for the version check and .venv\Scripts\python.exe plus runtime\requirements.lock in the corresponding commands:

py -3 --version                         # must report 3.11 or newer
uv venv .venv --python 3.13
# External mode only; use the plugin checkout's absolute lockfile path:
uv pip install --python .venv\Scripts\python.exe -r C:\absolute\path\to\dsh-vision-toolkit\runtime\requirements.lock

Point runtime.python at the same interpreter, save the Profile patch, and restart the Web Profile. Then open Settings → Vision Toolkit: the Runtime panel should show the resolved interpreter and Python version, and Run health check plus Test vision model should complete without the Python-version error. A final smoke test is to place a PNG/JPEG in the session workspace and call vision_glance.

The path fence automatically allows the session workspace and the platform temporary directory. On macOS/Linux the temporary root is /tmp. On Windows it is TEMP, then TMP, with the operating-system fallback if neither is set; model-generated /tmp/... paths are translated to that Windows directory before the normal realpath fence runs. No allowedDirs entry is needed for these platform temporary paths.

Use allowedDirs only for additional trusted input roots outside the workspace and platform temporary directory:

- id: vision-toolkit
  config:
    allowedDirs:
      # macOS/Linux example
      - /srv/vision-inputs
      # Windows example (use this instead on Windows)
      # - D:/vision-inputs

allowedDirs is an input allowlist, not the managed runtime cache. The managed runtime keeps its own files under $DSH_HOME/cache/dsh-vision-toolkit (or ~/.dsh/cache/dsh-vision-toolkit when DSH_HOME is unset); that directory does not need to be added. Environment-variable forms such as $env:TEMP and %TEMP% are not expanded inside allowedDirs, so configure extra roots with real absolute paths.

dsh plugin --profile web update @anionex/dsh-vision-toolkit
dsh plugin --profile web remove @anionex/dsh-vision-toolkit

- id: vision-toolkit
  disabled: true

Plugin updates

In Settings → Vision Toolkit, Check for updates queries the Profile's npm registry. For a direct registry installation, Update and restart installs only the exact version you confirmed, verifies it, and restarts an explicitly opted-in POSIX Web process on a fixed --port. Local/workspace/file/git/URL installs, Windows, dynamic ports, read-only Profiles, and manager-owned processes remain check-only.

The updater revalidates the Profile before mutation, snapshots the original manifest and lockfile, and holds a token-owned cross-process lock. The current Web process exits only after the restart helper confirms that the backup is readable and the lock handoff succeeded. When the Profile was already operational, the replacement must report both the target plugin version and a ready runtime; failed replacements restore the original manifest/lockfile and rebuild dependencies with a frozen lockfile before retrying the previous exact version. If automatic recovery itself fails, the backup and lock are preserved and their paths are written to $DSH_HOME/logs/vision-toolkit-restart.log. Detached restart requires DSH_VISION_TOOLKIT_ALLOW_DETACHED_RESTART=1; unsaved Settings or API-key input blocks installation.

Troubleshooting

Project status and limitations

The current release focuses on screenshot understanding, visual grounding, OCR, asset extraction, UI restoration, and pixel-level verification. It is not a video, audio, or camera-input system and does not automatically click GUI controls. Interactive box editing, remote service clusters, model voting, and cross-session visual caches are also outside the current scope.

Development and community

pnpm install --frozen-lockfile --trust-lockfile
pnpm run verify:portable
pnpm run build
pnpm test
TSX_TSCONFIG_PATH=tsconfig.json pnpm dlx tsx scripts/ui-restoration-example.ts --check

• Read CONTRIBUTING.md before contributing.
• Use GitHub Issues for bugs, focused feature requests, and usage questions; see SUPPORT.md for channel guidance.
• Report vulnerabilities privately through SECURITY.md.
• See CHANGELOG.md for releases and FUNDING.md for sponsorship details.
• Visit upstream agent-vision-toolkit for the general toolkit, cross-agent integrations, and visual-task playbooks.

agent-vision-toolkit was created by Anionex. This repository maintains its native DeepSeek Harness integration.

License

The plugin is available under the MIT License. The packaged upstream snapshot retains its original MIT license in vendor/agent-vision-toolkit/LICENSE.