At a Glance

Equips text-only models in DeepSeek Harness with "eyes" so they can directly interpret pasted screenshots and answer questions about image content. Additionally provides local visual processing capabilities including element localization, long screenshot OCR, pixel-level comparison, and UI reconstruction.

Core Features

• Paste to Use: Paste a screenshot into DSH Web, the text model automatically switches to the (Vision Toolkit) variant to understand the image—answering questions, OCR, and multi-image comparison all work out of the box
• Visual Localization: Find specific elements in images by name (e.g., "login button"), return original image pixel coordinates and generate preview images with bounding boxes
• Element Inventory: Enumerate all elements in an image by category (icons, buttons, etc.), return an indexed list with coordinates
• Long Screenshot OCR: Automatically split long screenshots into multiple chunks and merge them into Markdown, supporting resumable uploads and split preview
• Pixel-Level Comparison: Compare reference images with implementation images, output difference ratio, worst area, heatmap, and JSON report
• Local Image Processing: Complete cropping, SVG tracing, foreground matting, dominant color extraction, and local HTML screenshots without internet access

Technical Implementation

• Language: TypeScript (bundles both React client and Node backend)
• Key Dependencies: @deepseek-ai/cordis (DSH plugin runtime), @deepseek-ai/dsh-tools (native Tool registration), @deepseek-ai/dsh-skill (Skill injection), saxes (SVG parsing)
• Architecture Pattern: Mounts into DSH Profile layer as vision-toolkit via cordis.patch.yml; after startup registers one vision-tools Skill and 10 native Tools; Tool calls execute in a packaged agent-vision-toolkit Python snapshot via DSH Subprocess, running in an isolated Python virtual environment ($DSH_HOME/cache/dsh-vision-toolkit)
• Entry File: src/index.ts (server-side apply hook), frontend extension registered to Web Profile via src/client/index.tsx

Use Cases

Most useful when using text-only models like DeepSeek in DSH for UI reconstruction, screenshot debugging, visual regression testing, and long screenshot transcription. When you need to turn "screenshots" into "evidence the model can read," or want to extract icons from screenshots, extract brand dominant colors, or do local cropping—this plugin avoids repeatedly switching toolchains.

Prerequisites & Compatibility

Installation

dsh plugin --profile web add github:Anionex/dsh-vision-toolkit

Configuration Options

FAQ

Q: Do I need to apply for a vision model API Key myself after installation?

A: Not by default. The plugin includes a free Gemini 3.7 Flash vision service (vision.anionex.me), ready to paste images and ask questions out of the box; if you need higher quotas or a private endpoint, you can replace it with a self-provided OpenAI/Anthropic compatible service in Settings → Vision Tools.

Q: I pasted an image in Web, but the model still says it doesn't support image input?

A: Usually caused by page cache or Profile not switching to the variant with (Vision Toolkit) suffix. Restart the Web Profile and refresh the page, confirm the current model has switched to the (Vision Toolkit) variant; alternatively, put the image in the session workspace and trigger it via the /vision-tools Skill.

Q: The free vision service returns 429, what should I do?

A: This is temporary shared capacity exhaustion. Wait for the number of seconds specified in the Retry-After header in the error response, then retry; if this happens frequently, consider replacing with a self-provided endpoint in Settings (Groq, self-hosted OpenAI-compatible gateway both work).

Q: The error says the image is too large or pixel limit exceeded, how to handle?

A: The error will clearly indicate whether it's the byte limit (default 4 MiB) or pixel limit (default 20 million). First use vision_crop or external tools to crop/resize the image before calling the vision tool.

Q: vision_html_screenshot can't find Chrome, what should I do?

A: Install Chrome, Chromium, or Edge—any one of them works. Only this HTML screenshot tool is affected; the other 9 vision tools work normally.

Q: Is video, audio, or camera input supported?

A: Not supported in the current version. This plugin only processes static images (PNG/JPEG/GIF/WebP) and local HTML screenshots; it doesn't automatically click GUI elements, nor does it handle video streams, audio, or camera capture.

Q: Where does the plugin store its runtime data?

A: Pasted images are stored in the session workspace under .dsh-vision-toolkit/; the plugin's isolated Python virtual environment and runtime cache are stored in $DSH_HOME/cache/dsh-vision-toolkit (falls back to ~/.dsh/cache/dsh-vision-toolkit when DSH_HOME is not set).

Q: How to uninstall or temporarily disable this plugin?

A: Uninstall command is dsh plugin --profile web remove @anionex/dsh-vision-toolkit; to temporarily disable, set disabled: true in the Profile patch, then restart the Profile to take effect.

Difficulty Level

Advanced — Installation command is ready to use, but for the Python isolated environment, optional Chrome, and API Key credentials to work as expected, you need to understand the relationship between DSH Profile patch, Credential references, and Settings. Regular users can experience core features by simply pasting images with default configuration; no programming required.

Known Issues & Limitations

• Does not support video, audio, or camera input, and won't automatically click GUI (README.zh.md:393-395)
• Interactive annotation editing, remote service clusters, model voting, and cross-session visual caching are out of scope (README.zh.md:394)
• vision_html_screenshot must depend on one of Chrome/Chromium/Edge; when missing, only this tool is unavailable
• When automatic Python detection fails, manual runtime.python specification is needed; on Windows, fill in py, not py -3
• The shared free vision service has capacity limits, and will return 429 during peak times; retry according to Retry-After
• Maximum 5 images per request, 4 MiB per image, 20 million pixels per image, 4096 tokens max output (from README configuration section)

If you use DeepSeek or another text-only model in DeepSeek Harness (DSH), you may have run into the same problems: the model cannot see a screenshot, generic descriptions miss the point, buttons have no usable coordinates, and a rebuilt page can look “close enough” without a way to measure the remaining difference.

🏆 This project is the first comprehensive vision-tool plugin in the DeepSeek Harness ecosystem: it was initiated before internal beta and built during the beta with reference to agent-vision-toolkit.

Original work: The system and division of responsibilities behind these visual tools, together with the vision-skills Skill, were personally created and continuously refined by the author through long-term real-world use and repeated iteration.

Highlights

• Paste and use it immediately. Paste an image in DSH Web and the text-only route switches to its (Vision Toolkit) variant without manual path copying or model changes.
• A seamless image workflow. Native thumbnails, session history, and workspace paths stay intact; Web can preview artifacts and Headless can continue using the same structured results.
• One command to install. The built-in free Gemini 3.7 Flash vision service is ready after installation, with no API key required.
• Built-in free vision. The shared service works immediately after installation with a quota of 300 images per machine per day.
• Vision guided by intent. The agent extracts evidence for the task at hand, such as “Where is the error?” or “Where is the button?”, instead of returning a generic caption.
• A complete screenshot-to-verification loop. Reference images, HTML screenshots, difference regions, and pixel comparison work together for UI restoration.

agent-vision-toolkit gives an agent more than image captions: it can read, locate, crop, trace, rebuild, and verify visual work. DSH Vision Toolkit is its native DeepSeek Harness integration, bringing that workflow into Web and Headless Profiles.

This project has two layers:

1. Visual tools and a Skill: the agent learns when to inspect, ground, OCR, crop, trace, or compare pixels.
2. Native DSH integration: those capabilities live inside Profiles, sessions, Settings, Artifacts, and the Web UI, with a free Gemini 3.7 Flash vision service ready after installation.

Install and use it immediately. The default setup includes a free Gemini 3.7 Flash vision service and requires no API key. Cropping, pixel diffing, color analysis, foreground extraction, SVG tracing, and HTML screenshots run locally without spending vision API requests.

dsh plugin --profile web add @anionex/dsh-vision-toolkit

Upstream toolkit: Anionex/agent-vision-toolkit · Project website: agent-vision.anionex.me

Recent updates

• 2026-08-16 · Windows Python: Added Microsoft Store Python support, fixing first-time isolated-runtime setup failures for affected Windows users.
• 2026-08-17 · Free vision upgrade: Switched the built-in no-key service to Gemini 3.7 Flash and fixed Qwen/Gemini bounding-box coordinate order.
• 2026-08-16 · Better free vision: Switched the built-in no-key service to Groq Qwen3.6, improving image understanding without adding setup steps.
• 2026-08-16 · Image paste: Text-only routes now switch to a (Vision Toolkit) variant and keep a workspace path, fixing blocked pastes and images that could not be reused later.
• 2026-08-16 · More shared capacity: Expanded the free service capacity to reduce peak-time 429 responses.
• 2026-08-16 · Real model test: Added a full image-request test in Settings, fixing the false confidence caused by a successful /models request to a model that still cannot process images.

Who it is for

See it in action

Paste an image directly into DSH

Paste an image into the conversation. A text-only model can switch to its Vision Toolkit variant and inspect the image in the context of the user's question.

Screenshot to editable page

Left: the reference screenshot. Right: an editable HTML/CSS result. The result can continue into screenshot rendering and pixel comparison instead of ending as an image description.

Sketch to working interface

Left: a hand-drawn reference. Right: the working interface reconstructed from it.

Turn “looks close” into a verifiable result

The repository includes a reproducible UI-restoration example: the agent renders the reference and implementation, then uses difference regions, a heatmap, and a JSON report to guide the next correction.

Quick start: three steps

1. Install

dsh plugin --profile web add @anionex/dsh-vision-toolkit

You can install it into a Headless Profile too:

dsh plugin --profile headless add @anionex/dsh-vision-toolkit

2. Restart and check it

Restart a running Web Profile, then open Settings → Vision Toolkit. The free provider is already configured; run Test vision model to confirm it is reachable.

The first start prepares an isolated runtime: the plugin prefers a system Python 3.11+; when none is found, it downloads a hash-verified standalone Python (about 35 MB) from a pinned release source on first use. A normal installation does not require an agent-vision-toolkit source checkout or a local path setting.

3. Paste an image and describe the outcome you want

Paste a screenshot into the conversation or place an image in the session workspace, then invoke /vision-skills. For example:

Inspect this screenshot. Explain the error and tell me what to fix first.
Find the login button in the top-right corner, return original pixel coordinates, and make a boxed preview.
Crop this icon and convert it to SVG.
Rebuild the page from reference.png. After each pass, render it and run a pixel diff until the major differences are gone.

Common workflows

Toolbox

The plugin provides 10 tools that can be called independently or composed into a workflow:

Coordinates always use original-image pixels in x1,y1,x2,y2 form, so grounding output can feed directly into cropping, tracing, or later automation.

For a long HTML document, pass fullPage=true. The requested width and height remain the layout viewport, while the resulting PNG covers the complete document and reports pageHeight in CSS pixels.

How it works

The plugin keeps image understanding and deterministic local image processing in one Agent workflow. Expand the flow below for the implementation boundary.

flowchart LR
    Image["Screenshot or local HTML"] --> Skill["vision-skills Skill"]
    Skill --> Agent["Text agent selects a task"]
    Agent --> Vision["Use a vision model when image understanding is needed"]
    Agent --> Local["Run crop, SVG, and pixel work locally"]
    Vision --> Result["Answer, OCR, coordinates"]
    Local --> Artifact["PNG, SVG, heatmap, JSON"]
    Result --> Session["Continue reasoning and acting"]
    Artifact --> Session

Configuration and limits

Built-in free service

The default setup uses:

Base URL: https://vision.anionex.me/v1
Model:    gemini-3.7-flash
API Key:  https://agent-vision.anionex.me (filled automatically)

Requests that still use the previous qwen/qwen3.6-27b model name remain compatible and are routed to the Qwen backend.

This is a shared zero-configuration entry point, not an unlimited private endpoint. Request safeguards include:

These safeguards prevent unusually large requests from monopolizing memory or request time. When shared capacity is reached, the service returns a readable 429 response with Retry-After instead of collapsing into an unexplained model failure.

Existing clients that still send api_key="free" remain compatible.

Bring your own vision model

For higher quotas, private endpoints, or another model, change the provider in Settings → Vision Toolkit and store the API key as a DSH Credential. Settings stores the Credential reference and never reads the saved secret back into the browser.

Step-by-step Groq tutorial: Get a free Groq API key and use Qwen3.6-27B for image understanding. It includes screenshots for account/API-key setup, the exact Vision Toolkit settings, and working cURL and Python examples.

You can also configure a Profile patch:

- id: vision-toolkit
  config:
    provider:
      baseUrl: https://api.example.com/v1
      credential: MY_VISION_KEY
      model: your-vision-model
      protocol: openai

OpenAI Chat Completions-compatible endpoints and Anthropic Messages are supported. The Web Settings panel exposes the full provider, runtime, timeout, image-limit, and image-input-variant configuration.

For a trusted internal endpoint that uses a self-signed certificate or MITM proxy, start the DSH process with VISION_SSL_VERIFY=0. The plugin forwards that value to the isolated Python runtime; certificate verification remains enabled when the variable is unset or has any other value. The false values false, off, no, none, and disabled are also accepted, case-insensitively.

Requirements

• A DeepSeek Harness Web or Headless Profile.
• Node.js ^22.19.0 or >=24.0.0.
• Python 3.11+ is usually not needed in advance: the plugin prefers a system Python and otherwise downloads a pinned standalone Python 3.13 automatically, preparing its own isolated environment. Only that first automatic download needs network access.
• Only vision_html_screenshot requires Chrome, Chromium, or Edge.
• Inputs must be PNG, JPEG, GIF, or WebP files in the session workspace, the platform temporary directory, or an explicitly allowed directory.

Configure the Python runtime

By default the plugin picks a system Python 3.11+, or downloads a standalone Python when none is found; most users never need to configure this section. The rest is for advanced setups where automatic discovery fails, a specific interpreter is required, or an external runtime is used.

The packaged managed runtime creates its own isolated virtual environment. runtime.python selects the Python executable used to bootstrap or refresh that environment; it does not replace the managed environment with the interpreter's global site-packages. Set it when automatic discovery fails or when several Python installations exist. The override is also used by runtime.mode: external.

Python 3.11 or newer is required; the automatically downloaded standalone Python is 3.13.15 and, like a system interpreter, is only used to bootstrap the isolated environment. Without an override, the plugin tries python3 then python on macOS/Linux, and python, py -3, then python3 on Windows, before falling back to the standalone download. A configured value is passed as one executable name or path, not as a shell command with arguments, so use py (not py -3) for the Windows launcher; use an absolute path when you need a specific version.

Configure it in the Profile patch:

- id: vision-toolkit
  config:
    runtime:
      # macOS/Linux system Python
      python: python3
      # Or a project-local environment:
      # python: /absolute/path/to/project/.venv/bin/python
      # Windows venv (forward slashes also work in YAML):
      # python: C:/Users/you/project/.venv/Scripts/python.exe
      # Windows launcher, when its default Python is 3.11+:
      # python: py

For a managed runtime, create the project-local interpreter and point runtime.python at it. The plugin installs the locked dependencies into its own managed cache, so installing the lockfile into this bootstrap environment is optional:

python3 --version                         # must report 3.11 or newer
uv venv .venv --python 3.13

For runtime.mode: external, install the locked dependencies using the runtime/requirements.lock from the DSH Vision Toolkit plugin checkout, then point runtime.agentVisionToolkitPath at a separate exact agent-vision-toolkit snapshot. The packaged vendor/agent-vision-toolkit directory is such a snapshot when it has not been modified:

uv pip install --python .venv/bin/python \
  -r /absolute/path/to/dsh-vision-toolkit/runtime/requirements.lock

- id: vision-toolkit
  config:
    runtime:
      mode: external
      python: /absolute/path/to/dsh-vision-toolkit/.venv/bin/python
      agentVisionToolkitPath: /absolute/path/to/dsh-vision-toolkit/vendor/agent-vision-toolkit

On Windows, use py -3 --version for the version check and .venv\Scripts\python.exe plus runtime\requirements.lock in the corresponding commands:

py -3 --version                         # must report 3.11 or newer
uv venv .venv --python 3.13
# External mode only; use the plugin checkout's absolute lockfile path:
uv pip install --python .venv\Scripts\python.exe -r C:\absolute\path\to\dsh-vision-toolkit\runtime\requirements.lock

Point runtime.python at the same interpreter, save the Profile patch, and restart the Web Profile. Then open Settings → Vision Toolkit: the Runtime panel should show the resolved interpreter and Python version, and Run health check plus Test vision model should complete without the Python-version error. A final smoke test is to place a PNG/JPEG in the session workspace and call vision_glance.

The path fence automatically allows the session workspace and the platform temporary directory. On macOS/Linux the temporary root is /tmp. On Windows it is TEMP, then TMP, with the operating-system fallback if neither is set; model-generated /tmp/... paths are translated to that Windows directory before the normal realpath fence runs. No allowedDirs entry is needed for these platform temporary paths.

Use allowedDirs only for additional trusted input roots outside the workspace and platform temporary directory:

- id: vision-toolkit
  config:
    allowedDirs:
      # macOS/Linux example
      - /srv/vision-inputs
      # Windows example (use this instead on Windows)
      # - D:/vision-inputs

allowedDirs is an input allowlist, not the managed runtime cache. The managed runtime keeps its own files under $DSH_HOME/cache/dsh-vision-toolkit (or ~/.dsh/cache/dsh-vision-toolkit when DSH_HOME is unset); that directory does not need to be added. Environment-variable forms such as $env:TEMP and %TEMP% are not expanded inside allowedDirs, so configure extra roots with real absolute paths.

dsh plugin --profile web update @anionex/dsh-vision-toolkit
dsh plugin --profile web remove @anionex/dsh-vision-toolkit

- id: vision-toolkit
  disabled: true

Plugin updates

In Settings → Vision Toolkit, Check for updates queries the Profile's npm registry. For a direct registry installation, Update and restart installs only the exact version you confirmed, verifies it, and restarts an explicitly opted-in POSIX Web process on a fixed --port. Local/workspace/file/git/URL installs, Windows, dynamic ports, read-only Profiles, and manager-owned processes remain check-only.

The updater revalidates the Profile before mutation, snapshots the original manifest and lockfile, and holds a token-owned cross-process lock. The current Web process exits only after the restart helper confirms that the backup is readable and the lock handoff succeeded. When the Profile was already operational, the replacement must report both the target plugin version and a ready runtime; failed replacements restore the original manifest/lockfile and rebuild dependencies with a frozen lockfile before retrying the previous exact version. If automatic recovery itself fails, the backup and lock are preserved and their paths are written to $DSH_HOME/logs/vision-toolkit-restart.log. Detached restart requires DSH_VISION_TOOLKIT_ALLOW_DETACHED_RESTART=1; unsaved Settings or API-key input blocks installation.

Troubleshooting

Project status and limitations

The current release focuses on screenshot understanding, visual grounding, OCR, asset extraction, UI restoration, and pixel-level verification. It is not a video, audio, or camera-input system and does not automatically click GUI controls. Interactive box editing, remote service clusters, model voting, and cross-session visual caches are also outside the current scope.

Development and community

pnpm install --frozen-lockfile --trust-lockfile
pnpm run verify:portable
pnpm run build
pnpm test
TSX_TSCONFIG_PATH=tsconfig.json pnpm dlx tsx scripts/ui-restoration-example.ts --check

• Read CONTRIBUTING.md before contributing.
• Use GitHub Issues for bugs, focused feature requests, and usage questions; see SUPPORT.md for channel guidance.
• Report vulnerabilities privately through SECURITY.md.
• See CHANGELOG.md for releases and FUNDING.md for sponsorship details.
• Visit upstream agent-vision-toolkit for the general toolkit, cross-agent integrations, and visual-task playbooks.

agent-vision-toolkit was created by Anionex. This repository maintains its native DeepSeek Harness integration.

License

The plugin is available under the MIT License. The packaged upstream snapshot retains its original MIT license in vendor/agent-vision-toolkit/LICENSE.