# dsh-vision-toolkit

> Give vision to plain text models in DSH: paste images for Q&A, element localization, long-image OCR, UI reconstruction, and pixel comparison.

## Metadata

- Author: [@Anionex](https://github.com/Anionex)
- Repo: <https://github.com/Anionex/dsh-vision-toolkit.git>
- GitHub: [Anionex/dsh-vision-toolkit](https://github.com/Anionex/dsh-vision-toolkit)
- Stars: 613
- Language: TypeScript
- License: [MIT](https://spdx.org/licenses/MIT.html)
- Homepage: <https://agent-vision.anionex.me>
- Topics: `agent-skills`, `agent-vision-toolkit`, `computer-vision`, `deepseek`, `deepseek-harness`, `dsh`, `dsh-plugin`, `gui-automation`, `ocr`, `plugin`, `python`, `screenshot-testing`, `text-only-llm`, `typescript`, `ui-restoration`, `vision-language-model`, `vision-tools`
- Forks: 30
- Open Issues: 5
- Last push: 2026-08-17T16:18:45.000Z
- Added: 2026-08-13T00:00:00.000Z

## Install

```bash
dsh plugin --profile web add github:Anionex/dsh-vision-toolkit
```

## Wiki

## At a Glance
Equips text-only models in DeepSeek Harness with "eyes" so they can directly interpret pasted screenshots and answer questions about image content. Additionally provides local visual processing capabilities including element localization, long screenshot OCR, pixel-level comparison, and UI reconstruction.

## Core Features
- Paste to Use: Paste a screenshot into DSH Web, the text model automatically switches to the `(Vision Toolkit)` variant to understand the image—answering questions, OCR, and multi-image comparison all work out of the box
- Visual Localization: Find specific elements in images by name (e.g., "login button"), return original image pixel coordinates and generate preview images with bounding boxes
- Element Inventory: Enumerate all elements in an image by category (icons, buttons, etc.), return an indexed list with coordinates
- Long Screenshot OCR: Automatically split long screenshots into multiple chunks and merge them into Markdown, supporting resumable uploads and split preview
- Pixel-Level Comparison: Compare reference images with implementation images, output difference ratio, worst area, heatmap, and JSON report
- Local Image Processing: Complete cropping, SVG tracing, foreground matting, dominant color extraction, and local HTML screenshots without internet access

## Technical Implementation
- **Language**: TypeScript (bundles both React client and Node backend)
- **Key Dependencies**: `@deepseek-ai/cordis` (DSH plugin runtime), `@deepseek-ai/dsh-tools` (native Tool registration), `@deepseek-ai/dsh-skill` (Skill injection), `saxes` (SVG parsing)
- **Architecture Pattern**: Mounts into DSH Profile layer as `vision-toolkit` via `cordis.patch.yml`; after startup registers one `vision-tools` Skill and 10 native Tools; Tool calls execute in a packaged `agent-vision-toolkit` Python snapshot via DSH Subprocess, running in an isolated Python virtual environment (`$DSH_HOME/cache/dsh-vision-toolkit`)
- **Entry File**: `src/index.ts` (server-side `apply` hook), frontend extension registered to Web Profile via `src/client/index.tsx`

## Use Cases
Most useful when using text-only models like DeepSeek in DSH for UI reconstruction, screenshot debugging, visual regression testing, and long screenshot transcription. When you need to turn "screenshots" into "evidence the model can read," or want to extract icons from screenshots, extract brand dominant colors, or do local cropping—this plugin avoids repeatedly switching toolchains.

## Prerequisites & Compatibility
| Dependency | Min Version | Description |
|---|---|---|
| DeepSeek Harness | >= 0.1.0-rc.6 | Declared in peerDependencies as `@deepseek-ai/dsh-*` series |
| Node.js | ^22.19.0 \|\| >=24.0.0 | From package.json engines field |
| Python | 3.11+ | In managed mode, plugin automatically prepares isolated venv; Windows launcher use `py` instead of `py -3` |
| Chrome / Chromium / Edge | Latest | Only required for `vision_html_screenshot` tool; other tools unaffected |
| OS | macOS / Windows / Linux | Cross-platform; on Windows `/tmp/...` paths are automatically rewritten to `TEMP`/`TMP` directories |
| Native Modules | None | Only external runtime is isolated Python subprocess; no native pre-install required |

## Installation
```bash
dsh plugin --profile web add github:Anionex/dsh-vision-toolkit
```

## Configuration Options
| Config | Type | Description | Default |
|---|---|---|---|
| provider.baseUrl | string | API address of the vision model service | `https://vision.anionex.me/v1` (built-in free service) |
| provider.credential | string | Credential name storing API Key in DSH; runtime resolves key by reference | `ANIONEX_FREE_VISION` (built-in shared key) |
| provider.model | string | Vision model name | `gemini-3.7-flash` |
| provider.protocol | openai / anthropic | Call protocol | `openai` |
| provider.anthropicThinking | omit / disabled / adaptive | Anthropic thinking field strategy; `omit` means don't touch model default | `omit` |
| provider.userAgent | string | User-Agent for calling vision service and connectivity tests | Chrome 126 desktop UA |
| language | zh / en | Language for vision model output text | `zh` |
| timeoutMs | integer (1000-600000) | Maximum wait milliseconds for a single remote call | `15000` |
| maxImageBytes | integer (1024-268435456) | Byte limit for input images; automatic lossless/lossy compression when exceeded | `4194304` (4 MiB) |
| maxImagePixels | integer (1-268435456) | Decoded pixel limit for input images; automatic downsampling when exceeded | `20000000` |
| concurrency | integer (1-16) | Maximum concurrent vision tool calls within a single session | `4` |
| runtime.mode | managed / external | managed uses packaged snapshot and isolated venv; external uses self-provided agent-vision-toolkit snapshot | `managed` |
| runtime.python | string | Python interpreter used to bootstrap or refresh isolated environment; requires 3.11+ | Auto-detect |
| runtime.agentVisionToolkitPath | string | Path to exact agent-vision-toolkit snapshot in external mode | Not set (not allowed in managed mode) |
| allowedDirs | string array | Additional readable input root directories outside workspace | `[]` |
| imageInputVariants.enabled | boolean | Whether to generate image input variant routes for host-declared text-only models | `true` |
| imageInputVariants.providers | string array | Only inject variants for these provider IDs; empty means all | `[]` |
| imageInputVariants.autoSwitch | boolean | Whether to automatically switch session to corresponding `(Vision Toolkit)` variant when pasting images | `true` |

## FAQ
**Q: Do I need to apply for a vision model API Key myself after installation?**

A: Not by default. The plugin includes a free Gemini 3.7 Flash vision service (vision.anionex.me), ready to paste images and ask questions out of the box; if you need higher quotas or a private endpoint, you can replace it with a self-provided OpenAI/Anthropic compatible service in Settings → Vision Tools.

**Q: I pasted an image in Web, but the model still says it doesn't support image input?**

A: Usually caused by page cache or Profile not switching to the variant with `(Vision Toolkit)` suffix. Restart the Web Profile and refresh the page, confirm the current model has switched to the `(Vision Toolkit)` variant; alternatively, put the image in the session workspace and trigger it via the `/vision-tools` Skill.

**Q: The free vision service returns 429, what should I do?**

A: This is temporary shared capacity exhaustion. Wait for the number of seconds specified in the `Retry-After` header in the error response, then retry; if this happens frequently, consider replacing with a self-provided endpoint in Settings (Groq, self-hosted OpenAI-compatible gateway both work).

**Q: The error says the image is too large or pixel limit exceeded, how to handle?**

A: The error will clearly indicate whether it's the byte limit (default 4 MiB) or pixel limit (default 20 million). First use `vision_crop` or external tools to crop/resize the image before calling the vision tool.

**Q: `vision_html_screenshot` can't find Chrome, what should I do?**

A: Install Chrome, Chromium, or Edge—any one of them works. Only this HTML screenshot tool is affected; the other 9 vision tools work normally.

**Q: Is video, audio, or camera input supported?**

A: Not supported in the current version. This plugin only processes static images (PNG/JPEG/GIF/WebP) and local HTML screenshots; it doesn't automatically click GUI elements, nor does it handle video streams, audio, or camera capture.

**Q: Where does the plugin store its runtime data?**

A: Pasted images are stored in the session workspace under `.dsh-vision-toolkit/`; the plugin's isolated Python virtual environment and runtime cache are stored in `$DSH_HOME/cache/dsh-vision-toolkit` (falls back to `~/.dsh/cache/dsh-vision-toolkit` when DSH_HOME is not set).

**Q: How to uninstall or temporarily disable this plugin?**

A: Uninstall command is `dsh plugin --profile web remove @anionex/dsh-vision-toolkit`; to temporarily disable, set `disabled: true` in the Profile patch, then restart the Profile to take effect.

## Difficulty Level
Advanced — Installation command is ready to use, but for the Python isolated environment, optional Chrome, and API Key credentials to work as expected, you need to understand the relationship between DSH Profile patch, Credential references, and Settings. Regular users can experience core features by simply pasting images with default configuration; no programming required.

## Known Issues & Limitations
- Does not support video, audio, or camera input, and won't automatically click GUI (README.zh.md:393-395)
- Interactive annotation editing, remote service clusters, model voting, and cross-session visual caching are out of scope (README.zh.md:394)
- `vision_html_screenshot` must depend on one of Chrome/Chromium/Edge; when missing, only this tool is unavailable
- When automatic Python detection fails, manual `runtime.python` specification is needed; on Windows, fill in `py`, not `py -3`
- The shared free vision service has capacity limits, and will return `429` during peak times; retry according to `Retry-After`
- Maximum 5 images per request, 4 MiB per image, 20 million pixels per image, 4096 tokens max output (from README configuration section)

---

This document is auto-generated by [deepseek-plugin.org](https://deepseek-plugin.org). HTML page: [dsh-vision-toolkit](https://deepseek-plugin.org/plugins/Anionex/dsh-vision-toolkit)
Wiki generated by AI (model: `MiniMax-M2.5`)