forkprobe

69Stars4Forks1Issues0Watchers

ForkProbe's DeepSeek Harness plugin runs the same task in parallel across multiple candidate Skills, generating comparable local HTML reports. Once the user selects the winning result, the Agent continues executing the task using the winning Skill.

Jayden-X-LJayden-X-L
·Jayden-X-L/forkprobe.git·jayden-x-l.github.io/forkprobe
Language
Python
License
MIT
Branch
main
agent-workflowai-agentsclaude-codecodexdeepseek-harnessdsh-pluginpptxresearch-writing

Install

$ dsh plugin --profile web add github:Jayden-X-L/forkprobe

Run the command above in your terminal to install this plugin via the dsh CLI. You can switch Profile in the top-right corner. New to dsh? Read the beginner tutorial

One-Sentence Pitch

The DeepSeek Harness native plugin for ForkProbe. It runs the same task in parallel across multiple candidate Skills, generating a comparable local HTML report that lets you pick a winner after reviewing the actual outputs, then allows the current DSH Agent to continue the task along the winning Skill.

Core Capabilities

  • Register forkprobe_compare tool: After user confirms the shortlist, run the same task in parallel across multiple candidate Skills via DSH subagents, and write results to a local HTML Report
  • Register forkprobe_resume tool: When the Report's wait window times out or the Agent restarts, scan the workspace logs directory to retrieve the user-selected winner
  • Provide AI Judge scoring and summary on a 0-10 scale as a reference, but the final choice is always the winner the user clicks on the Report
  • Candidate subagents are forced to use toolFilter.allow=[], prohibiting any tool calls to prevent file modifications, deletions, or recursive triggering of ForkProbe
  • Report path, verdict logs, and product sidecars all reside in the current DSH workspace, with no escape via output_path to outside the workspace
  • Optional anonymous Winner sharing: Only when user checks the box will task type, candidate Skill names, and final choice be uploaded, without including task input or candidate outputs

Technical Implementation

  • Language: TypeScript (ESM, type: module)
  • Key Dependencies: @deepseek-ai/cordis (Cordis plugin runtime), @deepseek-ai/dsh-tools (tool registration), @deepseek-ai/dsh-subagent (native candidate subagent startup), @deepseek-ai/schemastery (config Schema); plus Python 3 + Jinja2 as local bridge
  • Architecture Pattern: Cordis plugin; cordis.patch.yml inserts forkprobe-dsh node in the host; inject = ['tools', 'subagents'], apply() registers two tools via ctx.tools.register, internally starts candidate subagents via ctx.subagents.start, and uses child_process.spawn to invoke Python scripts for candidate preparation and Report rendering
  • Entry File: dsh-plugin/src/index.ts (exports name = 'forkprobe-dsh', inject, Config, apply)

Use Cases

When you're unsure which Skill to use for the current task and don't want to guess based on Skill descriptions alone: have DSH Agent recommend a set of candidates, confirm after reviewing, then use forkprobe to run in parallel, output a comparable local HTML Report, click "Continue" on the Report to select the winner, and let the Agent continue with the remaining task. Commonly used for small-scale A/B/C comparisons on text tasks like academic polishing, natural language rewriting, and reviewer response drafting.

Prerequisites & Compatibility

DependencyMinimum VersionDescription
DeepSeek Harness0.1.0-rc.6+From peerDependencies.dsh-agent/dsh-llm/dsh-subagent
Node.js>=22.19From engines.node
Python 33.xPlugin invokes scripts/prepare_native_compare.py and scripts/finalize_native_compare.py, must be findable in PATH (default command python3)
Jinja2LatestReport template templates/report.html.j2 rendered by Python
PlatformCross-platform (macOS / Windows / Linux)Provided by Node.js + Python cross-platform combo, no platform restrictions in source
Native ModulesNoneOnly uses Node built-ins node:child_process, node:fs/promises, node:crypto, node:path, node:url

Installation

dsh plugin --profile web add github:Jayden-X-L/forkprobe

Configuration Options

ConfigTypeDescriptionDefault
providerstringProvider name used by candidate subagent; must be a Provider currently registered in DSH and supporting tool filteringspawn
pythonExecutablestringCommand name for invoking Python scripts (modify when python3 is not in PATH or version switching is needed)python3
maxCandidatesinteger (2-8)Maximum number of candidate Skills allowed in one comparison (including baseline); out-of-range throws error5
verdictTimeoutSecondsinteger (30-7200)Maximum time forkprobe_compare waits for user to click Continue on Report; timeout requires using forkprobe_resume to retrieve1800 (30 minutes)
maxDepthinteger (1-8)Maximum recursive depth for candidate subagents when Provider supports nested depth limiting3

FAQ

Q: What problem does this plugin solve?

A: When you're unsure which AI Skill to use for the current task, it runs the same task in parallel with a baseline and multiple candidate Skills, generating a comparable local HTML Report that lets you pick a winner after reviewing actual outputs, then the current DSH Agent continues the task along the winning Skill.

Q: Do I have to review the candidate list before running?

A: Yes. forkprobe_compare requires confirmed=true to execute; otherwise it throws an error requiring the candidate shortlist to be displayed and confirmed first.

Q: Is Python 3 required?

A: Yes. The plugin depends on local Python 3 to execute scripts/prepare_native_compare.py and scripts/finalize_native_compare.py for loading Skills and rendering Reports; it also requires the Jinja2 template engine installed. The Python command can be overridden via pythonExecutable config.

Q: Will candidate runs modify files on my computer?

A: No. Each candidate subagent is set to toolFilter.allow=[] (disallowing any tool calls); candidates only output text, never write files or recursively invoke ForkProbe.

Q: What if I time out without clicking the "Continue" button?

A: Just invoke forkprobe_resume. It scans the latest verdict log in the workspace forkprobe-logs/ directory and returns the user-selected winner output to the current DSH Agent.

Q: How to uninstall?

A: Run dsh plugin --profile web remove forkprobe-dsh (also execute once under headless profile), and Harness will remove the forkprobe_compare and forkprobe_resume tools registered by the plugin.

Q: Are reports and candidate outputs uploaded to the cloud?

A: Not by default. Reports and candidate artifacts stay in the local workspace; only when user checks "anonymous sharing" in the Report will task type, candidate Skill names, and final choice be uploaded to ForkProbe's official Worker, without task input, files, or candidate outputs; set FORKPROBE_TELEMETRY=0 to force disable.

Q: Will it replace Harness's built-in tools?

A: No. The plugin only registers two new tools forkprobe_compare and forkprobe_resume via Cordis; it doesn't replace any built-in services or copy DSH credentials.

Learning Curve

Beginner — After installation, you only need the Agent to recommend candidates, confirm, then invoke forkprobe_compare; user interaction adds just one step of "click Continue on Report"; config options have reasonable defaults and work cross-platform.

Known Issues & Limitations

  • forkprobe_compare requires confirmed to be explicitly true, otherwise throws error; model cannot skip confirmation to automatically trigger runs
  • Candidate count must fall within 2 ≤ skills ≤ maxCandidates (default 5, max 8) range, otherwise error
  • When the local verdict server in Report is unreachable or wait exceeds verdictTimeoutSeconds (default 30 minutes), returns awaiting_verdict status, requiring forkprobe_resume to retrieve user's subsequent choice
  • pythonExecutable defaults to looking for python3; if system only has python, or PATH differs on Windows, manually adjust this setting
  • Anonymous sharing events default to forkprobe-selection-telemetry.workers.dev; on some restricted networks events stay in local outbox, not affecting winner save and Agent continuation
← Back to plugin directory