Skip to content

Adonis0123/adonis-skills / skills/agent-dispatch

agent-dispatch

Flagged

Choose the lightest agent setup for a coding, review, drafting or design-research task: do it yourself, fan out to 2-3 workers with independent parts across the Claude Code, Codex, Grok Build, Cursor or other CLIs and accounts on this machine, run a scoped architecture loop, or send one brief to independent researchers; pick a cross-model reviewer, test budget and cleanup; track dispatched work, external waits and user decisions on a Run Board. Use when work may need delegation, or the user asks to dispatch, fan out, parallelize, run several agents or accounts, get another AI's review or independent design opinions, or asks how dispatched work is going. Chinese triggers: 派活、并行、要不要多开几个 agent、分给多个 Claude 账号 / codex / grok、找另一个 AI 审、对比优秀设计、多方意见、进度看板、派出的任务进行到哪了。Not for unrelated read-only questions, trivial edits, research with no delegation, a single review of an existing diff (agentic-review-handoff), or the machine-wide session list (sideby sessions). Inside another named workflow, only size its workers.

FileKindSize
GLOSSARY.mdmarkdown2.1 KB
SKILL.mdskill md27.8 KB
evals/README.mdmarkdown7.6 KB
evals/README.zh-CN.mdmarkdown7.5 KB
evals/bench/compare.pyscript5.3 KB
evals/bench/run_bench.pyscript12.3 KB
evals/bench/tasks.jsonconfig6.6 KB
evals/bench/test_bench.pyscript4.4 KB
evals/build_plugin_eval.pyscript11.3 KB
evals/evals.jsonconfig31.4 KB
evals/results.mdmarkdown15.8 KB
evals/test_eval_harness.pyscript3.5 KB
evals/trigger-eval-set.jsonconfig1.6 KB
fixtures/probe-full.jsonconfig1.6 KB
fixtures/probe-single.jsonconfig690 B
fixtures/run-midrun/B9.jsonlother191 B
fixtures/run-midrun/deploy-log.txtmarkdown42 B
fixtures/run-midrun/run.jsonconfig1.8 KB
fixtures/run-sessions/B9.jsonlother191 B
fixtures/run-sessions/deploy-log.txtmarkdown42 B
fixtures/run-sessions/ended.jsonlother95 B
fixtures/run-sessions/run.jsonconfig2.1 KB
fixtures/run-sessions/sessions.jsonconfig804 B
fixtures/session-index.jsonconfig401 B
fixtures/sideby-stub/sessions.jsonconfig1.5 KB
fixtures/sideby-stub/sidebyexecutablescript4.1 KB
references/accounts.mdmarkdown10.4 KB
references/context-budget.mdmarkdown2.9 KB
references/fallbacks.mdmarkdown4.7 KB
references/launch.mdmarkdown16.1 KB
references/measure.mdmarkdown3.3 KB
references/research-brief.mdmarkdown1.3 KB
references/run-board.mdmarkdown48.7 KB
references/task-spec.mdmarkdown5.0 KB
scripts/board.cssother12.6 KB
scripts/board_interaction.jsscript3.7 KB
scripts/board_markdown.jsscript2.9 KB
scripts/board_render.pyscript17.2 KB
scripts/dispatch_board.pyscript95.3 KB
scripts/isolated_case.pyscript1.6 KB
scripts/probe_agents.pyscript12.9 KB
scripts/sideby_cli.pyscript3.2 KB
scripts/sideby_sessions.pyscript3.3 KB
scripts/test_billing_and_quota.pyscript10.9 KB
scripts/test_board_interaction.jsscript2.1 KB
scripts/test_board_markdown.jsscript2.9 KB
scripts/test_board_reload_state.jsscript2.7 KB
scripts/test_bug_regressions.pyscript4.7 KB
scripts/test_dispatch_board.pyscript68.9 KB
scripts/test_harness_limits.pyscript11.7 KB
scripts/test_lane_states.pyscript7.5 KB
scripts/test_probe_agents.pyscript19.2 KB
scripts/test_review_fixes.pyscript6.7 KB
scripts/test_rounds_and_watch.pyscript7.3 KB

evals/README.md

View on GitHub
# agent-dispatch evals

[简体中文](README.zh-CN.md)

Three checks, from cheapest to the one that answers "does dispatching pay off". Run them before changing a tier rule, after upgrading the model or a CLI the skill dispatches to, and change one rule per measured change (`../references/measure.md`). Record every run in [`results.md`](results.md).

| Check                                 | Question it answers                                                                                                            | What it cannot show                                                                               | Cost                                                   |
| ------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------ |
| A. Behavior with vs without the skill | Does loading the skill make the agent pick the right tier and follow the Run rules, compared with no skill?                    | Whether the chosen tier produces better code. Routing accuracy is not an architecture comparison. | 24 cases × 3 runs × 2 arms; about $25-55, 30-40 min    |
| B. Solo vs dispatch outcome benchmark | On real tasks from this repository's history, does an agent using the skill finish as many tasks as one working alone, faster? | Tasks unlike the ones in `bench/tasks.json`, or other models.                                     | 3 tasks × 3 runs × 2 arms; hours, real worker accounts |
| Trigger eval                          | Does the description make Claude load the skill for the right prompts?                                                         | Anything after loading.                                                                           | 12 prompts × 3 runs                                    |

## A. Behavior with vs without the skill

Cases live in [`evals.json`](evals.json) (prompt, fixtures, assertions). It is the only place to edit them; the suite for `claude plugin eval` is generated:

```bash
python3 skills/agent-dispatch/evals/build_plugin_eval.py
claude plugin eval . --eval-dir evals/agent-dispatch --runs 3 --ablation with-without --scaffold \
  --allow-tools Bash Write Edit --trust-plugin --no-publish -j 4 --judge-model sonnet \
  --output-dir <out> --json <out>/result.json
```

Run from the repository root. The suite lands in `evals/agent-dispatch/` (gitignored; `claude plugin eval` refuses an eval directory inside `skills/`). The with-skill arm loads only this skill; the baseline arm has no access to it, including copies installed in your home directory. Each assertion is its own LLM grader, so a case scores the share of assertions held (`score` / `scoreWithout` / `delta` per case in `result.json`) and the report names the assertion that failed; the judge sees the transcript with middle messages elided, so a step buried in a long run can read as missing. `partial: true` means the run did not finish. A run that ends on an API error (for example "You've hit your session limit") scores 0 without grading and the run log counts them; rerun those cases on another account. `--case` matches one plain `*` glob (no `[..]` or `{..}`), so to rerun several cases copy their generated directories into another gitignored directory and pass it as `--eval-dir`. Cases that use a Run ledger fixture tell the agent the ledger is a static snapshot and a background `watch` only needs to be described: on the copy, files turn 5 minutes old with nothing running, `watch` wakes, and the last message reports that false silence instead of answering the user. Some assertions name ledger commands the baseline cannot know, so part of the delta is "knows the commands", not only "picks better". The sandbox has no `sideby`, so cases that list `fixtures/sideby-stub/sideby` get a stand-in that answers `sessions`, `send` and a few other commands from fixture data in the real `--json` shapes and sends nothing; the agent is told to put it on `PATH`. Ledger and sideby cases are judged on the whole trace, because a correct final answer need not repeat which command it came from. Unit tests for the suite builder and the stub: `python3 -m unittest discover -s skills/agent-dispatch/evals -p 'test_*.py'`.

## B. Solo vs dispatch outcome benchmark

Tasks are real commits of this repository ([`bench/tasks.json`](bench/tasks.json)). Each trial clones the repository at the commit's parent, gives a headless Claude Code session the task's issue-style prompt, then copies the commit's own tests over the result and runs the acceptance commands. Only the outcome is graded.

```bash
B=skills/agent-dispatch/evals/bench
python3 $B/run_bench.py --out <dir outside the repo> --fake reference --runs 1 --arms solo   # every task must PASS
python3 $B/run_bench.py --out <dir outside the repo> --fake noop --runs 1 --arms solo        # every task must FAIL
python3 $B/run_bench.py --out <dir> --runs 3 --jobs 3 \
  --agent-cmd "sideby run {account} --" --accounts claude:001,claude:002,claude:008
python3 $B/run_bench.py --out <dir> ... --skip-done   # resume: run only trials missing from results.jsonl
python3 $B/compare.py <dir>/results.jsonl --md <dir>/report.md
```

- Arms: `solo` (the agent works alone) and `dispatch` (the agent uses `/agent-dispatch` and may start workers on other CLIs and accounts). Both arms of one repetition use the same coordinator account.
- Safety: every trial works in its own clone with the `origin` remote removed and `git push` disallowed; its dispatch ledgers stay in the clone. Workers run with bypassed permissions inside that clone, so run it only on a machine where that is acceptable.
- Verdict (`compare.py`): dispatch holds only if it completes at least as many tasks as solo, no task gets worse, and its median wall time is lower. A task whose arms differ by one run is marked "check transcript" (`<dir>/logs/<task>-<arm>-<rep>.jsonl`, clone in `<dir>/work/`).
- Ceiling effect: when both arms pass every run, the benchmark only measures time and cost, and the review a dispatch tier adds shows up as cost with no visible benefit. Add tasks that a solo run fails at least once before reading the verdict as "dispatch does not pay".
- Adding a task: pick a commit whose tests decide the change, write an issue-style prompt that names every interface the tests use without giving the solution, list the test files as `hidden_tests` and the commands as `accept`, then run both `--fake` checks on it (`--only <id>`). Ten tasks are the target; three are enough to start.
- Unit tests for the harness: `python3 -m unittest discover -s skills/agent-dispatch/evals/bench -p 'test_*.py'`.

## Trigger eval

[`trigger-eval-set.json`](trigger-eval-set.json) lists prompts that should and should not load the skill. Build one case per prompt and run it with `--ablation none` (the default adds a baseline arm that cannot load the skill):

```bash
python3 skills/agent-dispatch/evals/build_plugin_eval.py --trigger
claude plugin eval . --eval-dir evals/agent-dispatch-trigger --runs 3 --ablation none --allow-tools Bash Write Edit \
  --trust-plugin --no-publish -j 4 --output-dir <out> --json <out>/result.json
```

Each case has one free grader: the Skill tool loaded agent-dispatch (`NN-fires`) or never did (`NN-stays-off`, `min: 0` and `max: 0`). The run sees only this copy of the skill, so a copy installed in your home directory cannot answer first, which is why skill-creator's `scripts/run_eval.py` reads every query as "not triggered" on a machine that has the skill installed. Runs stop at 15 turns; a run that starts real work may hit that cap.
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall