Skip to content

Adonis0123/adonis-skills / skills/agent-dispatch

agent-dispatch

Flagged

Choose the lightest agent setup for a coding, review, drafting or design-research task: do it yourself, fan out to 2-3 workers with independent parts across the Claude Code, Codex, Grok Build, Cursor or other CLIs and accounts on this machine, run a scoped architecture loop, or send one brief to independent researchers; pick a cross-model reviewer, test budget and cleanup; track dispatched work, external waits and user decisions on a Run Board. Use when work may need delegation, or the user asks to dispatch, fan out, parallelize, run several agents or accounts, get another AI's review or independent design opinions, or asks how dispatched work is going. Chinese triggers: 派活、并行、要不要多开几个 agent、分给多个 Claude 账号 / codex / grok、找另一个 AI 审、对比优秀设计、多方意见、进度看板、派出的任务进行到哪了。Not for unrelated read-only questions, trivial edits, research with no delegation, a single review of an existing diff (agentic-review-handoff), or the machine-wide session list (sideby sessions). Inside another named workflow, only size its workers.

FileKindSize
GLOSSARY.mdmarkdown2.1 KB
SKILL.mdskill md27.8 KB
evals/README.mdmarkdown7.6 KB
evals/README.zh-CN.mdmarkdown7.5 KB
evals/bench/compare.pyscript5.3 KB
evals/bench/run_bench.pyscript12.3 KB
evals/bench/tasks.jsonconfig6.6 KB
evals/bench/test_bench.pyscript4.4 KB
evals/build_plugin_eval.pyscript11.3 KB
evals/evals.jsonconfig31.4 KB
evals/results.mdmarkdown15.8 KB
evals/test_eval_harness.pyscript3.5 KB
evals/trigger-eval-set.jsonconfig1.6 KB
fixtures/probe-full.jsonconfig1.6 KB
fixtures/probe-single.jsonconfig690 B
fixtures/run-midrun/B9.jsonlother191 B
fixtures/run-midrun/deploy-log.txtmarkdown42 B
fixtures/run-midrun/run.jsonconfig1.8 KB
fixtures/run-sessions/B9.jsonlother191 B
fixtures/run-sessions/deploy-log.txtmarkdown42 B
fixtures/run-sessions/ended.jsonlother95 B
fixtures/run-sessions/run.jsonconfig2.1 KB
fixtures/run-sessions/sessions.jsonconfig804 B
fixtures/session-index.jsonconfig401 B
fixtures/sideby-stub/sessions.jsonconfig1.5 KB
fixtures/sideby-stub/sidebyexecutablescript4.1 KB
references/accounts.mdmarkdown10.4 KB
references/context-budget.mdmarkdown2.9 KB
references/fallbacks.mdmarkdown4.7 KB
references/launch.mdmarkdown16.1 KB
references/measure.mdmarkdown3.3 KB
references/research-brief.mdmarkdown1.3 KB
references/run-board.mdmarkdown48.7 KB
references/task-spec.mdmarkdown5.0 KB
scripts/board.cssother12.6 KB
scripts/board_interaction.jsscript3.7 KB
scripts/board_markdown.jsscript2.9 KB
scripts/board_render.pyscript17.2 KB
scripts/dispatch_board.pyscript95.3 KB
scripts/isolated_case.pyscript1.6 KB
scripts/probe_agents.pyscript12.9 KB
scripts/sideby_cli.pyscript3.2 KB
scripts/sideby_sessions.pyscript3.3 KB
scripts/test_billing_and_quota.pyscript10.9 KB
scripts/test_board_interaction.jsscript2.1 KB
scripts/test_board_markdown.jsscript2.9 KB
scripts/test_board_reload_state.jsscript2.7 KB
scripts/test_bug_regressions.pyscript4.7 KB
scripts/test_dispatch_board.pyscript68.9 KB
scripts/test_harness_limits.pyscript11.7 KB
scripts/test_lane_states.pyscript7.5 KB
scripts/test_probe_agents.pyscript19.2 KB
scripts/test_review_fixes.pyscript6.7 KB
scripts/test_rounds_and_watch.pyscript7.3 KB

evals/build_plugin_eval.py

View on GitHub
#!/usr/bin/env python3
"""Build a `claude plugin eval` suite from evals.json.

evals.json is the only place cases are edited. This script regenerates the
suite (one case directory per eval) so `claude plugin eval` can score the skill
with and without it loaded (`--ablation with-without`).

Layout written (default: <repo>/evals/agent-dispatch, outside skills/, because
`claude plugin eval` refuses an eval dir inside a loaded skills/ directory):

    <out>/<NN>-<name>/
        prompt.md            frontmatter (limits, tools) + evaluation note + the user prompt
        case.yaml            plugins: [<this skill dir>], context.scaffold_script
        scaffold.sh          copies ./fixtures into the run's empty workspace (needs --scaffold)
        fixtures/...         the files the eval lists (whole ledger dir when a file sits in one)
        graders/aNN.md       one llm judge per assertion (case score = share held)
        graders/skill-fired.md  tool_used Skill (plugin-fired indicator, not scored in two-arm runs)

With --trigger it instead writes one case per query in trigger-eval-set.json
(default out: <repo>/evals/agent-dispatch-trigger), graded only on whether the
Skill tool loaded this skill (should_trigger) or never did. Run that suite
without --ablation: the run sees only this copy of the skill, so an installed
copy cannot answer first.

Usage:
    python3 skills/agent-dispatch/evals/build_plugin_eval.py [--out DIR] [--only 1,7,22]
    python3 skills/agent-dispatch/evals/build_plugin_eval.py --trigger [--out DIR]
"""

from __future__ import annotations

import argparse
import json
import os
import re
import shutil
import sys
from pathlib import Path

EVALS_DIR = Path(__file__).resolve().parent
SKILL_DIR = EVALS_DIR.parent
REPO_ROOT = SKILL_DIR.parent.parent
DEFAULT_OUT = REPO_ROOT / "evals" / SKILL_DIR.name
CASE_DIR_RE = re.compile(r"^\d{2}-")

# Tools a case may use. Bash/Write/Edit still need `--allow-tools Bash Write Edit` on the command line.
ALLOWED_TOOLS = ["Read", "Glob", "Grep", "Skill", "Agent", "TodoWrite", "Bash", "Write", "Edit"]
MAX_TURNS = 40
TIMEOUT_SECONDS = 900
# A trigger case only needs the first decision; the cap keeps a run that starts real work cheap.
TRIGGER_MAX_TURNS = 15
TRIGGER_TIMEOUT_SECONDS = 300

SCAFFOLD = """#!/usr/bin/env bash
# Generated by build_plugin_eval.py. Copies this case's fixtures into the run's empty workspace,
# so the prompt's relative paths (fixtures/...) resolve from the working directory.
set -euo pipefail
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
if [ -d "$here/fixtures" ]; then
  cp -R "$here/fixtures" ./fixtures
fi
"""


def q(value: str) -> str:
    """A YAML scalar: JSON strings are valid YAML double-quoted scalars."""
    return json.dumps(value, ensure_ascii=False)


def fixture_sources(files: list[str]) -> list[str]:
    """Paths (relative to the skill dir) to copy: listed files, widened to the ledger dir they sit in."""
    out: list[str] = []
    for rel in files:
        parts = Path(rel).parts
        if len(parts) < 2 or parts[0] != "fixtures":
            raise SystemExit(f"fixture path must start with fixtures/: {rel}")
        # A file inside fixtures/<dir>/ belongs to a Run ledger whose run.json points at sibling files.
        src = str(Path(*parts[:2])) if len(parts) > 2 else rel
        if src not in out:
            out.append(src)
    return out


def evaluation_note(files: list[str]) -> str:
    if not files:
        return ""
    lines = ["[评测环境说明] 你的工作目录里有 fixtures/ 目录,本次请求用到的文件:" + "、".join(files) + "。"]
    probes = [f for f in files if Path(f).name.startswith("probe-")]
    if probes:
        lines.append(f"{probes[0]} 是在这台机器上探测可用 agent CLI 和账号得到的输出;"
                     "评测沙箱里看不到这台机器的真实账号,所以把这份文件当作本机的探测结果。")
    # The sandbox has no sideby; the stub answers sessions/send from fixture data so send and its receipt can be shown.
    stubs = sorted({str(Path(f).parent) for f in files if Path(f).name == "sideby"})
    if stubs:
        lines.append(f"{stubs[0]}/sideby 代表这台机器上装好的 sideby:用夹具数据回答 sessions、send 等命令,消息不会真的发到别的会话。"
                     f"Bash 的环境变量不跨命令保留,每条要用 sideby 的命令前都加 `PATH=\"$PWD/{stubs[0]}:$PATH\"`,"
                     "把它当作 sideby 已在 PATH 上。")
    # Without this, a backgrounded watch on the static copy wakes once its copied files turn 5 minutes old,
    # and the run's last message reports that false silence instead of answering the user.
    ledgers = sorted({"/".join(Path(f).parts[:2]) for f in files if Path(f).parts[1:2] and Path(f).parts[1].startswith("run-")})
    if ledgers:
        lines.append(f"{'、'.join(ledgers)} 是 Run 台账的静态快照:背后没有真的在跑的 AI、会话或部署,复制后不会再有新进展,"
                     "文件时间是复制时间而不是真实进度。要挂后台 watch 时,说明你会怎么挂即可,不用真的挂上等它。")
    return "\n".join(lines) + "\n\n---\n\n"


def rubric(case: dict, index: int) -> str:
    """One llm grader per assertion, so a case scores the share of requirements held and the report names
    the one that failed (a single all-or-nothing grader hides both)."""
    files = case.get("files", [])
    # Ledger and sideby cases are judged on the commands run, which a final answer need not repeat.
    focus = "trace" if any(f.startswith("fixtures/run-") or Path(f).name == "sideby" for f in files) else "last_message"
    return f"""---
type: llm
focus: {focus}
---

You are grading how an AI coding agent handled this user request (it may answer in Chinese):

{case["prompt"]}

Judge ONE requirement:

{case["assertions"][index]}

PASS if it holds in what the agent said or did. A requirement about a step (copying a file, running a
command, recording a decision) holds if the agent did it or, when it was only asked for a plan, explicitly
planned it. Judge substance, not wording, language or formatting. A requirement that names a choice (for
example a tier such as L0/L1/L2) holds when the agent made that choice in substance even if it used
different labels. Other requirements of this case are graded separately; ignore them.

Reference for the intended behaviour (use it to interpret the requirement, not as extra requirements):
{case["expected_output"]}

FAIL if the requirement is missing, contradicted, or only hedged without a decision.
"""


SKILL_FIRED = """---
type: tool_used
tool: Skill
input_match: '"skill"\\s*:\\s*"(?:[\\w-]+:)?{name}"'
---
"""


def write_case(case: dict, out: Path, skill_name: str) -> str:
    name = f"{case['id']:02d}-{case['name']}"
    case_dir = out / name
    case_dir.mkdir(parents=True)
    files = case.get("files", [])

    for src in fixture_sources(files):
        source = SKILL_DIR / src
        if not source.exists():
            raise SystemExit(f"case {name}: missing fixture {src}")
        dest = case_dir / src
        dest.parent.mkdir(parents=True, exist_ok=True)
        if source.is_dir():
            shutil.copytree(source, dest)
        else:
            shutil.copy2(source, dest)

    tools = ", ".join(ALLOWED_TOOLS)
    (case_dir / "prompt.md").write_text(
        f"""---
name: {name}
description: {q(f"agent-dispatch eval {case['id']} (generated from skills/{skill_name}/evals/evals.json)")}
tags: [{skill_name}]
expected_outcome: {q(case["expected_output"])}
max_turns: {MAX_TURNS}
timeout_seconds: {TIMEOUT_SECONDS}
allowed_tools: [{tools}]
---

{evaluation_note(files)}{case["prompt"]}
""", encoding="utf-8")

    plugin_rel = os.path.relpath(SKILL_DIR, case_dir)
    case_yaml = [f'schema_version: "1.1"', f"name: {name}", f"plugins: [{q(plugin_rel)}]"]
    if files:
        case_yaml += ["context:", "  scaffold_script: scaffold.sh"]
        scaffold = case_dir / "scaffold.sh"
        scaffold.write_text(SCAFFOLD, encoding="utf-8")
        scaffold.chmod(0o755)
    (case_dir / "case.yaml").write_text("\n".join(case_yaml) + "\n", encoding="utf-8")

    graders = case_dir / "graders"
    graders.mkdir()
    for i in range(len(case["assertions"])):
        (graders / f"a{i + 1:02d}.md").write_text(rubric(case, i), encoding="utf-8")
    (graders / "skill-fired.md").write_text(SKILL_FIRED.replace("{name}", skill_name), encoding="utf-8")
    return name


def write_trigger_case(index: int, item: dict, out: Path, skill_name: str) -> str:
    should = bool(item["should_trigger"])
    name = f"{index:02d}-{'fires' if should else 'stays-off'}"
    case_dir = out / name
    (case_dir / "graders").mkdir(parents=True)
    tools = ", ".join(ALLOWED_TOOLS)
    (case_dir / "prompt.md").write_text(
        f"""---
name: {name}
description: {q(f"agent-dispatch trigger {index} (generated from skills/{skill_name}/evals/trigger-eval-set.json)")}
tags: [{skill_name}, trigger]
expected_outcome: {q(("loads" if should else "does not load") + f" the {skill_name} skill")}
max_turns: {TRIGGER_MAX_TURNS}
timeout_seconds: {TRIGGER_TIMEOUT_SECONDS}
allowed_tools: [{tools}]
---

{item["query"]}
""", encoding="utf-8")
    plugin_rel = os.path.relpath(SKILL_DIR, case_dir)
    (case_dir / "case.yaml").write_text(
        f'schema_version: "1.1"\nname: {name}\nplugins: [{q(plugin_rel)}]\n', encoding="utf-8")
    grader = SKILL_FIRED.replace("{name}", skill_name)
    if not should:
        grader = grader.replace("tool: Skill\n", "tool: Skill\nmin: 0\nmax: 0\n")
    (case_dir / "graders" / "skill-fired.md").write_text(grader, encoding="utf-8")
    return name


def main() -> int:
    parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument("--out", type=Path, default=DEFAULT_OUT, help=f"suite directory (default: {DEFAULT_OUT.relative_to(REPO_ROOT)})")
    parser.add_argument("--only", default="", help="comma-separated eval ids to generate (default: all)")
    parser.add_argument("--trigger", action="store_true", help="build the trigger suite from trigger-eval-set.json")
    args = parser.parse_args()
    if args.trigger and args.out == DEFAULT_OUT:
        args.out = DEFAULT_OUT.with_name(DEFAULT_OUT.name + "-trigger")

    data = json.loads((EVALS_DIR / "evals.json").read_text(encoding="utf-8"))
    skill_name = data.get("skill_name") or SKILL_DIR.name
    only = {int(x) for x in args.only.split(",") if x.strip()}
    cases = [c for c in data["evals"] if not only or c["id"] in only]

    out = args.out.resolve()
    if out == SKILL_DIR or SKILL_DIR in out.parents:
        print("error: --out must be outside the skill directory (plugin eval rejects an eval dir under skills/)", file=sys.stderr)
        return 1
    out.mkdir(parents=True, exist_ok=True)
    # Replace only generated case dirs; leave results/ and anything else untouched.
    for child in out.iterdir():
        if child.is_dir() and CASE_DIR_RE.match(child.name):
            shutil.rmtree(child)

    if args.trigger:
        items = json.loads((EVALS_DIR / "trigger-eval-set.json").read_text(encoding="utf-8"))
        names = [write_trigger_case(i, item, out, skill_name) for i, item in enumerate(items, 1)]
    else:
        names = [write_case(c, out, skill_name) for c in cases]
    print(f"wrote {len(names)} case(s) to {out}")
    return 0


if __name__ == "__main__":
    sys.exit(main())
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall