Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

scripts/skill-creator/aggregate_benchmark.py

View on GitHub
#!/usr/bin/env python3
"""
aggregate_benchmark.py — Compute statistics across eval runs in an iteration workspace.

Reads grading.json from each eval directory. Computes mean, standard deviation, and
delta (with_skill minus without_skill) for pass_rate, time_seconds, and tokens.

Produces:
  {workspace}/benchmark.json   Machine-readable statistics
  {workspace}/benchmark.md     Human-readable summary
"""

import argparse
import json
import math
import sys
from datetime import datetime, timezone
from pathlib import Path


def build_parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(
        description="Aggregate benchmark statistics from eval grading results",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=__doc__,
    )
    p.add_argument("workspace", help="Path to iteration workspace directory (e.g. skill-workspace/iteration-1)")
    p.add_argument("--skill-name", required=True, help="Name of the skill being benchmarked")
    return p


def find_eval_dirs(workspace: Path) -> list[Path]:
    """Find all eval directories that contain grading.json."""
    eval_dirs = []
    for child in sorted(workspace.iterdir()):
        if child.is_dir() and (child / "grading.json").exists():
            eval_dirs.append(child)
    return eval_dirs


def load_grading(eval_dir: Path) -> dict | None:
    """Load grading.json from an eval directory."""
    grading_path = eval_dir / "grading.json"
    try:
        return json.loads(grading_path.read_text())
    except (json.JSONDecodeError, OSError) as e:
        print(f"WARNING: Could not load {grading_path}: {e}", file=sys.stderr)
        return None


def load_timing(eval_dir: Path, configuration: str) -> dict:
    """Load timing.json for a given configuration (with_skill or without_skill)."""
    timing_path = eval_dir / configuration / "timing.json"
    try:
        return json.loads(timing_path.read_text())
    except (json.JSONDecodeError, OSError):
        return {"duration_seconds": 0.0, "tokens_total": 0}


def mean(values: list[float]) -> float:
    if not values:
        return 0.0
    return sum(values) / len(values)


def stddev(values: list[float]) -> float:
    if len(values) < 2:
        return 0.0
    m = mean(values)
    variance = sum((v - m) ** 2 for v in values) / (len(values) - 1)
    return math.sqrt(variance)


def aggregate(workspace: Path, skill_name: str) -> dict:
    eval_dirs = find_eval_dirs(workspace)
    if not eval_dirs:
        print(f"ERROR: No eval directories with grading.json found in {workspace}", file=sys.stderr)
        sys.exit(1)

    with_skill_pass_rates = []
    without_skill_pass_rates = []
    with_skill_tokens = []
    without_skill_tokens = []
    with_skill_durations = []
    without_skill_durations = []

    eval_results = []

    for eval_dir in eval_dirs:
        grading = load_grading(eval_dir)
        if grading is None:
            continue

        config = grading.get("configuration")
        if config not in ("with_skill", "without_skill"):
            print(f"WARNING: {eval_dir.name}/grading.json missing 'configuration' field, skipping", file=sys.stderr)
            continue
        pass_rate = float(grading.get("pass_rate", 0.0))

        with_timing = load_timing(eval_dir, "with_skill")
        without_timing = load_timing(eval_dir, "without_skill")

        if config == "with_skill":
            with_skill_pass_rates.append(pass_rate)
            with_skill_tokens.append(float(with_timing.get("tokens_total", 0)))
            with_skill_durations.append(float(with_timing.get("duration_seconds", 0)))
        else:
            without_skill_pass_rates.append(pass_rate)
            without_skill_tokens.append(float(without_timing.get("tokens_total", 0)))
            without_skill_durations.append(float(without_timing.get("duration_seconds", 0)))

        # Try to load the paired configuration if this is with_skill grading
        # (eval dirs may contain only one grading.json; paired data comes from timing files)
        without_pass_rate = None
        paired_grading_path = eval_dir / "grading_without.json"
        if paired_grading_path.exists():
            try:
                paired = json.loads(paired_grading_path.read_text())
                without_pass_rate = float(paired.get("pass_rate", 0.0))
            except (json.JSONDecodeError, OSError):
                pass

        eval_results.append(
            {
                "eval_id": eval_dir.name,
                "configuration": config,
                "pass_rate": pass_rate,
                "pass_count": grading.get("pass_count", 0),
                "fail_count": grading.get("fail_count", 0),
                "without_skill_pass_rate": without_pass_rate,
                "with_skill_tokens": with_timing.get("tokens_total", 0),
                "with_skill_duration": with_timing.get("duration_seconds", 0),
                "without_skill_tokens": without_timing.get("tokens_total", 0),
                "without_skill_duration": without_timing.get("duration_seconds", 0),
            }
        )

    # Compute aggregates
    ws_mean = mean(with_skill_pass_rates)
    wos_mean = mean(without_skill_pass_rates)
    delta = ws_mean - wos_mean if with_skill_pass_rates and without_skill_pass_rates else None

    benchmark = {
        "skill_name": skill_name,
        "workspace": str(workspace),
        "timestamp": datetime.now(timezone.utc).isoformat(),
        "eval_count": len(eval_results),
        "with_skill": {
            "pass_rate": {
                "mean": round(ws_mean, 4),
                "stddev": round(stddev(with_skill_pass_rates), 4),
                "min": round(min(with_skill_pass_rates), 4) if with_skill_pass_rates else 0.0,
                "max": round(max(with_skill_pass_rates), 4) if with_skill_pass_rates else 0.0,
            },
            "tokens": {
                "mean": round(mean(with_skill_tokens), 1),
                "stddev": round(stddev(with_skill_tokens), 1),
            },
            "time_seconds": {
                "mean": round(mean(with_skill_durations), 2),
                "stddev": round(stddev(with_skill_durations), 2),
            },
        },
        "without_skill": {
            "pass_rate": {
                "mean": round(wos_mean, 4),
                "stddev": round(stddev(without_skill_pass_rates), 4),
                "min": round(min(without_skill_pass_rates), 4) if without_skill_pass_rates else 0.0,
                "max": round(max(without_skill_pass_rates), 4) if without_skill_pass_rates else 0.0,
            },
            "tokens": {
                "mean": round(mean(without_skill_tokens), 1),
                "stddev": round(stddev(without_skill_tokens), 1),
            },
            "time_seconds": {
                "mean": round(mean(without_skill_durations), 2),
                "stddev": round(stddev(without_skill_durations), 2),
            },
        },
        "delta": {
            "pass_rate": round(delta, 4) if delta is not None else None,
            "description": "with_skill minus without_skill; positive means skill helps",
        },
        "eval_results": eval_results,
    }

    return benchmark


def render_markdown(benchmark: dict) -> str:
    ws = benchmark["with_skill"]
    wos = benchmark["without_skill"]
    delta = benchmark["delta"]["pass_rate"]
    delta_str = f"+{delta:.1%}" if delta is not None and delta > 0 else (f"{delta:.1%}" if delta is not None else "N/A")

    lines = [
        f"# Benchmark: {benchmark['skill_name']}\n",
        f"**Generated**: {benchmark['timestamp']}  \n",
        f"**Evals**: {benchmark['eval_count']}\n\n",
        "## Pass Rate\n\n",
        "| Configuration | Mean | StdDev | Min | Max |\n",
        "|--------------|------|--------|-----|-----|\n",
        f"| with_skill   | {ws['pass_rate']['mean']:.1%} | {ws['pass_rate']['stddev']:.1%} | {ws['pass_rate']['min']:.1%} | {ws['pass_rate']['max']:.1%} |\n",
        f"| without_skill | {wos['pass_rate']['mean']:.1%} | {wos['pass_rate']['stddev']:.1%} | {wos['pass_rate']['min']:.1%} | {wos['pass_rate']['max']:.1%} |\n",
        f"| **delta** | **{delta_str}** | — | — | — |\n\n",
        "## Token Usage\n\n",
        "| Configuration | Mean Tokens | StdDev |\n",
        "|--------------|-------------|--------|\n",
        f"| with_skill   | {ws['tokens']['mean']:.0f} | {ws['tokens']['stddev']:.0f} |\n",
        f"| without_skill | {wos['tokens']['mean']:.0f} | {wos['tokens']['stddev']:.0f} |\n\n",
        "## Duration (seconds)\n\n",
        "| Configuration | Mean | StdDev |\n",
        "|--------------|------|--------|\n",
        f"| with_skill   | {ws['time_seconds']['mean']:.1f}s | {ws['time_seconds']['stddev']:.1f}s |\n",
        f"| without_skill | {wos['time_seconds']['mean']:.1f}s | {wos['time_seconds']['stddev']:.1f}s |\n\n",
        "## Per-Eval Results\n\n",
        "| Eval | Config | Pass Rate | Pass | Fail |\n",
        "|------|--------|-----------|------|------|\n",
    ]

    for er in benchmark["eval_results"]:
        lines.append(
            f"| {er['eval_id']} | {er['configuration']} | {er['pass_rate']:.1%} | {er['pass_count']} | {er['fail_count']} |\n"
        )

    return "".join(lines)


def main() -> int:
    parser = build_parser()
    args = parser.parse_args()
    workspace = Path(args.workspace).resolve()

    if not workspace.exists():
        print(f"ERROR: Workspace directory does not exist: {workspace}", file=sys.stderr)
        return 1

    benchmark = aggregate(workspace, args.skill_name)

    benchmark_json = workspace / "benchmark.json"
    benchmark_json.write_text(json.dumps(benchmark, indent=2))
    print(f"Written: {benchmark_json}", file=sys.stderr)

    benchmark_md = workspace / "benchmark.md"
    benchmark_md.write_text(render_markdown(benchmark))
    print(f"Written: {benchmark_md}", file=sys.stderr)

    delta = benchmark["delta"]["pass_rate"]
    if delta is not None:
        sign = "+" if delta > 0 else ""
        print(f"Pass rate delta: {sign}{delta:.1%} (with_skill vs without_skill)")
    else:
        print("Pass rate delta: N/A (missing one or both configurations)")

    return 0


if __name__ == "__main__":
    sys.exit(main())
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall