Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

references/agent-comparison/optimize-phase.md

View on GitHub
# Agent Comparison — Phase 5: OPTIMIZE (Autoresearch)

## Overview

Phase 5 runs an automated optimization loop that improves a markdown target's frontmatter `description` using trigger-rate eval tasks, then selects the best measured variants through beam search or single-path search.

This phase is for routing/trigger optimization, not full code-generation benchmarking. Invoke it when the user says "optimize this skill", "optimize the description", or "run autoresearch". The existing manual A/B comparison (Phases 1-4) remains the path for full agent benchmarking.

---

## Step 1: Validate Optimization Target and Goal

Confirm the target file exists, has YAML frontmatter with a `description`, and the optimization goal is clear:

```bash
# Target must be a markdown file with frontmatter description
test -f skills/{target}/SKILL.md
rg -n '^description:' skills/{target}/SKILL.md

# Goal should be specific and measurable
# Good: "improve error handling instructions"
# Bad: "make it better"
```

---

## Step 2: Prepare Trigger-Rate Eval Tasks

```bash
python3 skills/meta/toolkit/scripts/agent-comparison/optimize_loop.py \
    --target skills/{target}/SKILL.md \
    --goal "{optimization goal}" \
    --benchmark-tasks skills/meta/toolkit/references/agent-comparison/optimization-tasks.example.json \
    --train-split 0.6 \
    --verbose
```

Supported task schemas:
- Flat `tasks` list with optional `"split": "train" | "test"` per task
- Top-level `train` and `test` arrays

Every task must include:
- `query`: the routing prompt to test
- `should_trigger`: whether the target should trigger for that prompt

If no split markers are present, the loop does a reproducible random split with seed `42`.

---

## Step 3: Run Baseline Evaluation

The loop automatically evaluates the unmodified target against the train set before starting iteration. This establishes the score to beat, and records a held-out baseline if test tasks exist.

---

## Step 4: Enter Optimization Loop

The `optimize_loop.py` script handles the full loop:
- Calls `generate_variant.py` to propose a new frontmatter `description` through `claude -p`
- Evaluates each variant against train tasks
- Runs either:
  - single-path hill climbing: `--beam-width 1 --candidates-per-parent 1`
  - beam search with top-K retention: keep the best `K` improving candidates each round
- Accepts variants that beat their parent by more than `--min-gain` (default 0.02)
- Rejects variants that don't improve or break hard gates
- Checks held-out test set every `--holdout-check-cadence` rounds for Goodhart divergence
- Stops on convergence (`--revert-streak-limit` rounds without any ACCEPT), Goodhart alarm, or max iterations

```bash
python3 skills/meta/toolkit/scripts/agent-comparison/optimize_loop.py \
    --target skills/{target}/SKILL.md \
    --goal "{optimization goal}" \
    --benchmark-tasks skills/meta/toolkit/references/agent-comparison/optimization-tasks.example.json \
    --max-iterations 20 \
    --min-gain 0.02 \
    --train-split 0.6 \
    --beam-width 3 \
    --candidates-per-parent 2 \
    --revert-streak-limit 8 \
    --holdout-check-cadence 5 \
    --report optimization-report.html \
    --output-dir /tmp/optimize-run/iterations \
    --verbose
```

Omit `--model` to use Claude Code's configured default model, or pass it explicitly if you need a specific override.

The `--report` flag generates a live HTML dashboard that auto-refreshes every 10 seconds, showing a convergence chart, iteration table, and review/export controls.

### Recommended Modes

- Short default optimization: default flags only
- Fast single-path optimization: `--beam-width 1 --candidates-per-parent 1 --max-iterations 3 --revert-streak-limit 3`
- True autoresearch sweep: `--max-iterations 20 --beam-width 3 --candidates-per-parent 2 --revert-streak-limit 20`
- Conservative search with strict keeps: raise `--min-gain` above `0.02`
- Exploratory search that accepts small wins: use `--min-gain 0.0`

### Live Eval Defaults

Live eval defaults are intentionally short:
- one optimization round
- three trigger-eval runs per query
- one trigger-eval worker
- no holdout cadence unless explicitly requested

For real repo skills at `skills/<name>/SKILL.md`, the live evaluator prefers an isolated git worktree so the candidate content is scored at the real skill path. This is the default `--eval-mode auto` behavior and avoids scoring the installed skill instead of the candidate. The registered-skill path also evaluates the current working copy, not just `HEAD`, so local uncommitted edits are measured correctly.

---

## Step 5: Present Results in UI

If you passed `--report optimization-report.html`, open the generated file in a browser. The report shows:
- Progress dashboard (status, baseline vs best, accepted/rejected counts)
- Convergence chart (train solid line, held-out dashed line, baseline dotted)
- Iteration table with verdict, composite score, delta, and change summary
- Expandable inline diffs per iteration (click any row)

---

## Step 6: Review Accepted Snapshots

Not all ACCEPT iterations are real improvements — some may be harness artifacts. The user reviews the accepted iterations as candidate snapshots from the original target:
- Inspect each accepted iteration's diff in the report
- Use "Preview Combined" only as a comparison aid in the UI
- Use "Export Selected" to download a review JSON describing the selected snapshot diff
- In beam mode, review the retained frontier candidates first; they are the strongest candidates from the latest round

---

## Step 7: Apply Selected Improvements to Target File

Apply one reviewed improvement to the original target file.

- If you want the best single accepted variant, use `<output-dir>/best_variant.md`.
- Beam search still writes a single `best_variant.md`: the highest-scoring accepted candidate seen anywhere in the run.
- Choose scope deliberately:
  - `description-only` for routing-trigger work
  - `body-only` for behavioral work on the skill instructions themselves
- If you exported selected diffs, treat that JSON as review material only. It is not auto-applied by the current tooling, and the current workflow does not support merging multiple accepted diffs into a generated patch.

```bash
# Review the best accepted variant before applying
cat /tmp/optimize-run/iterations/best_variant.md | head -20

# Replace the target with the best accepted variant
cp /tmp/optimize-run/iterations/best_variant.md skills/{target}/SKILL.md
```

---

## Step 8: Run Final Evaluation on Full Task Set (Train + Test)

After applying improvements, run a final evaluation on ALL tasks (not just train) to verify the improvements generalize. Use evaluation-only mode by rerunning the optimizer with `--max-iterations 0`, which records the baseline for the current file without generating fresh variants:

```bash
python3 skills/meta/toolkit/scripts/agent-comparison/optimize_loop.py \
  --target skills/{target}/SKILL.md \
  --goal "{same goal}" \
  --benchmark-tasks {full-task-file}.json \
  --max-iterations 0 \
  --report optimization-report.html \
  --output-dir /tmp/optimize-run/final-check \
  --verbose
```

Compare final scores to the baseline to confirm net improvement. In beam mode, the final report and `results.json` also include:
- `beam_width`
- `candidates_per_parent`
- `holdout_check_cadence`
- per-iteration frontier metadata (`selected_for_frontier`, `frontier_rank`, `parent_iteration`)

---

## Step 9: Report the run

State the outcome in the final report: `{target}` improved `{baseline}` → `{best}` over `{iterations}` iterations, `{accepted}/{total}` candidates accepted, stop reason `{reason}`, and the change summaries. `results.json` in the run directory is the durable record.

---

## Gate

Optimization complete. Results reviewed. Cherry-picked improvements applied and verified against full task set. Outcome reported and `results.json` written.

---

## Current Reality Check

The current optimizer is in a solid state for:
- deterministic proof runs
- isolated live evaluation of existing registered skills
- short live optimization of `read-only-ops`, with the accepted description change now applied and validated against `references/read-only-ops-short-tasks.json`
- short live body optimization of `socratic-debugging`, with the accepted instruction-body update now applied and validated against `references/socratic-debugging-body-short-tasks.json`, now producing clean skill-triggered first-turn outputs instead of fallback chatter

One live-harness caveat remains:
- temporary renamed skill copies do not yet show reliable live trigger improvements through the dynamic command alias path

That caveat does not affect deterministic proof runs or live checks against existing registered skills, but it does mean the current system is stronger for optimizing real in-repo skills than arbitrary renamed temp clones.

For body optimization runs, the blind evaluator now rejects responses that:
- never triggered the target skill
- mention blocked skill/tool access
- fall back into generic "I'll guide you directly" behavior

---

## Optional Extensions

These are off by default. Enable explicitly when needed:
- **Multiple Runs**: Run each benchmark 3x to account for variance
- **Blind Evaluation**: Hide agent identity during quality grading
- **Extended Benchmark Suite**: Run additional domain-specific tests
- **Historical Tracking**: Compare against previous benchmark runs
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall