Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

references/agent-comparison/optimization-guide.md

View on GitHub
# Autoresearch Optimization Guide

## Scope

The current autoresearch loop supports two optimization scopes:

- `description-only`: mutate the frontmatter `description` and score it with
  trigger-rate eval tasks
- `body-only`: mutate the instruction body and score it with `blind_compare`
  behavioral tasks

This is useful for improving skill routing accuracy and for short, repeatable
instruction-body improvements on real registered skills.

It is not a replacement for the manual agent benchmark workflow in Phases 1-4.
If you want to compare real code-generation quality across benchmark tasks, use
the normal A/B process.

## Supported Targets

- `skills/<name>/SKILL.md`
- Other markdown targets with valid YAML frontmatter and a non-empty
  `description`

The loop rejects targets without parseable frontmatter or without a
`description`, because trigger-rate evaluation depends on the target text that
drives routing.

## Supported Task Formats

Two task families are supported:

### Trigger-rate tasks

Every trigger-rate task must include:

- `query`: the prompt to test
- `should_trigger`: whether the target should trigger for that prompt

Optional fields:

- `name`: label shown in logs and reports
- `split`: `train` or `test`
- `complexity`: used for stratified splitting when `split` is omitted

Flat task list:

```json
{
  "tasks": [
    {
      "name": "positive-1",
      "split": "train",
      "complexity": "complex",
      "query": "write table-driven Go tests with subtests and helper functions",
      "should_trigger": true
    },
    {
      "name": "negative-1",
      "split": "test",
      "complexity": "complex",
      "query": "debug a Kubernetes pod stuck in CrashLoopBackOff",
      "should_trigger": false
    }
  ]
}
```

Explicit train/test sets:

```json
{
  "train": [
    {
      "name": "positive-1",
      "query": "write race-safe Go tests for a worker pool",
      "should_trigger": true
    }
  ],
  "test": [
    {
      "name": "negative-1",
      "query": "optimize a PostgreSQL indexing strategy",
      "should_trigger": false
    }
  ]
}
```

### Blind body-compare tasks

Every blind body-compare task must include:

- `query`: the prompt to test
- `eval_mode: blind_compare`
- `judge`: currently `heuristic_socratic_debugging`

Optional fields:

- `name`: label shown in logs and reports
- `split`: `train` or `test`
- `min_score`: minimum candidate score required for the task to count as passed

Example:

```json
{
  "tasks": [
    {
      "name": "socratic-first-turn",
      "query": "Help me think through this bug. My Python script sometimes returns None instead of a dict when the cache is warm. Please do not solve it for me directly.",
      "eval_mode": "blind_compare",
      "judge": "heuristic_socratic_debugging",
      "min_score": 0.7,
      "split": "train"
    }
  ]
}
```

Within one run, tasks must all belong to the same family. The optimizer rejects
mixed trigger-rate and blind body-compare task sets.

If no split markers are present, the loop performs a reproducible random split
using `--train-split` and seed `42`.

`run_eval.py` now accepts the same common task-file wrappers:

- raw list: `[{"query": "...", "should_trigger": true}]`
- task wrapper: `{"tasks": [...]}`
- query wrapper: `{"queries": [...]}`
- split wrapper: `{"train": [...], "test": [...]}`

## Command

Short default run:

```bash
python3 skills/meta/toolkit/scripts/agent-comparison/optimize_loop.py \
  --target skills/programming/programming/SKILL.md \
  --goal "improve routing precision without losing recall" \
  --benchmark-tasks skills/meta/toolkit/references/agent-comparison/optimization-tasks.example.json \
  --report optimization-report.html \
  --output-dir /tmp/optimize-run/iterations \
  --verbose
```

Longer search:

```bash
python3 skills/meta/toolkit/scripts/agent-comparison/optimize_loop.py \
  --target skills/programming/programming/SKILL.md \
  --goal "improve routing precision without losing recall" \
  --benchmark-tasks skills/meta/toolkit/references/agent-comparison/optimization-tasks.example.json \
  --train-split 0.6 \
  --max-iterations 20 \
  --min-gain 0.02 \
  --beam-width 3 \
  --candidates-per-parent 2 \
  --revert-streak-limit 20 \
  --holdout-check-cadence 5 \
  --report optimization-report.html \
  --output-dir /tmp/optimize-run/iterations \
  --verbose
```

By default this uses Claude Code's configured model via `claude -p`. Pass `--model` only when you want to override that explicitly.

Useful flags:

- `--dry-run`: exercise the loop mechanics without calling Claude Code
- `--report`: write a live HTML report
- `--output-dir`: persist iteration snapshots and `results.json`
- `--eval-mode auto|registered|alias`: choose how live trigger eval is isolated
- `--beam-width`: retain the best K improving candidates per round
- `--candidates-per-parent`: generate multiple sibling variants from each frontier candidate
- `--revert-streak-limit`: stop after N rounds without any ACCEPT candidates
- `--holdout-check-cadence`: evaluate the global best on held-out tasks every N rounds
- `--parallel-eval N`: run behavioral eval tasks in parallel isolated worktrees

Short defaults:

- `--max-iterations 1`
- `--revert-streak-limit 1`
- `--holdout-check-cadence 0`
- trigger eval `--num-workers 1`
- trigger eval `--runs-per-query 1`

Recommended search presets:

- Short proof run:
  - default flags only
- Single-path local search:
  - `--beam-width 1 --candidates-per-parent 1 --max-iterations 3 --revert-streak-limit 3`
- Balanced beam search:
  - `--beam-width 3 --candidates-per-parent 2`
- Aggressive exploration:
  - `--beam-width 5 --candidates-per-parent 3 --min-gain 0.0`

## Live Eval Isolation Modes

`run_eval.py` now has three modes:

- `auto`: default. If the target is a real repo skill at `skills/<name>/SKILL.md`, live eval runs in an isolated git worktree with the candidate content patched into the real path. Otherwise it falls back to alias mode.
- `registered`: force isolated worktree evaluation of a real registered skill.
- `alias`: force legacy dynamic command-file evaluation.

For real registered skills, `auto` is the preferred mode. It prevents the evaluator
from accidentally scoring the installed skill instead of the candidate under test.
It also patches the current working-copy skill content into the isolated worktree,
so local uncommitted edits are evaluated correctly.

## Evaluation Model

The loop follows the ADR-131 structure:

1. Hard gates
2. Weighted composite score
3. Held-out regression checks
4. Frontier retention

### Layer 1: Hard Gates

An iteration is rejected immediately if any of these mechanical validity gates fail:

- `parses`
- `compiles`
- `protected_intact`

For description optimization, `parses` and `protected_intact` are the most
important gates. Protected sections fenced by `DO NOT OPTIMIZE` markers must be
preserved verbatim.

### Layer 2: Composite Score

The loop converts evaluation results into a weighted composite score using the
built-in weights in `optimize_loop.py`. Task accuracy affects the component
dimensions (`correctness`, `error_handling`, `language_idioms`, `testing`,
`efficiency`) without zeroing the entire score. This preserves optimization
signal for incremental improvements when a task set is not yet perfect.

A candidate is accepted only if it beats its parent by more than `--min-gain`.

### Layer 3: Held-Out Regression Check

Every `--holdout-check-cadence` rounds, the current global best variant is
scored on the held-out test set. If held-out performance drops below the
baseline while train performance has improved, the loop raises a Goodhart
alarm and stops.

### Layer 4: Frontier Retention

When beam search is enabled:

- each frontier candidate generates `--candidates-per-parent` siblings
- every sibling is scored independently
- the top `--beam-width` ACCEPT candidates become the next frontier
- `best_variant.md` still tracks the single best candidate seen anywhere in the run

When `--beam-width 1 --candidates-per-parent 1`, the behavior collapses back to
the original single-path optimizer.

## Optimization Scopes

The optimizer supports two mutation scopes:

- `description-only`: replace only the YAML frontmatter `description`
- `body-only`: replace only the markdown body below the frontmatter

`generate_variant.py` reconstructs the full file around the selected scope so
the unchanged parts stay intact. Use `description-only` for routing-trigger
work and `body-only` for behavioral work judged from the skill's actual output.

For body optimization, pair `--optimization-scope body-only` with
`blind_compare` tasks so generation and evaluation are measuring the same
surface area.

## Iteration Artifacts

When `--output-dir` is set, the loop writes:

- `001/variant.md`
- `001/scores.json`
- `001/verdict.json`
- `001/diff.patch`
- `best_variant.md`
- `results.json`

`results.json` also records search metadata such as `beam_width`,
`candidates_per_parent`, and per-iteration frontier selection markers.

When `--report` is set, it also writes a live HTML dashboard showing:

- status, baseline, best score, accepted/rejected counts
- convergence chart
- iteration table with diffs
- review/export controls for accepted snapshot diffs from the original target

## Current Validation Status

What is currently demonstrated:
- deterministic end-to-end improvement runs with readable artifacts
- isolated live optimization for existing registered skills via temporary git worktrees
- blind body-eval runs that require actual skill-trigger evidence before scoring
- score calculations and accept/reject decisions that match the weighted rubric
- short live proof on `skills/process/read-only-ops/SKILL.md` using
  `references/read-only-ops-short-tasks.json`, improving from one failed positive
  to `2/2` live passes after the accepted description update
- short live body optimization on `skills/process/socratic-debugging/SKILL.md` using
  `references/socratic-debugging-body-short-tasks.json`, improving from `7.85`
  to `8.45` after the accepted instruction-body update; the current baseline now
  evaluates cleanly and non-improving body variants are rejected

What remains imperfect:
- live optimization of temporary renamed skill copies still fails to show measured improvement through the dynamic command alias path

So the current tooling is operational for real registered skills and deterministic proof runs, but not yet fully proven for arbitrary temporary renamed clones.

## Short Live Commands

Routing optimization on a real registered skill:

```bash
python3 skills/meta/toolkit/scripts/agent-comparison/optimize_loop.py \
  --target skills/process/read-only-ops/SKILL.md \
  --goal "Improve read-only routing precision for realistic user prompts." \
  --benchmark-tasks skills/meta/toolkit/references/agent-comparison/read-only-ops-short-tasks.json
```

Body optimization on a real registered skill:

```bash
python3 skills/meta/toolkit/scripts/agent-comparison/optimize_loop.py \
  --target skills/process/socratic-debugging/SKILL.md \
  --goal "Improve the first response so it asks exactly one question, avoids direct diagnosis, avoids code examples, and does not add tool-permission preamble." \
  --benchmark-tasks skills/meta/toolkit/references/agent-comparison/socratic-debugging-body-short-tasks.json \
  --optimization-scope body-only
```

The blind body path now fails closed: if the intended skill does not trigger, or
the response falls back into tool-blocked/direct-guidance chatter, the run is
scored as a failure instead of being treated as a weak improvement.

## Choosing Good Eval Tasks

1. Include both positive and negative prompts.
2. Put realistic user phrasing in both train and held-out sets.
3. Keep at least one out-of-domain negative example in held-out.
4. Do not let the eval set collapse into benchmark keywords only.

## Limitations

Current limitations are intentional and documented:

- The loop does not execute full code-generation benchmarks.
- Pattern-based benchmark tasks with `prompt`, `expected_patterns`, and
  `forbidden_patterns` are not supported by `optimize_loop.py`.
- For full agent quality comparisons, continue to use the manual benchmark and
  grading flow in Phases 1-4.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall