Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

references/blind-ab-tests.md

View on GitHub
# Writing blind A/B tests

Use this guide when you compare two agents, prompts, reviewers, or routers and the result decides whether a change ships. Each rule comes from a run in this repo that cost money and still had to be judged by hand, or that nearly reported a wrong answer. The code checks live in `scripts/routing-ab-test.py` (`--pilot-check`, and the `--gate` CEILING/FLOOR verdict). The routing procedure is in `docs/router-ab-runbook.md`.

## 1. Prove the corpus can discriminate first

A test that every arm passes, or every arm fails, judges the corpus, not the arms. fix-strategy-v1 scored 8/8 against 8/8 (`scripts/routing-ab-results/fix-strategy-v1/VERDICT.md`), so it could never have shown a difference.

- Run a pilot on about 20% of cases before the main run, with four arms: the baseline, a degraded control, a second baseline run (the A/A arm), and a reference arm of different quality, such as a stronger model. The reference arm is required: a case the baseline fails discriminates only when another arm passes it, and the degraded control rarely does. Without it, a stable 60%/60%/0% pilot fails as NON-DISCRIMINATING, and guarding every baseline failure leaves the baseline at 100% (CEILING).
- Keep baseline accuracy or recall within 30-70% on the analysis set. Guard cases count toward neither accuracy nor the control gap, so they cannot pad a ceiling baseline into the band. Above 70%, harden the cases. Below 30%, check the grader before you blame the cases.
- Require every case to discriminate: baseline and A/A agree on it, and the control or the reference arm gets the other result. A flip between baseline and A/A is noise, not discrimination. A case every arm passes is a ceiling case; one every arm fails is a floor case. Rework or drop each one, or keep it as a guard outside the analysis set and report it as non-discriminating. Log each guard and its reason in the change log. Guards may be at most 20% of cases, and the analysis set keeps at least 8; past that the pilot fails as TOO-MANY-GUARDS, so harden the corpus instead.
- Gate it: `python3 scripts/routing-ab-test.py --pilot-check --control-arm NAME --aa-arm NAME --extra-arm NAME`, or pass `--per-case FILE` for a review A/B outside the routing harness. Name every arm in the pilot data; name the reference arm with `--extra-arm NAME` and each guard with `--guard-case ID`. Leave the candidate arm out of the pilot data: the check rejects unnamed arms. Per-case values must be JSON `true`/`false`, not strings. On FAIL, rework the cases. Do not start the main run.
- If the main run still ends with both arms at 95% or above (or 5% or below) and fewer than 6 discordant pairs, `--gate` returns CEILING or FLOOR (exit 3). Report that the run judged nothing. Never report it as "no difference".
- For a lower-is-better metric, such as false positives per review, record a case as passing when it shows the good outcome, and declare that direction before the pilot.

In the reviewer-isolation pilot, 6 of 11 cases failed that protocol's per-defect version of this rule (among A1, A2, and C, at least one run misses a seeded defect and at least one catches one). Three replacement cases, all real bugs, hit the ceiling in the re-pilot and were dropped. Expect to lose cases here, and budget for it.

## 2. Build cases from real defects

- Take seeded defects from real bugs in git history: a fix commit names the bug, its pre-fix state is the case, and the fix is the answer key. Of 21 defects in the final reviewer corpus, 18 came from history.
- Add distractors: code that looks wrong but is correct. A finding that matches a distractor is a false positive you can count without judging.
- Add clean cases with no defect, to measure noise findings.
- Write synthetic cases only to fill a gap, and report results with and without them. In the reviewer run, one synthetic plan drove the largest share of the gap.
- Record each case's origin (commit or synthetic) in the key.

## 3. Keep the corpus author blind to the candidate

Freeze the corpus and the answer key before anyone reads the candidate arm's text. An author who has seen the treatment picks defects it is built to catch. Exclude the candidate arm from the pilot, so pilot hardening cannot tune cases toward it.

## 4. Make the arms differ in one thing

- Keep the same model, tools, dispatch text, output format, and verdict vocabulary in every arm. Measure total prompt length and keep each arm within 15% of the baseline.
- Strip anything that tells an arm it is the experiment, such as a source credit or a hook name.
- When a candidate bundles several changes, add an ablation arm for each part. The reviewer run's goldfish arm carried both isolation and a rubric. The no-rubric arm (BR) showed that the rubric caused most of the loss (B minus BR was -11.9 points).
- Give every run a fresh workspace, and delete it afterwards.

## 5. Validate the grader before you trust it

- **Agreement gate.** Hand-label a fresh set (20 to 30 items) before the grader sees it, and require at least 90% agreement. Write the matching rule down before you label. Check every grader revision on a set it has never seen; a rule fitted on the set that exposed it is untested. In the reviewer run, two matcher rounds failed (80.0%, 76.7%) before a third passed (93.3%), and both real-defect graders failed (50%, 75%). FP was therefore counted from distractor matches only, and the verdict said so.
- **A/A control.** Run the baseline twice. The gap between the two runs is your noise. A pairwise judge on an A/A pair should read near 0.5. In the reviewer run, one read 0.333 with n = 9, so treat any pairwise read within about 0.17 of that as noise.
- **Degraded-arm control.** Run an arm built to lose, such as a style-only reviewer or a manifest with stripped descriptions. It must lose by at least 30 points (the reviewer-isolation positive-control bar) and by more than twice the A/A gap. If it does not, the grader or the corpus cannot see quality.
- Spot-check about 5 main-run grader answers against the key.

## 6. Cancel position bias

Pairwise judges favor one slot. Jev favored the second-listed review by about 0.12. Ask every pairwise question in both orders, swapping the state and the option order, and average the two reads. Report both order reads beside the mean. A single-order number is not a result.

## 7. Size the run from measured noise

- Use the A/A gap or the bootstrap band from the pilot to choose repeats. With a pilot band of ±8.2 points, 2 repeats give 8.2/√2 = 5.8, under half of a 15-point keep bar.
- Report the run-to-run spread beside every contrast. A contrast inside one arm's own rep-to-rep spread (14.3 points for one reviewer arm) is not an effect, in either direction.
- For a paired pass/fail metric, use McNemar with at least 6 discordant pairs. Fewer is UNDERPOWERED, not a null.

## 8. Pre-register, then freeze

- Before the main run, write the question, metrics, keep bar, analysis script, and stopping rules. Hash the protocol, corpus, key, and arm files, and store the hashes.
- Log every change after a pilot in a dated change log, with the numbers that drove it, before you run the change.
- Re-verify every hash at the end. A changed gate after the results invalidates the run.

## 9. Audit with an independent recompute

Before you report, recompute the headline numbers from the raw outputs with a separate script, keyed by run id. Grep every prompt and system file for answer-key text, to prove the key did not leak. Re-hash the frozen inputs. In the reviewer run, the recompute matched for every arm and repeat, the key grep found 0 hits, and the re-hash found 0 mismatches.

## Report

Report the verdict against each keep-bar line, the per-case hit table, every control result, grader agreement numbers, both pairwise orders, run-to-run spread, cost, and a threats list with the residual risk of each threat. State what a null result does and does not rule out.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall