Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

references/skill-creator/enrichment-workflow.md

View on GitHub
# Enrichment Workflow

Deep reference for the enrichment loop described in SKILL.md → "Enriching existing skills".
Read this file when executing any phase of the loop.

---

## AUDIT phase

Goal: establish a factual baseline before touching anything.

**Depth inventory** — count what the skill currently has:

```
references/   count files (target: 3+)
scripts/      count files (target: 1+ deterministic tools)
agents/       count files (target: 0–2 bundled subagents)
```

Record counts in `enrichment-workspace/audit.json`:

```json
{
  "skill": "skill-name",
  "audit_date": "YYYY-MM-DD",
  "references_count": 0,
  "scripts_count": 0,
  "agents_count": 0,
  "depth_verdict": "thin | adequate | rich",
  "gaps": ["no pattern catalog", "no before/after examples", "no validation checklist"]
}
```

**Baseline runs** — run the skill against 2–3 prompts that exercise its core domain.
Use `scripts/run_eval.py` with `--skill-path` pointing at the skill directory.
Save each output to `enrichment-workspace/baseline/eval-N/output.md`.

Capture what is missing from baseline outputs: generic advice, no domain idioms,
no concrete examples, no checklists, no error scenarios — these become research targets.

**Depth verdict**:
- `thin`: references < 2 files AND no scripts → enrichment warranted
- `adequate`: 2–4 references OR 1+ scripts → enrichment may help, evaluate carefully
- `rich`: 4+ references AND scripts AND agents → enrichment unlikely to move needle;
  consider description optimization instead

---

## RESEARCH phase

Goal: find knowledge that will change the skill's output behavior — not summaries,
but patterns, checklists, before/after examples, and common mistakes.

**Step 1 — read for gaps**
Read the skill's SKILL.md and all files in its `references/` directory.
List what knowledge would change a model's output if it had it:
- Specific checklists the domain uses
- Before/after examples of correct vs incorrect patterns
- Common mistakes practitioners make
- Decision criteria for choosing between approaches
- Validation rules that are non-obvious

**Step 2 — consult domain-research-targets.md**
Look up the skill's domain in `references/domain-research-targets.md`.
It lists primary sources (highest authority), secondary sources (patterns and
examples), and what to extract from each.

**Step 3 — gather knowledge**
For each source in the domain table:

*Official docs*: Read methodology sections, not API reference. Extract:
- Named patterns with rationale
- Failure modes the docs explicitly warn against
- Decision trees or "when to use X vs Y" guidance

*Secondary sources* (blogs, talks, books): Extract:
- Before/after examples (these are gold — models respond to concrete diffs)
- Common mistake catalogs with explanation of why they are mistakes
- Checklists practitioners actually use

**What to capture** (format for reference files):

```markdown
## Pattern: Name

**When**: [situation where this applies]
**Do this**:
```code
good example
```
**Not this**:
```code
bad example
```
**Why**: [one-sentence rationale]
```

Aim to collect at minimum:
- 5–10 named patterns with before/after examples
- 1 checklist of 8–15 items practitioners use
- 5–10 common mistakes with explanations
- Any domain-specific validation criteria

---

## ENRICH phase

Goal: add research content to the skill in a form that changes behavior at execution time.

**Where content goes**:

| Content type | Target location |
|---|---|
| Pattern catalog with before/after | `references/patterns.md` (new) |
| Checklist practitioners use | `references/checklist.md` (new) |
| Common mistakes | `references/preferred-patterns.md` (new, or add to existing) |
| Validation criteria | `references/validation-criteria.md` (new) |
| Repeatable deterministic operation | `scripts/tool-name.py` (new) |

**Structuring reference files**:
- Lead with the most behaviorally impactful content (checklists, before/after examples)
- Group by use-case phase, not by alphabet
- Keep each file focused — one theme per file is easier for the skill to load selectively
- Include the "why" for each pattern; models generalize reasoning better than rules

**Updating SKILL.md**:
Add exactly one line per new reference file to the existing "## Reference files" section:
```
- `references/patterns.md` — [domain] pattern catalog: N named patterns with before/after examples
```
Do not expand SKILL.md prose. The orchestrator stays lean; depth lives in references.

**Scripts**:
When research reveals a repeatable mechanical operation (e.g., "run these 4 checks
in sequence"), extract it to `scripts/`. Scripts save tokens on every invocation
and ensure consistency. Use argparse, write to stdout, exit non-zero on error.

**The focus test**: before adding any content, ask — would this change what a model
outputs when executing the skill? If yes, add it. If it is background context the
model already has from training, skip it.

---

## TEST phase

Goal: measure whether the enrichment changed output quality.

**Write test prompts** — 2–3 prompts that specifically exercise the domain knowledge
you just added. If you added a pattern catalog for Go error handling, write prompts
that require correct error wrapping. Prompts must be realistic and specific.

**Run A/B eval**:
```bash
# Run enriched skill
python3 skills/meta/toolkit/scripts/skill-creator/run_eval.py \
  --skill-path skills/target-skill \
  --prompt "realistic test prompt" \
  --output enrichment-workspace/iteration-1/with-enrichment/eval-1/

# Run baseline (no skill)
python3 skills/meta/toolkit/scripts/skill-creator/run_eval.py \
  --prompt "realistic test prompt" \
  --output enrichment-workspace/iteration-1/baseline/eval-1/
```

Run both with identical prompts. Save all outputs under `enrichment-workspace/iteration-N/`.

**Workspace structure**:
```
enrichment-workspace/
├── audit.json
├── baseline/
│   ├── eval-1/output.md
│   └── eval-2/output.md
├── iteration-1/
│   ├── with-enrichment/
│   │   ├── eval-1/output.md
│   │   └── eval-2/output.md
│   ├── baseline/
│   │   ├── eval-1/output.md
│   │   └── eval-2/output.md
│   └── comparisons/
│       ├── eval-1-comparison.json
│       └── eval-2-comparison.json
└── iteration-2/
    └── ...
```

---

## EVALUATE phase

Goal: determine objectively whether the enriched version is better.

**Dispatch comparator** for each test prompt pair.
Load `agents/comparator.md` (bundled with skill-creator). Feed it both outputs
labeled "Output A" and "Output B" — do not reveal which is enriched. The comparator:
- Scores both on depth, accuracy, actionability, domain idioms (0–10 each)
- Picks a winner with cited evidence
- Saves to `enrichment-workspace/iteration-N/comparisons/eval-N-comparison.json`

**Decision rule**:
- Enriched wins 2/3 prompts or better → **PUBLISH**
- Tie (1–1 with 2 prompts, or 1–1–1 with 3) → run analyzer, then **RETRY**
- Baseline wins majority → run analyzer, then **RETRY**

**Run analyzer on loss/tie**:
Load `agents/analyzer.md`. Feed it: the comparison results, the enrichment content
added, and the baseline outputs. Ask it to identify specifically what the enriched
version lacked. Common findings:
- Content was added but never referenced in SKILL.md phases (the skill doesn't know to load it)
- Examples were too abstract — model didn't recognize them as patterns to apply
- Research angle was wrong for the prompt type being tested

Record the analyzer's findings in `enrichment-workspace/iteration-N/analysis.md`.
These drive the next research angle.

---

## PUBLISH phase

Goal: commit validated improvements cleanly so they can be reviewed and merged.

**Branch**:
```bash
git checkout -b feat/enrich-{skill-name}
```

**Stage only enrichment artifacts**:
```bash
git add skills/target-skill/references/
git add skills/target-skill/scripts/      # if scripts were added
git add skills/skill-name/SKILL.md        # pointer lines only
```

Do not commit `enrichment-workspace/` — it is ephemeral eval data.

**Commit message**:
```
feat(target-skill): enrich with {domain} patterns and checklist

Added {N} reference files covering {what}: pattern catalog with before/after
examples, practitioner checklist, and common mistake catalog. Enriched version
wins {2 or 3}/3 blind A/B evals against baseline on {domain} prompts.
```

**Push and create PR**:
```bash
git push -u origin feat/enrich-{skill-name}
gh pr create \
  --title "feat(target-skill): enrich with domain knowledge" \
  --body "Enrichment loop result: N reference files added, wins X/3 blind evals."
```

---

## Retry logic in detail

After a failed evaluation, pick the next research angle based on what the analyzer found:

**Iteration 1 — official docs + canonical best practices**
Focus: what the domain's authoritative sources say to do. Patterns and guidelines
from official documentation, language specs, or framework guides. This catches the
most common gap: missing canonical patterns.

**Iteration 2 — common mistakes + failure modes**
Focus: what practitioners actually get wrong. PR review comments, SO questions,
post-mortems, "gotchas" sections in docs. This adds the flip side: what NOT to do
and why. Models often produce safer output when they know the failure modes.

**Iteration 3 — advanced patterns + edge cases**
Focus: what experts know that beginners don't. Performance trade-offs, non-obvious
interactions, when the "standard" pattern breaks down. Only worth pursuing if
iterations 1 and 2 produced improvement but not enough.

**If iteration 3 still fails**:
Do not silently degrade. Report to the user:
- What enrichment was tried (3 research angles)
- What the comparator found lacking in each iteration
- Hypothesis for why the skill is hard to enrich (domain may require runtime context,
  not static reference content)
- Recommendation: accept current state, redesign the eval prompts, or try a
  fundamentally different enrichment approach (e.g., bundled agent instead of reference file)
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall