Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

references/system-upgrade/component-audit-checklists.md

View on GitHub
# Component Audit Checklists Reference

> **Scope**: Per-component-type audit checklists for agents, skills, hooks, and routing tables. Covers what to inspect, detection commands, and stale-pattern signals.
> **Version range**: vexjoy-agent, all versions
> **Generated**: 2026-04-15 — adapt checks to current frontmatter schema

---

## Overview

The AUDIT phase scans affected component types to identify what needs upgrading, deprecating,
or creating. The default scope is the 10 most-recently-modified agents + all hooks + affected
routing tables. Comprehensive audit (all 120+ components) is opt-in. The most common failure
is reading file names without opening the components — that detects presence, not stale patterns.

---

## Pattern Table: Scope by Change Signal

| Signal Type | Default Audit Scope | Skip |
|-------------|--------------------|----|
| New Claude tool available | All agents with `allowed-tools` list | Skills, hooks without tool use |
| New hook event type | All hooks matching event type | Agents, skills |
| Deprecated API | All components using that API string | Unrelated components |
| Routing trigger added | routing tables + affected agents | Hooks, scripts |
| Model name change | All agents with `model:` frontmatter | Hooks, routing tables |

---

## Agent Audit Checklist

For each agent in scope, check these fields in order:

```bash
# 1. Check model pins against the model policy (opus, fable, or inherit; haiku only read-only; sonnet only exempt)
python3 scripts/validate_component_models.py

# 2. Check allowed-tools for deprecated tools
grep -A 10 "allowed-tools:" agents/*.md | grep "deprecated_tool_name"

# 3. Check routing triggers for gaps (compare against routing table)
grep -A 20 "triggers:" agents/*.md

# 4. Check pairs_with for removed agents/skills
grep -A 10 "pairs_with:" agents/*.md

# 5. Check for outdated entries (patterns already resolved in prior upgrades)
grep -rn "Anti-Pattern" agents/*.md
```

**Per-agent audit fields to verify**:

| Field | Check | Stale Signal |
|-------|-------|-------------|
| `model:` | Is this a current model name? | Model ID no longer in active release |
| `allowed-tools:` | Are all listed tools still available? | Tool removed from Claude Code |
| `routing.triggers` | Does at least one trigger match the user's actual phrasing? | All triggers are technical jargon users never say |
| `routing.pairs_with` | Do all listed components still exist? | Agent or skill in the list no longer in the repo |
| `version:` | Is version incremented after substantive changes? | Version stuck at `1.0.0` across many commits |

---

## Skill Audit Checklist

Skills are invoked via the Skill tool by name. Check:

```bash
# 1. Find skills with no routing triggers (unroutable)
grep -L "triggers:" skills/*/*/SKILL.md

# 2. Find skills referencing removed scripts
grep -rn "python3 scripts/" skills/*/*/SKILL.md | while read match; do
  script=$(echo "$match" | grep -oP 'scripts/[^ ]+\.py')
  [ -f "$script" ] || echo "MISSING: $script in $match"
done

# 3. Check phase count matches header count
grep -c "^## Phase\|^### Phase" skills/*/*/SKILL.md
```

**Per-skill audit fields**:

| Field | Check | Stale Signal |
|-------|-------|-------------|
| Phase references | Do all script calls resolve to real files? | `FileNotFoundError` at runtime |
| Agent dispatch | Do dispatched agents still exist? | Agent removed but skill still dispatches it |
| Output contract | Does the output schema match what downstream tools expect? | Silent data loss in pipelines |

---

## Hook Audit Checklist

Hooks are the most fragile components — they fire on every tool call and exit code matters.

```bash
# 1. Find hooks with syntax errors (will kill sessions)
for f in hooks/*.py; do python3 -c "import ast; ast.parse(open('$f').read())" 2>&1 | grep -v "^$" && echo "SYNTAX ERROR: $f"; done

# 2. Find hooks registered in settings but not deployed to ~/.claude/hooks/
python3 -c "
import json, os
s = json.load(open(os.path.expanduser('~/.claude/settings.json')))
for event_hooks in s.get('hooks', {}).values():
    for h in event_hooks:
        p = h.get('command', '')
        if 'hooks/' in p:
            fname = p.split('hooks/')[-1].split()[0]
            if not os.path.exists(os.path.expanduser(f'~/.claude/hooks/{fname}')):
                print(f'NOT DEPLOYED: {fname}')
"

# 3. Find hooks that import missing modules
grep -rn "^from hooks\.\|^import hooks\." hooks/*.py

# 4. Check exit codes are consistent
grep -n "sys.exit" hooks/*.py | grep -v "sys.exit(0)\|sys.exit(1)\|sys.exit(2)"
```

**Hook exit code contract**:

| Exit Code | Meaning | When to Use |
|-----------|---------|-------------|
| `0` | Allow — no message shown to user | Normal pass-through |
| `1` | Block — show stderr to user as warning | Soft gate (user can override) |
| `2` | Block — show stderr as error | Hard gate (cannot override) |

**Critical**: Exit code `2` on every PreToolUse deadlocks the session entirely. Hooks with
syntax errors silently return non-zero. Always verify with `python3 -c "import ast; ast.parse(...)"`.

---

## Routing Index Audit Checklist

Routing lives in each component's frontmatter (`routing.triggers`, `routing.force_route`). `scripts/generate-skill-index.py` and `scripts/generate-agent-index.py` build the gitignored `skills/INDEX.json` and `agents/INDEX.json` from it; the old `skills/meta/do/references/routing-tables.md` was folded into those indexes.

```bash
# 1. Regenerate the indexes
python3 scripts/generate-skill-index.py && python3 scripts/generate-agent-index.py

# 2. Find index entries whose file is gone, and entries with no triggers
python3 scripts/validate-index-integrity.py

# 3. Find agents on disk that the agent index lacks
comm -23 \
  <(ls agents/*.md | xargs -I{} basename {} .md | grep -v '^base-instructions$' | LC_ALL=C sort) \
  <(python3 -c "import json; print('\n'.join(sorted(json.load(open('agents/INDEX.json'))['agents'])))")

# 4. Find indexed skills the routing manifest cannot see
python3 scripts/check-routing-drift.py
```

---

<!-- no-pair-required: section header, not a standalone failure mode block -->
## Pattern Catalog

### Open and Read Component Frontmatter During Audit

**Detection**:
```bash
# Check if AUDIT report cites specific fields (it should, not just filenames)
grep -c "triggers:\|allowed-tools:\|model:" task_plan.md
# Should be > 0 for a real audit
```

**Why this matters**: File existence confirms a component exists; it does not confirm it's stale or
current. Triggers that are never phrased by actual users will never route. A model name frozen
at an old value will use the wrong capability tier.

Do instead: Open and read each component's frontmatter and body. Check specific fields.

---

### Scope Audit to Affected Component Types

**Detection**:
```bash
# Detect if audit report lists all 120+ components for a 2-hook change
wc -l task_plan.md
# An audit for 2 hooks should not produce a 500-line task plan
```

**Why this matters**: Auditing 120+ skills for a single hook event change produces noise and makes
it impossible to distinguish affected from unaffected components. Tier assignment degrades.

Do instead: Scope audit to the component types identified in the Change Manifest signal column.

---

## Error-Fix Mappings

| Symptom | Root Cause | Fix |
|---------|-----------|-----|
| Session deadlocks on every tool call | Hook with exit code 2 deployed to `~/.claude/hooks/` with syntax error | `python3 -c "import ast; ast.parse(...)"` on every hook before deploying |
| Agent dispatch fails silently | `pairs_with` references a removed component | Audit `pairs_with` for all in-scope agents |
| Routing gap — user triggers fall through to default | Routing table references old trigger text | Call the Skill tool with `toolkit`. Do this after agent changes. |
| Script call at runtime raises FileNotFoundError | Skill references moved or deleted script | `find scripts/ -name "*.py"` and reconcile all skill references |

---

## Detection Commands Reference

```bash
# Agents and skills: model pins outside the 2026-11-01 policy
python3 scripts/validate_component_models.py

# Hooks: syntax errors
for f in hooks/*.py; do python3 -m py_compile "$f" 2>&1 | grep -v "^$" && echo "ERROR: $f"; done

# Hooks: undeployed (registered in settings but missing from ~/.claude/hooks/)
# (see full command in Hook Audit Checklist section above)

# Routing index: entries pointing at missing files
# (see full commands in Routing Index Audit Checklist section above)

# Skills: broken script references
grep -rn "python3 scripts/" skills/*/*/SKILL.md
```
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall