Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

references/agent-creator/agent-eval-design.md

View on GitHub
# Agent Evaluation Design

> **Scope**: How to design activation evals (does the right agent get picked?) and output evals (does the picked agent do the work well?) for a vexjoy-agent operator. Both kinds of eval are designed at scaffold time, not retrofitted after routing failures appear.
> **Load when**: scaffolding a new agent, debugging a routing miss, or auditing whether an existing agent's description still matches its body.
> **After plan 009**: repo agents are retired, so in this repo the "agent" under test is a skill reference. Activation means the router picks the owning skill and the Reference Loading Table signal loads the reference; output means a main-thread or `general-purpose` run that loaded the reference delivers the work. The same method applies unchanged to a private overlay agent.

---

## Two Evals, One Agent

A working agent passes two independent tests:

| Eval | Question | What it scores | When it runs |
|------|----------|----------------|--------------|
| **Activation eval** | Did the router pick this agent for the right requests? | The `description` + `triggers` fields | At scaffold time and whenever description changes |
| **Output eval** | Did the picked agent deliver the work? | The agent body, references, and gates | After dispatch, on real or synthetic tasks |

A perfect output eval cannot save an agent the router never picks. A perfect activation eval cannot save an agent that picks correctly but produces wrong work. Design both, in this order.

---

## Activation Eval

The router reads `name` and `description` first. The activation eval tests whether those two fields produce the right routing decision across realistic phrasings.

### Three case classes

Every agent needs cases in all three columns. Three each is the floor; five each is comfortable.

| Class | What it is | Purpose |
|-------|------------|---------|
| **should-trigger** | Phrases a real user would say when they want this agent | Confirms the description covers the intended domain |
| **should-not-trigger** | Phrases that look related but belong to a different agent | Confirms the boundary clause excludes off-domain work |
| **near-miss** | Phrases on the edge — same words, different intent | Confirms the description disambiguates rather than triggering on keyword overlap |

### Worked example: `kubernetes` skill, `references/kubernetes-helm.md`

Loading-table signal: *"Kubernetes deployments and Helm charts: manifest authoring, values overrides, release upgrades, RBAC. Not for cluster diagnostics — see kubernetes-troubleshooter."*

| Phrase | Class | Should this agent route? | Why |
|--------|-------|--------------------------|-----|
| "write a helm chart for our redis deployment" | should-trigger | yes | Direct domain match: helm + chart + deployment |
| "override the image tag in values.yaml for staging" | should-trigger | yes | Adjacent term match: values overrides |
| "upgrade the prometheus release to 2.45" | should-trigger | yes | Adjacent term: release upgrade |
| "the pods are crashlooping in production" | should-not-trigger | no — route to `kubernetes-troubleshooter` | Boundary clause excludes diagnostics |
| "explain how kubernetes namespaces work" | should-not-trigger | no — route to a docs/explainer | Domain-adjacent but no authoring task |
| "helm me figure out why my chart isn't installing" | near-miss | yes — keyword "helm" + authoring problem | Resolves to authoring; the install failure is the symptom |
| "kubernetes is helmed by the control plane" | near-miss | no | Keyword "helm" appears but as a verb in unrelated context |

### Recording the cases

Keep the cases in your SCAFFOLD-phase notes. Do not commit them. Format:

```markdown
## Activation eval seeds — {agent-name}

### should-trigger
1. <phrase>
2. <phrase>
3. <phrase>

### should-not-trigger (with redirect target)
1. <phrase> → <other-agent-or-skill>
2. <phrase> → <other-agent-or-skill>
3. <phrase> → <other-agent-or-skill>

### near-miss
1. <phrase> — <should it trigger? why?>
2. <phrase> — <should it trigger? why?>
3. <phrase> — <should it trigger? why?>
```

### Manual pass vs. automated pass

For most agents, a mental routing pass over the cases is enough at scaffold time:

1. Read each phrase.
2. Predict which agent the router selects.
3. Compare against the expected routing.
4. Adjust the description until predictions match expectations.

For load-bearing agents (called many times per day, or guarding destructive actions), back the cases with a script that runs the actual router against each phrase and asserts the selected agent.

### Fixing a failing activation case

| Failure | Fix the description by |
|---------|------------------------|
| should-trigger phrase routes elsewhere | Add the missing adjacent term to the description |
| should-not-trigger phrase routes here | Tighten the boundary clause (`Not for X — see Y`) |
| near-miss routes wrong direction | Replace ambiguous keyword in description with the disambiguating phrase from the case |

After a description change, re-run all cases — fixing one can break another.

---

## Output Eval

Once the right agent is picked, the output eval scores the work it produces. Output evals are heavier than activation evals — design them with fewer cases but richer assertions.

### Dimensions to score

| Dimension | Question | Signal |
|-----------|----------|--------|
| **Task success** | Did the agent produce the requested artifact? | File exists, command exits 0, test passes |
| **Format adherence** | Does the output match the contract (frontmatter, headings, table format)? | Parser passes, schema validates |
| **Tool choice** | Did the agent use the right tools, and only those? | Tool log review; no Edit calls from a Reviewer |
| **Validation steps** | Did the agent run its own gates before claiming completion? | Phase artifacts present, gates produced output |
| **Citation quality** | Are factual claims backed by file paths or tool results? | Spot check: pick three claims, verify the citation |
| **Failure handling** | When a gate fails, does the agent stop and report instead of rationalizing past it? | Inject a synthetic failure; observe the response |

### Case design

Output eval cases come from three sources:

1. **Happy-path tasks** — the obvious requests this agent handles every day
2. **Edge cases** — boundary conditions, missing inputs, conflicting constraints
3. **Regression cases** — every production miss becomes a permanent case

The first two come from scaffold time. The third comes from production misses; turn a recurring miss into a test.

### Train / validation separation

When the description is being optimized for activation, split the cases:

| Set | Purpose | Size |
|-----|---------|------|
| **Train** | Cases used to iterate on the description | 60% |
| **Validation** | Cases held back to catch overfit | 40% |

If the train set passes 100% but the validation set fails, the description is memorizing phrases instead of capturing the domain. Rewrite for generalization.

For output evals, the same split applies when iterating on the agent body, references, or gates.

---

## Quick Authoring Checklist

When scaffolding a new agent:

1. Write the description per `agent-frontmatter-template.md` Description Craft.
2. List 3 should-trigger phrases. The description's adjacent terms cover at least 2.
3. List 2 should-not-trigger phrases with redirect targets. The boundary clause excludes them.
4. List 2 near-miss phrases with verdicts and reasoning.
5. Mental-pass the cases against the description. Fix the description if any case fails.
6. (Output eval) When the agent is load-bearing, try 3 happy-path tasks with observable success criteria.

This is the activation half of Post-Write Checklist item 7 in `agent-frontmatter-template.md`.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall