notque/vexjoy-agent / skills/meta/toolkit
toolkit
VerifiedToolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.
references/skill-creator/bundled-components.md
View on GitHub# Skill Creator Bundled Components
## Bundled Agents
The `agents/` directory contains prompts for specialized subagents used by this skill. Read them when you need to spawn the relevant subagent.
- `agents/grader.md` -- Evaluate assertions against outputs with cited evidence
- `agents/comparator.md` -- Blind A/B comparison of two outputs
- `agents/analyzer.md` -- Post-hoc analysis of why one version beat another
## Bundled Scripts
- `scripts/run_eval.py` -- Execute a skill against a test prompt via `claude -p`
- `scripts/aggregate_benchmark.py` -- Compute pass rate statistics across runs
- `scripts/optimize_description.py` -- Train/test description optimization loop
- `scripts/package_results.py` -- Consolidate iteration artifacts into a report
- `scripts/eval_compare.py` -- Generate blind comparison HTML viewer
## Workspace Layout
Organize eval results by iteration. Keep `skill-workspace/` in the session scratchpad, outside the repo. Do not commit it:
```
skill-workspace/
├── evals/evals.json
├── iteration-1/
│ ├── eval-descriptive-name/
│ │ ├── with_skill/outputs/
│ │ ├── without_skill/outputs/
│ │ └── grading.json
│ └── benchmark.json
└── iteration-2/
└── ...
```
## Eval evals.json Format
Save test cases to `evals/evals.json` in the workspace, not in the skill directory. Eval data is throwaway:
```json
{
"skill_name": "example-skill",
"evals": [
{
"id": 1,
"name": "descriptive-name",
"prompt": "The realistic user prompt",
"assertions": []
}
]
}
```
## Description Optimization
After the skill works well, optimize the description for triggering accuracy.
Generate 20 eval queries -- 10 that should trigger, 10 that should not. The should-not queries are most important: near-misses from adjacent domains, not obviously irrelevant queries.
Run the optimization loop:
```bash
python3 ~/.claude/skills/toolkit/scripts/skill-creator/optimize_description.py \
--skill-path path/to/skill \
--eval-set evals/trigger-eval.json \
--max-iterations 5
```
This splits queries 60/40 train/test, evaluates the current description (3 runs per query for reliability), proposes improvements based on failures, and selects the best description by test-set score to avoid overfitting.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |