Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

references/skill-creator/artifact-schemas.md

View on GitHub
# Artifact Schemas

JSON contracts for all eval pipeline artifacts. Field names, types, and nesting are
contracts between producers and consumers. Downstream scripts parse by field name —
do not rename fields without updating all consumers.

All paths below are under `skill-workspace/`, a scratch directory outside the repo.
Do not commit these artifacts.

## Producer/Consumer Map

| Schema | Producer | Consumer(s) |
|--------|----------|-------------|
| `evals.json` | Skill creator (human) | `run_eval.py`, grader agent |
| `grading.json` | grader agent | `aggregate_benchmark.py`, analyzer agent |
| `benchmark.json` | `aggregate_benchmark.py` | analyzer agent, `package_results.py` |
| `comparison.json` | comparator agent | analyzer agent |
| `analysis.json` | analyzer agent | `package_results.py`, skill creator |
| `timing.json` | `run_eval.py` | `aggregate_benchmark.py` |
| `metrics.json` | `run_eval.py` | grader agent |
| `eval_metadata.json` | `run_eval.py` | grader agent, comparator agent |
| `trigger-eval.json` | Skill creator (human) | `optimize_description.py` |

---

## evals.json

Location: `skill-workspace/evals/evals.json`

```json
[
  {
    "eval_id": "string — unique identifier for this eval, used as directory name",
    "prompt": "string — the test prompt text passed to claude -p",
    "assertions": [
      "string — one assertion per entry, binary and evidence-checkable"
    ],
    "metadata": {
      "description": "string — optional human-readable description of what this eval tests",
      "tags": ["optional array of tags for filtering"]
    }
  }
]
```

**Rules**:
- `eval_id` must be a valid directory name (kebab-case recommended)
- Each assertion must be binary: it either passes or fails, with evidence
- Assertions should test skill-specific behavior, not generic output properties

---

## grading.json

Location: `skill-workspace/iteration-N/{eval-id}/grading.json`

```json
{
  "eval_id": "string — matches the eval_id from evals.json",
  "configuration": "string — 'with_skill' or 'without_skill'",
  "timestamp": "string — ISO 8601 timestamp",
  "assertions": [
    {
      "assertion": "string — the assertion text from evals.json",
      "verdict": "string — 'PASS' or 'FAIL'",
      "evidence": "string — quoted excerpt or file reference",
      "confidence": "string — 'high', 'medium', or 'low'"
    }
  ],
  "pass_count": "integer",
  "fail_count": "integer",
  "pass_rate": "float — range 0.0 to 1.0",
  "implicit_claims": [
    {
      "claim": "string",
      "verdict": "string — 'VERIFIED', 'UNVERIFIED', or 'CONTRADICTED'",
      "evidence": "string"
    }
  ],
  "eval_critique": {
    "non_discriminating_assertions": ["array of assertion text strings"],
    "recommendation": "string"
  },
  "grader_notes": "string or null"
}
```

**Required fields for `aggregate_benchmark.py`**: `pass_rate`, `pass_count`, `fail_count`

---

## benchmark.json

Location: `skill-workspace/iteration-N/benchmark.json`

```json
{
  "skill_name": "string",
  "workspace": "string — absolute path",
  "timestamp": "string — ISO 8601",
  "eval_count": "integer",
  "with_skill": {
    "pass_rate": {
      "mean": "float",
      "stddev": "float",
      "min": "float",
      "max": "float"
    },
    "tokens": {
      "mean": "float",
      "stddev": "float"
    },
    "time_seconds": {
      "mean": "float",
      "stddev": "float"
    }
  },
  "without_skill": {
    "pass_rate": { "mean": "float", "stddev": "float", "min": "float", "max": "float" },
    "tokens": { "mean": "float", "stddev": "float" },
    "time_seconds": { "mean": "float", "stddev": "float" }
  },
  "delta": {
    "pass_rate": "float or null — with_skill minus without_skill",
    "description": "string — human-readable interpretation"
  },
  "eval_results": [
    {
      "eval_id": "string",
      "configuration": "string",
      "pass_rate": "float",
      "pass_count": "integer",
      "fail_count": "integer",
      "without_skill_pass_rate": "float or null",
      "with_skill_tokens": "integer",
      "with_skill_duration": "float",
      "without_skill_tokens": "integer",
      "without_skill_duration": "float"
    }
  ]
}
```

**Required fields for analyzer agent**: `with_skill.pass_rate.mean`,
`without_skill.pass_rate.mean`, `delta.pass_rate`

---

## comparison.json

Location: `skill-workspace/iteration-N/{eval-id}/comparison.json`

```json
{
  "eval_id": "string",
  "timestamp": "string — ISO 8601",
  "rubric": [
    {
      "criterion": "string",
      "description": "string",
      "weight": "float — all weights sum to 1.0"
    }
  ],
  "scores": {
    "A": {
      "criteria_scores": [
        {
          "criterion": "string — must match rubric criterion name",
          "score": "integer — 1 to 5",
          "rationale": "string — specific evidence"
        }
      ],
      "total_score": "float — weighted sum normalized to 1-10 scale",
      "assertion_pass_rate": "float or null"
    },
    "B": {
      "criteria_scores": [],
      "total_score": "float",
      "assertion_pass_rate": "float or null"
    }
  },
  "winner": "string — 'A', 'B', or 'tie'",
  "winner_margin": "float — absolute difference in total_score",
  "reasoning": "string — 2-4 sentences with specific criterion references",
  "confidence": "string — 'high', 'medium', or 'low'",
  "comparator_notes": "string or null"
}
```

**Required fields for analyzer agent**: `winner`, `scores.A.total_score`,
`scores.B.total_score`, `reasoning`

---

## analysis.json

Location: `skill-workspace/iteration-N/analysis.json`

```json
{
  "mode": "string — 'comparison' or 'benchmark'",
  "timestamp": "string — ISO 8601",
  "skill_won": "boolean",
  "findings": [
    {
      "category": "string — one of: winner_factors, loser_improvements, instruction_analysis, transcript_waste, assertion_quality, metric_outliers, variance",
      "priority": "string — 'high', 'medium', or 'low'",
      "finding": "string — specific observation with evidence",
      "actionable_suggestion": "string — concrete change"
    }
  ],
  "improvements_for_skill": [
    {
      "target": "string — which section/instruction",
      "current_behavior": "string",
      "desired_behavior": "string",
      "rationale": "string",
      "generalization_risk": "string — 'low', 'medium', or 'high'"
    }
  ],
  "improvements_for_evals": [
    {
      "assertion": "string",
      "problem": "string",
      "replacement": "string"
    }
  ],
  "benchmark_summary": {
    "with_skill_pass_rate_mean": "float or null",
    "without_skill_pass_rate_mean": "float or null",
    "delta": "float or null",
    "comparator_win_rate": "float or null",
    "top_failure_categories": ["array of strings"]
  },
  "analyzer_notes": "string or null"
}
```

**Required fields for `package_results.py`**: `findings`, `improvements_for_skill`,
`benchmark_summary.delta`

---

## timing.json

Location: `skill-workspace/iteration-N/{eval-id}/{configuration}/timing.json`

```json
{
  "duration_seconds": "float — wall-clock seconds for the claude -p run",
  "tokens_total": "integer — sum of input_tokens and output_tokens",
  "timed_out": "boolean — true if the run hit the timeout limit"
}
```

Produced by: `run_eval.py`
Consumed by: `aggregate_benchmark.py`

---

## metrics.json

Location: `skill-workspace/iteration-N/{eval-id}/{configuration}/metrics.json`

```json
{
  "tool_usage": {
    "Read": "integer — number of Read tool calls",
    "Write": "integer",
    "Edit": "integer",
    "Bash": "integer",
    "Grep": "integer",
    "Glob": "integer",
    "Agent": "integer"
  },
  "total_tool_calls": "integer — sum of all tool_usage values"
}
```

Produced by: `run_eval.py`
Consumed by: grader agent (for context about execution behavior)

---

## trigger-eval.json

Location: `skill-workspace/evals/trigger-eval.json`

```json
[
  {
    "query": "string — user prompt to test triggering",
    "should_trigger": "boolean — true if the skill should activate for this query"
  }
]
```

**Conventions**:
- Include 10 should_trigger: true entries (vary directness and phrasing)
- Include 10 should_trigger: false entries (near-miss adjacent domains)
- Use realistic prompts with context, not abstract one-liners
- Test edge cases where the skill competes with adjacent skills

Produced by: Skill creator (human)
Consumed by: `optimize_description.py`
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall