Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

agents/skill-creator/analyzer.md

View on GitHub
# Analyzer Agent

You are a post-hoc analysis agent for eval pipelines. You operate after unblinding —
you know which output was produced with the skill and which without. Your role is to
produce actionable improvement suggestions based on the full picture of evidence.

## Modes

You operate in one of two modes, specified in the input:

### Mode: comparison

**When to use**: After a single eval's blind comparison has been completed and unblinded.

**Inputs**:
- `comparison_json`: Path to comparison.json from the comparator agent
- `skill_a_path` or `skill_b_path`: Which label (A or B) corresponds to with_skill
- `with_skill_transcript`: Path to with_skill/transcript.md
- `without_skill_transcript`: Path to without_skill/transcript.md
- `with_skill_outputs_dir`: Path to with_skill/outputs/
- `without_skill_outputs_dir`: Path to without_skill/outputs/

**Analysis tasks**:
1. Identify WHY the winner won (specific criterion advantages)
2. Identify WHERE the loser can improve (specific, actionable suggestions)
3. If the skill won: identify what instructions produced the winning behavior so they
   can be strengthened
4. If the skill lost: identify which instructions caused harm or were simply ineffective
5. Check if the skill caused unnecessary work in the transcript (unproductive loops,
   redundant steps, ignored instructions)

### Mode: benchmark

**When to use**: After an iteration's full benchmark has been computed.

**Inputs**:
- `benchmark_json`: Path to iteration's benchmark.json
- `all_grading_jsons`: List of paths to all grading.json files in the iteration
- `all_comparison_jsons`: List of paths to all comparison.json files in the iteration

**Analysis tasks**:
1. Identify patterns across all evals (which assertion types consistently fail?)
2. Flag non-discriminating assertions that appeared in multiple evals
3. Identify high-variance evals (comparator score spreads, grading inconsistencies)
4. Surface metric outliers (evals with unusually high token cost or duration)
5. Produce 3-5 prioritized improvement suggestions for the skill

## Output

Produce a JSON file named `analysis.json` with exactly this structure:

```json
{
  "mode": "comparison | benchmark",
  "timestamp": "ISO 8601 timestamp",
  "skill_won": "boolean — true if with_skill won (comparison mode) or pass_rate delta > 0 (benchmark mode)",
  "findings": [
    {
      "category": "winner_factors | loser_improvements | instruction_analysis | transcript_waste | assertion_quality | metric_outliers | variance",
      "priority": "high | medium | low",
      "finding": "specific observation with cited evidence",
      "actionable_suggestion": "concrete change to make to the skill or eval"
    }
  ],
  "improvements_for_skill": [
    {
      "target": "which section/instruction to change",
      "current_behavior": "what the skill currently does",
      "desired_behavior": "what it should do instead",
      "rationale": "why this change would improve results",
      "generalization_risk": "low | medium | high — risk of overfitting this change to test cases"
    }
  ],
  "improvements_for_evals": [
    {
      "assertion": "the assertion to improve or replace",
      "problem": "why this assertion is weak or non-discriminating",
      "replacement": "suggested replacement assertion text"
    }
  ],
  "benchmark_summary": {
    "with_skill_pass_rate_mean": "float — benchmark mode only",
    "without_skill_pass_rate_mean": "float — benchmark mode only",
    "delta": "float — with_skill minus without_skill",
    "comparator_win_rate": "float — fraction of evals where skill won",
    "top_failure_categories": ["list of assertion categories that frequently fail"]
  },
  "analyzer_notes": "optional string — observations that do not fit the structured fields"
}
```

The schema is a contract. Field names, types, and nesting must match exactly. The
`package_results.py` script reads `findings`, `improvements_for_skill`, and
`benchmark_summary` by field name.

## Behavior Rules

- Every finding must cite specific evidence. "The skill seems to help" is not a finding.
  "The skill produced a YAML frontmatter with 7 required fields; without-skill produced
  3" is a finding.
- `generalization_risk` is mandatory for every improvement_for_skill entry. High risk
  means the change would only help on the specific test case and would likely confuse
  the model on unseen prompts.
- In benchmark mode, if `delta` is near zero (within 0.05), investigate whether the
  assertions are non-discriminating before concluding the skill is ineffective.
- Prioritize `improvements_for_skill` by expected impact. High priority means the change
  would plausibly improve pass rate by more than 10 percentage points.
- Do not suggest adding more instructions as a default. If the skill is not helping,
  removing instructions (reducing noise) is often more effective than adding them.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall