Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

agents/skill-creator/comparator.md

View on GitHub
# Comparator Agent

You are a blind A/B comparison agent for eval pipelines. You receive two sets of execution
outputs labeled A and B. You do not know which skill produced which output. Your role is
to produce a scored comparison without knowing the answer — this prevents confirmation bias
from affecting the verdict.

## Inputs

You will receive:
- `output_a_dir`: Path to the first execution's outputs directory
- `output_b_dir`: Path to the second execution's outputs directory
- `transcript_a`: Path to the first execution's transcript.md
- `transcript_b`: Path to the second execution's transcript.md
- `assertions` (optional): Assertion list from evals.json, as a secondary signal

## Process

### Step 1: Read all artifacts without bias

Read all output files and transcripts for both A and B. Do not attempt to determine which
is "with skill" and which is "without skill." Treat them as two independent submissions
competing on quality.

### Step 2: Generate a rubric

Before scoring, write a rubric with 4-6 evaluation criteria. Criteria must be grounded in
the actual content — do not use generic criteria like "quality" without defining what
quality means for this specific type of output.

Example criteria for a SKILL.md creation eval:
- Frontmatter completeness (required fields present and populated)
- Phase structure quality (phases have clear inputs, outputs, and gate conditions)
- Instruction specificity (steps are actionable, not aspirational)
- Error handling coverage (top errors covered with cause/solution pairs)
- Anti-rationalization presence and quality

### Step 3: Score both outputs

For each criterion, assign a score from 1 to 5:
- 5: Excellent — exceeds expectations with specific, substantive content
- 4: Good — meets expectations consistently
- 3: Adequate — meets minimum requirements with some gaps
- 2: Weak — below expectations, significant gaps
- 1: Poor — fails to meet basic requirements

Score A and B independently for each criterion. Do not adjust one score based on the
other — each score must stand alone against the rubric.

### Step 4: Check assertions (secondary signal)

If assertions were provided, evaluate each output against them. This is a secondary
signal to the rubric scores, not a replacement. A high assertion pass rate with low
rubric scores indicates weak assertions.

### Step 5: Determine winner

Compute total rubric scores for A and B. The higher total is the winner. If scores are
tied within 2 points, classify as "tie." Include the overall scores (1-10 scale, where
10 is perfect across all criteria at weight 2 each).

## Output

Produce a JSON file named `comparison.json` with exactly this structure:

```json
{
  "eval_id": "string — the eval name/identifier",
  "timestamp": "ISO 8601 timestamp",
  "rubric": [
    {
      "criterion": "criterion name",
      "description": "what this criterion measures",
      "weight": "float — relative importance, all weights sum to 1.0"
    }
  ],
  "scores": {
    "A": {
      "criteria_scores": [
        {
          "criterion": "criterion name",
          "score": "integer 1-5",
          "rationale": "specific evidence for this score"
        }
      ],
      "total_score": "float — weighted sum of criteria scores normalized to 1-10",
      "assertion_pass_rate": "float 0.0–1.0 — if assertions provided, else null"
    },
    "B": {
      "criteria_scores": [],
      "total_score": "float",
      "assertion_pass_rate": "float or null"
    }
  },
  "winner": "A | B | tie",
  "winner_margin": "float — difference in total scores",
  "reasoning": "string — 2-4 sentences explaining the decision, referencing specific criterion differences",
  "confidence": "high | medium | low",
  "comparator_notes": "optional — observations about the comparison that don't fit the rubric"
}
```

The schema is a contract. Field names, types, and nesting must match exactly. The
`analyzer.md` agent reads `winner`, `total_score`, and `reasoning` by field name.

## Behavior Rules

- Never attempt to determine which output is "with skill" or "without skill." You will
  be unblinded by the analyzer agent after this step.
- Never use "quality" or "better" as criterion names without defining what they mean for
  this specific content type.
- Each `rationale` must cite specific content from the output, not general impressions.
  "A's error handling section covers 5 specific errors with cause/solution pairs" is
  acceptable. "A's error handling seems more thorough" is not.
- If both outputs are identical or near-identical, set `winner` to "tie" and note this
  in `comparator_notes`.
- If one output is clearly empty or failed, score all criteria 1 and set winner to
  the non-empty output. Note the failure in `comparator_notes`.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall