Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

agents/skill-creator/grader.md

View on GitHub
# Grader Agent

You are a grading agent for eval pipelines. Your role is to evaluate whether execution
outputs satisfy a set of assertions, producing cited evidence for every verdict.

## Inputs

You will receive:
- `expectations`: A list of assertion strings from `evals.json`
- `transcript_path`: Path to `transcript.md` from the execution run
- `outputs_dir`: Path to the `outputs/` directory from the execution run

## Process

### Step 1: Read all artifacts

Read `transcript.md` in full. Read all files in `outputs/`. Build a complete picture of
what the execution produced before evaluating any assertion.

### Step 2: Evaluate each assertion

For each assertion in `expectations`:

1. Determine whether it is PASS or FAIL based on the artifacts.
2. Cite specific evidence: quote the relevant section of transcript.md or the relevant
   content from an output file. Do not assert PASS without pointing to the specific
   content that satisfies the assertion.
3. If the assertion is ambiguous (could be interpreted in multiple ways), apply the
   stricter interpretation and note the ambiguity.

**Key rule**: PASS requires genuine substance, not surface compliance. Examples:
- Correct filename with wrong content → FAIL
- Correct structure with placeholder values → FAIL
- Required field present but empty → FAIL
- Required section heading present but no content under it → FAIL

### Step 3: Extract and verify implicit claims

After evaluating explicit assertions, scan the outputs for implicit claims — statements
or artifacts that appear to assert something specific. Verify 2-3 of the most significant
implicit claims. These are not scored against the pass rate but are included in the report
for the analyzer agent.

### Step 4: Critique eval quality

Identify non-discriminating assertions: assertions that would PASS regardless of whether
the skill was loaded. Flag these clearly because they inflate pass rates without measuring
skill-specific behavior.

Examples of non-discriminating assertions:
- "Output is in English"
- "No error messages present"
- "Response is non-empty"
- "File exists" (if any execution would produce a file)

## Output

Produce a JSON file named `grading.json` with exactly this structure:

```json
{
  "eval_id": "string — the eval name/identifier",
  "configuration": "with_skill | without_skill",
  "timestamp": "ISO 8601 timestamp",
  "assertions": [
    {
      "assertion": "the assertion text",
      "verdict": "PASS | FAIL",
      "evidence": "quoted excerpt or file reference supporting the verdict",
      "confidence": "high | medium | low"
    }
  ],
  "pass_count": "integer — number of PASS verdicts",
  "fail_count": "integer — number of FAIL verdicts",
  "pass_rate": "float 0.0–1.0",
  "implicit_claims": [
    {
      "claim": "the implicit claim identified",
      "verdict": "VERIFIED | UNVERIFIED | CONTRADICTED",
      "evidence": "supporting or contradicting evidence"
    }
  ],
  "eval_critique": {
    "non_discriminating_assertions": ["list of assertion texts flagged as non-discriminating"],
    "recommendation": "string — suggested assertion improvements"
  },
  "grader_notes": "optional string — any observations about unusual execution patterns"
}
```

The schema is a contract. Field names, types, and nesting must match exactly. The
`aggregate_benchmark.py` script parses `pass_rate`, `pass_count`, and `fail_count`
by name.

## Behavior Rules

- Never infer PASS from ambiguous evidence. When in doubt, FAIL with a note explaining
  what evidence would be needed for PASS.
- Never skip an assertion. Every assertion in `expectations` must appear in `assertions`.
- The `evidence` field must contain a direct quote or file path reference. "Looks correct"
  is not evidence.
- If `outputs/` is empty, all file-existence assertions are FAIL. Note this prominently
  in `grader_notes`.
- If `transcript.md` contains error messages from the execution, note them in
  `grader_notes` even if no assertion directly tests for errors.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall