Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

scripts/skill-creator/run_eval.py

View on GitHub
#!/usr/bin/env python3
"""
run_eval.py — Execute a skill against a test prompt via claude -p subprocess.

Produces in --output-dir:
  outputs/         All files written during the run
  transcript.md    Full execution log
  timing.json      Token count and wall-clock duration
  metrics.json     Tool usage counts
"""

import argparse
import json
import os
import shutil
import subprocess
import sys
import tempfile
import time
from pathlib import Path


def build_parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(
        description="Execute a skill against a test prompt via claude -p",
        formatter_class=argparse.RawDescriptionHelpFormatter,
        epilog=__doc__,
    )
    p.add_argument("--skill-path", required=True, help="Path to the skill directory (contains SKILL.md)")
    p.add_argument("--prompt", required=True, help="Test prompt text to run")
    p.add_argument("--output-dir", required=True, help="Directory to write outputs, transcript, timing, metrics")
    p.add_argument("--model", default=None, help="Claude model for claude -p (default: your configured model)")
    p.add_argument("--no-skill", action="store_true", help="Run without loading the skill (baseline run)")
    p.add_argument("--timeout", type=int, default=300, help="Max seconds to wait for claude -p (default: 300)")
    return p


def check_claude_available() -> None:
    """Verify claude CLI is in PATH. Exit 1 with actionable message if not."""
    if shutil.which("claude") is None:
        print(
            "ERROR: 'claude' CLI not found in PATH.\n"
            "Install with: npm install -g @anthropic-ai/claude-code\n"
            "Verify with: which claude && claude --version",
            file=sys.stderr,
        )
        sys.exit(1)


def prepare_output_dir(output_dir: Path) -> Path:
    """Create output directory structure. Returns outputs/ subdirectory."""
    output_dir.mkdir(parents=True, exist_ok=True)
    outputs = output_dir / "outputs"
    outputs.mkdir(exist_ok=True)
    return outputs


def build_claude_command(
    skill_path: Path,
    prompt: str,
    outputs_dir: Path,
    model: str | None,
    no_skill: bool,
) -> list[str]:
    """Construct the claude -p command with appropriate flags."""
    cmd = [
        "claude",
        "-p",
        prompt,
        *(["--model", model] if model else []),
        "--output-format",
        "json",
    ]

    if not no_skill:
        skill_md = skill_path / "SKILL.md"
        if not skill_md.exists():
            print(f"ERROR: SKILL.md not found at {skill_md}", file=sys.stderr)
            sys.exit(1)
        cmd.extend(["--system-prompt-file", str(skill_md)])

    # Ask claude to write outputs to the outputs directory
    cmd.extend(
        [
            "--working-dir",
            str(outputs_dir),
        ]
    )

    return cmd


def count_tools(transcript_text: str) -> dict:
    """Count tool invocations by type from transcript text."""
    import re

    tool_pattern = re.compile(r'"tool":\s*"([^"]+)"')
    counts: dict[str, int] = {}
    for match in tool_pattern.finditer(transcript_text):
        tool = match.group(1)
        counts[tool] = counts.get(tool, 0) + 1
    return counts


def run_eval(args: argparse.Namespace) -> int:
    check_claude_available()

    skill_path = Path(args.skill_path).resolve()
    output_dir = Path(args.output_dir).resolve()
    outputs_dir = prepare_output_dir(output_dir)

    cmd = build_claude_command(
        skill_path=skill_path,
        prompt=args.prompt,
        outputs_dir=outputs_dir,
        model=args.model,
        no_skill=args.no_skill,
    )

    print(f"Running: {' '.join(cmd[:4])} ...", file=sys.stderr)
    start_time = time.monotonic()

    try:
        result = subprocess.run(
            cmd,
            capture_output=True,
            text=True,
            timeout=args.timeout,
            cwd=str(outputs_dir),
        )
    except subprocess.TimeoutExpired:
        print(f"ERROR: claude -p timed out after {args.timeout}s", file=sys.stderr)
        (output_dir / "transcript.md").write_text(
            f"# Execution Timeout\n\nRun timed out after {args.timeout} seconds.\n"
        )
        _write_timing(output_dir, duration=float(args.timeout), tokens=0, timed_out=True)
        _write_metrics(output_dir, tool_counts={})
        return 1

    duration = time.monotonic() - start_time

    # Write transcript
    transcript_lines = [
        "# Execution Transcript\n",
        f"**Model**: {args.model or 'configured default'}\n",
        f"**Skill loaded**: {not args.no_skill}\n",
        f"**Duration**: {duration:.2f}s\n",
        f"**Exit code**: {result.returncode}\n\n",
        "## stdout\n\n```\n",
        result.stdout or "(empty)",
        "\n```\n\n## stderr\n\n```\n",
        result.stderr or "(empty)",
        "\n```\n",
    ]
    transcript_text = "".join(transcript_lines)
    (output_dir / "transcript.md").write_text(transcript_text)

    # Parse token counts from JSON output if available
    tokens = 0
    try:
        response = json.loads(result.stdout)
        usage = response.get("usage", {})
        tokens = usage.get("input_tokens", 0) + usage.get("output_tokens", 0)
    except (json.JSONDecodeError, AttributeError):
        pass

    _write_timing(output_dir, duration=duration, tokens=tokens, timed_out=False)
    _write_metrics(output_dir, tool_counts=count_tools(result.stdout + result.stderr))

    if result.returncode != 0:
        print(
            f"WARNING: claude -p exited with code {result.returncode}. Check transcript.md for details.",
            file=sys.stderr,
        )
        return result.returncode

    print(f"Eval complete. Outputs: {output_dir}", file=sys.stderr)
    return 0


def _write_timing(output_dir: Path, duration: float, tokens: int, timed_out: bool) -> None:
    timing = {
        "duration_seconds": round(duration, 3),
        "tokens_total": tokens,
        "timed_out": timed_out,
    }
    (output_dir / "timing.json").write_text(json.dumps(timing, indent=2))


def _write_metrics(output_dir: Path, tool_counts: dict) -> None:
    metrics = {
        "tool_usage": tool_counts,
        "total_tool_calls": sum(tool_counts.values()),
    }
    (output_dir / "metrics.json").write_text(json.dumps(metrics, indent=2))


def main() -> int:
    parser = build_parser()
    args = parser.parse_args()
    return run_eval(args)


if __name__ == "__main__":
    sys.exit(main())
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall