Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

scripts/skill-creator/eval_compare.py

View on GitHub
#!/usr/bin/env python3
"""Generate blind A/B comparison HTML from eval workspace data.

Scans workspace, collects output files, runs deterministic checks
(go build, go vet, go test -race where applicable), loads grading
and blind comparison data, injects into compare.html template.
Outputs compare_report.html.

Usage:
    python3 eval_compare.py <workspace_dir>
    python3 eval_compare.py --help
"""

import argparse
import json
import os
import subprocess
import sys
from pathlib import Path


def build_parser() -> argparse.ArgumentParser:
    p = argparse.ArgumentParser(
        description="Generate blind A/B comparison HTML from eval workspace data.",
        epilog="Workspace must contain compare.html template and iteration-*/ directories.",
    )
    p.add_argument("workspace", type=Path, help="Path to the eval workspace directory")
    p.add_argument(
        "--output", type=Path, default=None, help="Output HTML path (default: <workspace>/compare_report.html)"
    )
    return p


def load_json_safe(path: Path) -> dict | None:
    """Load JSON from a file, returning None on any error."""
    try:
        return json.loads(path.read_text(encoding="utf-8"))
    except (json.JSONDecodeError, OSError, UnicodeDecodeError) as e:
        print(f"WARNING: Could not load {path}: {e}", file=sys.stderr)
        return None


def read_text_safe(path: Path) -> str:
    """Read text file with encoding fallback."""
    try:
        return path.read_text(encoding="utf-8", errors="replace")
    except OSError:
        return ""


def find_files(outputs_dir: Path) -> list[str]:
    """List all files relative to outputs dir."""
    files = []
    for root, _, filenames in os.walk(outputs_dir):
        for f in filenames:
            rel = os.path.relpath(Path(root, f), outputs_dir)
            files.append(rel)
    return sorted(files)


def count_go_lines(outputs_dir: Path) -> int:
    """Count total lines across all .go files."""
    total = 0
    for root, _, filenames in os.walk(outputs_dir):
        for f in filenames:
            if f.endswith(".go"):
                content = read_text_safe(Path(root, f))
                total += len(content.splitlines())
    return total


def get_code_preview(outputs_dir: Path, max_lines: int = 60) -> str:
    """Get preview of main .go file content."""
    for root, _, filenames in os.walk(outputs_dir):
        for f in sorted(filenames):
            if f.endswith(".go") and not f.endswith("_test.go"):
                content = read_text_safe(Path(root, f))
                lines = content.splitlines()
                if len(lines) > max_lines:
                    return "\n".join(lines[:max_lines]) + f"\n... ({len(lines) - max_lines} more lines)"
                return content
    return ""


def run_go_check(outputs_dir: Path, cmd: list[str], timeout: int = 30) -> str:
    """Run a go command in the outputs directory, return 'yes'/'no'/'clean'/'issues'."""
    # Find the go module root (prefer directory with go.mod)
    mod_root = None
    go_dirs = []
    for root, _, files in os.walk(outputs_dir):
        if "go.mod" in files:
            mod_root = root
            break
        if any(f.endswith(".go") for f in files):
            go_dirs.append(root)

    target = mod_root or (go_dirs[0] if go_dirs else None)
    if target is None:
        return "no_go_files"

    try:
        result = subprocess.run(cmd, cwd=target, capture_output=True, text=True, timeout=timeout)
        if result.returncode == 0:
            return "yes" if "build" in cmd or "test" in cmd else "clean"
        return "no" if "build" in cmd or "test" in cmd else "issues"
    except (subprocess.TimeoutExpired, FileNotFoundError):
        return "skip"


def load_grading(variant_dir: Path) -> dict | None:
    """Load and normalize grading.json."""
    path = variant_dir / "grading.json"
    if not path.exists():
        return None
    raw = load_json_safe(path)
    if raw is None:
        return None
    exps = raw.get("expectations", raw.get("assertions", []))
    normalized = []
    for e in exps:
        text = e.get("text", e.get("assertion", "?"))
        is_pass = e.get("passed") is True or e.get("verdict", "") == "PASS"
        evidence = e.get("evidence", "")
        normalized.append({"text": text, "passed": is_pass, "evidence": evidence})
    passed = sum(1 for n in normalized if n["passed"])
    tl = raw.get("pass_count")
    if tl is not None:
        passed = tl
    total = len(normalized)
    return {
        "expectations": normalized,
        "summary": {
            "passed": passed,
            "failed": total - passed,
            "total": total,
            "pass_rate": round(passed / total, 3) if total > 0 else 0,
        },
    }


def build_variant_data(variant_dir: Path) -> dict:
    """Build data dict for one variant."""
    outputs = variant_dir / "outputs"
    if not outputs.exists():
        return {}
    files = find_files(outputs)
    return {
        "lines": count_go_lines(outputs),
        "files": files,
        "fileCount": len(files),
        "code_preview": get_code_preview(outputs),
        "compiles": run_go_check(outputs, ["go", "build", "./..."]),
        "tests_pass": run_go_check(outputs, ["go", "test", "-race", "-count=1", "./..."]),
        "govet": run_go_check(outputs, ["go", "vet", "./..."]),
        "grading": load_grading(variant_dir),
    }


def find_iteration_dirs(workspace: Path) -> list[Path]:
    """Find all iteration-N directories, sorted by number."""
    dirs = sorted(workspace.glob("iteration-*"))
    return [d for d in dirs if d.is_dir()]


def is_optimization_data(data: object) -> bool:
    """Return True when the payload matches optimize_loop.py results."""
    if not isinstance(data, dict):
        return False
    iterations = data.get("iterations")
    if not isinstance(iterations, list):
        return False
    if "baseline_score" not in data:
        return False
    if "target" not in data:
        return False
    return all(isinstance(item, dict) and "number" in item and "verdict" in item for item in iterations)


def load_optimization_data(workspace: Path) -> dict | None:
    """Load optimization loop results when present in the workspace."""
    candidates = [
        workspace / "results.json",
        workspace / "evals" / "iterations" / "results.json",
        workspace / "out" / "results.json",
    ]
    for path in candidates:
        if path.exists():
            data = load_json_safe(path)
            if is_optimization_data(data):
                return data
    return None


def build_data(workspace: Path) -> dict:
    """Build full comparison data."""
    evals_path = workspace / "evals" / "evals.json"
    evals_meta = {}
    evals_raw = None
    if evals_path.exists():
        evals_raw = load_json_safe(evals_path)
        if evals_raw:
            for ev in evals_raw.get("evals", []):
                evals_meta[ev.get("name", ev.get("id", ""))] = ev

    evals_data = []
    benchmark = []

    # Use the latest iteration directory (or iteration-1 as fallback)
    iterations = find_iteration_dirs(workspace)
    if not iterations:
        return {
            "evals": [],
            "benchmark": [],
            "variantAName": "Variant A",
            "variantBName": "Variant B",
            "variantCName": "Variant C",
            "optimization": load_optimization_data(workspace),
        }

    iteration = iterations[-1]  # Latest iteration

    for eval_dir in sorted(iteration.iterdir()):
        if not eval_dir.is_dir():
            continue
        name = eval_dir.name
        a_data = build_variant_data(eval_dir / "variant-A")
        b_data = build_variant_data(eval_dir / "variant-B")
        c_data = build_variant_data(eval_dir / "variant-C")

        prompt = evals_meta.get(name, {}).get("prompt", "")

        # Load blind comparisons if available
        blind = (
            load_json_safe(eval_dir / "blind_comparison.json")
            if (eval_dir / "blind_comparison.json").exists()
            else None
        )
        blind_bc = (
            load_json_safe(eval_dir / "blind_comparison_bc.json")
            if (eval_dir / "blind_comparison_bc.json").exists()
            else None
        )

        eval_entry = {
            "name": name,
            "prompt": prompt,
            "variantA": a_data,
            "variantB": b_data,
            "blind_comparison": blind,
            "blind_comparison_bc": blind_bc,
        }
        if c_data:
            eval_entry["variantC"] = c_data
        evals_data.append(eval_entry)

        a_rate = a_data.get("grading", {}).get("summary", {}).get("pass_rate", 0) if a_data.get("grading") else 0
        b_rate = b_data.get("grading", {}).get("summary", {}).get("pass_rate", 0) if b_data.get("grading") else 0
        c_rate = c_data.get("grading", {}).get("summary", {}).get("pass_rate", 0) if c_data.get("grading") else 0
        bm = {"name": name, "aRate": a_rate, "bRate": b_rate}
        if c_data:
            bm["cRate"] = c_rate
        benchmark.append(bm)

    variants = evals_raw.get("variants", {}) if evals_raw else {}

    return {
        "evals": evals_data,
        "benchmark": benchmark,
        "variantAName": variants.get("A", {}).get("name", "Variant A"),
        "variantBName": variants.get("B", {}).get("name", "Variant B"),
        "variantCName": variants.get("C", {}).get("name", "Variant C"),
        "optimization": load_optimization_data(workspace),
    }


def main() -> int:
    parser = build_parser()
    args = parser.parse_args()

    workspace = args.workspace.resolve()
    template = workspace / "compare.html"
    output = (args.output or workspace / "compare_report.html").resolve()

    if not template.exists():
        print(f"Error: {template} not found", file=sys.stderr)
        return 1

    data = build_data(workspace)
    html = read_text_safe(template).replace("__DATA_PLACEHOLDER__", json.dumps(data, indent=2))
    output.write_text(html, encoding="utf-8")

    print(f"Report: {output}")
    print(f"Evals: {len(data['evals'])}")
    for ev in data["evals"]:
        a = ev.get("variantA", {})
        b = ev.get("variantB", {})
        print(
            f"  {ev['name']}: A={a.get('lines', 0)}L/{a.get('compiles', '?')} B={b.get('lines', 0)}L/{b.get('compiles', '?')}"
        )

    return 0


if __name__ == "__main__":
    sys.exit(main())
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall