Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

references/hook-development/telemetry-database.md

View on GitHub
# Telemetry Database Reference

Schema, API, and operations for the store the hooks write: routing telemetry, agent evidence, and governance events.

## Database Location

`~/.claude/learning/learning.db` (SQLite with WAL mode for concurrent access). Two other `learning.db` files exist on disk and carry a different, older schema — resolve the path through `learning_db_v2.get_db_path()` rather than hardcoding it.

## What the hooks write

| Table | Written by | Carries |
|---|---|---|
| `evidence_route_decisions` | `routing-decision-recorder.py` (PostToolUse:Agent) | One row per dispatch: agent, skill, pipeline, stack, marker |
| `routing_outcome_basis` | `routing-outcome-recorder.py` (SubagentStop), `routing-outcome-finalizer.py` (UserPromptSubmit), `routing-outcome-stop-fallback.py` (Stop: errored and stale entries; SessionEnd: every entry still pending) | How each outcome was scored, so a rate can be read against its basis |
| `evidence_events` / `evidence_sessions` | agent evidence hooks | Per-event success and failure records with target paths |
| `telemetry_runs` | `record_telemetry_run` callers | Per-run token, step, and tool-error totals |
| `governance_events` | ADR and creation-gate hooks | Gate decisions and their resolutions |

Every one of these is append-only from a hook's perspective. A hook records what happened; it never edits an agent or skill file.

## Schema

```sql
CREATE TABLE evidence_route_decisions (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT,
    marker TEXT,                   -- dispatch id; route-fit scoring reads one marker per event
    agent TEXT,
    skill TEXT,
    pipeline TEXT,
    stack TEXT,
    outcome TEXT,                  -- success | failure | neutral, set by the finalizer
    created_at TEXT DEFAULT (datetime('now'))
);

CREATE TABLE telemetry_runs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT,
    git_sha TEXT,
    topic TEXT,
    tokens INTEGER,
    steps INTEGER,
    tool_errors INTEGER,
    created_at TEXT DEFAULT (datetime('now'))
);

CREATE TABLE governance_events (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT,
    event_type TEXT,               -- the gate that fired
    detail TEXT,
    resolved_at TEXT,
    created_at TEXT DEFAULT (datetime('now'))
);
```

Read the live definitions in `hooks/lib/learning_db_v2.py` before writing a migration — the file is the source of truth and carries columns this summary omits.

## Core API (`hooks/lib/learning_db_v2.py`)

### Record a routing decision

```python
from learning_db_v2 import record_evidence_route_decision

record_evidence_route_decision(
    session_id=session_id,
    marker=dispatch_id,
    agent="general-purpose",  # `main` for main-thread routes; repo agents are retired (plan 009)
    skill="systematic-debugging",
)
```

One marker per event. Packing several markers into a single Bash or Workflow call keeps them recorded but forfeits route-fit scoring, which reads a lone marker per event.

### Update a routing outcome

```python
from learning_db_v2 import update_evidence_route_outcome

update_evidence_route_outcome(marker=dispatch_id, outcome="success")
```

Three-way and deterministic: failure on errors or rejection, success on explicit acceptance, neutral otherwise. Silence is neutral, never acceptance.

### Record an evidence event

```python
from learning_db_v2 import record_evidence_event

record_evidence_event(
    session_id=session_id,
    route_key="agent:skill",
    target="path/to/file.py",
    success=True,
)
```

### Read routing context

```python
from learning_db_v2 import get_evidence_route_context, get_evidence_decision

context = get_evidence_route_context("general-purpose:systematic-debugging")
advice = get_evidence_decision("general-purpose:systematic-debugging")
```

### Record a governance event

```python
from learning_db_v2 import record_governance_event, resolve_governance_event

event_id = record_governance_event(session_id=session_id, event_type="adr-gate", detail="blocked")
resolve_governance_event(event_id)
```

### Record a run's totals

```python
from learning_db_v2 import record_telemetry_run

record_telemetry_run(session_id=session_id, topic="quality-loop", tokens=n, steps=k, tool_errors=e)
```

## CLI (`scripts/learning-db.py`)

Every command below is read-only except the two recorders.

```bash
# Routing health — read this before reading any rate
python3 ~/.claude/scripts/learning-db.py route-health

# Routing statistics; --by is required
python3 ~/.claude/scripts/learning-db.py route-stats --by agent   # skill|force-route|errors|override|week|day

# Health-aware re-rank input
python3 ~/.claude/scripts/learning-db.py route-weights

# Compare two cohorts before and after a change
python3 ~/.claude/scripts/learning-db.py route-delta --from SHA_OR_DATE --to SHA_OR_DATE

# Agent evidence
python3 ~/.claude/scripts/learning-db.py evidence-recent
python3 ~/.claude/scripts/learning-db.py evidence-failures
python3 ~/.claude/scripts/learning-db.py evidence-file-history PATH
python3 ~/.claude/scripts/learning-db.py evidence-route-context AGENT:SKILL
python3 ~/.claude/scripts/learning-db.py evidence-decide AGENT:SKILL

# Reviewer cost and precision
python3 ~/.claude/scripts/learning-db.py review-roi
python3 ~/.claude/scripts/learning-db.py review-fps

# Enhancement skills seen stacked
python3 ~/.claude/scripts/learning-db.py stack-usage

# Recorders
python3 ~/.claude/scripts/learning-db.py record-routing-outcome ...
python3 ~/.claude/scripts/learning-db.py route-failure AGENT:SKILL --reason-file FILE --routing-relevant yes
```

## Outcome basis: read it before reading a rate

`route-health` reports the outcome basis and the silent-success share alongside the rates. A per-route error rate computed mostly from neutral outcomes describes the scorer, not the router. Silent failure is the central enemy here: it raises no error to grep, so the basis is what tells you whether a clean-looking rate means anything.

## Atomic writes and concurrency

Hooks fire on every tool call and share this database with read-only CLI queries.

- WAL mode is enabled at init; keep it.
- Set `PRAGMA busy_timeout` on any connection a hook opens. A hook that hits `SQLITE_BUSY` and exits non-zero blocks the tool call.
- File-based state outside SQLite uses write-to-temp-then-rename, never an in-place write.
- Any failure path still exits 0. A telemetry write is never worth blocking a tool on.

## Retention

Telemetry accumulates per dispatch, so it grows with use. `prune_ancillary` trims ancillary rows; run it as a maintenance step, never inside an event hook, because a hook's budget is under 50ms and a prune is not.
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall