Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

references/toolkit-evolution/evolve-preferred-patterns.md

View on GitHub
# Toolkit Evolution — Patterns, Errors, and Cost

Reference for Phase 3 CRITIQUE (fallback), failure modes, error handling, and cost estimates.

---

## Patterns to Detect and Fix

- **Improving without measuring** -- every change must have a baseline and A/B result. "It looks better" is not evidence.
- **Merging without validation** -- every winner must pass multi-persona critique (STRONG consensus) AND A/B testing (WIN status) before merge. The validation gates are the review.
- **Ignoring failed experiments** -- they are valuable data. Record them in the learning DB so the same idea is not re-proposed.
- **Improving everything at once** -- max 3 implementations per cycle. Focus compounds; scatter dissipates.
- **Running without diagnosis** -- do not propose solutions without evidence of problems first. Solutions looking for problems create phantom work.
- **Proposing duplicates** -- always check INDEX.json before proposing a new skill or capability. Extend existing skills when possible.
- **Discovery without evidence** -- the DISCOVER phase requires concrete data points (routing misses, manual workflows, community requests), not speculation. "It might be useful" is not a valid justification.
- **Discovering too often** -- discovery runs monthly, not nightly. Running it every cycle wastes budget on perspective agents that will produce the same gaps repeatedly.

---

## Error Handling

### Error: "learning-db.py not found"
Cause: Telemetry scripts not installed.
Solution: The CLI is at `~/.claude/scripts/learning-db.py`. If missing, skip the telemetry queries and diagnose from git log and dream reports. Record this gap as an improvement opportunity.

### Error: "No dream reports found"
Cause: Auto-dream has not run yet or state directory is empty.
Solution: Proceed without dream insights. Use git log and routing telemetry as primary data sources. Note that enabling auto-dream would improve future evolution cycles.

### Error: "No STRONG proposals after critique"
Cause: All proposals received MODERATE or WEAK consensus.
Solution: Report to the user that no high-confidence improvements were found this cycle. Record the proposals and critique feedback in the evolution report for future reference.

### Error: "A/B test inconclusive"
Cause: Test cases don't discriminate between baseline and candidate.
Solution: Review test case quality. Non-discriminating tests ("file exists") provide false signal. Write tests that exercise the specific behavior the proposal changes. If still inconclusive after better tests, shelve the proposal.

### Error: "Feature branch conflicts"
Cause: Multiple evolution implementations touch the same files.
Solution: Reduce to 1 implementation per cycle when conflicts arise. Alternatively, sequence implementations so later ones branch from earlier ones.

---

## Cost Estimate

A full evolution cycle runs all 7 phases and may dispatch multiple subagents. Estimated cost:
- Discovery (Phase 0, monthly): ~$0.50-0.75 (5 parallel perspective agents + dedup)
- Diagnosis + Proposal: ~$0.15 (reading files, querying DBs)
- Critique: ~$0.30 (3 persona agents evaluating proposals)
- Build: ~$0.50-1.50 (1-3 implementation agents)
- Validate: ~$0.50-1.50 (A/B test runs)
- Evolve: ~$0.10 (PR creation, learning DB writes)

Total without discovery: ~$1.50-3.50 per cycle. With discovery: ~$2.00-4.25.
Budget capped at $5.00 via wrapper script.
Nightly cost at full utilization: ~$45-105/month. Discovery adds ~$0.50-0.75/month (runs monthly, not nightly). Cycles with no STRONG proposals exit early (diagnosis + proposal only: ~$0.45).

---

## Phase 3 Inline Critique Fallback

Use this when the assessment skill (Multi-Persona Critique mode) is NOT available. Dispatch 3 parallel agents:

**Pragmatist**: "You are a pragmatist engineer. Evaluate each proposal on: implementation feasibility, time-to-value, and risk of unintended side effects. Rate each STRONG/MODERATE/WEAK with one sentence of justification."

**Purist**: "You are an architecture purist. Evaluate each proposal on: design coherence with existing toolkit patterns, long-term maintainability, and whether it solves a root cause vs a symptom. Rate each STRONG/MODERATE/WEAK with one sentence of justification."

**User Advocate**: "You are a user advocate. Evaluate each proposal on: how often real users hit this problem, whether the solution reduces friction, and whether it introduces new complexity users must learn. Rate each STRONG/MODERATE/WEAK with one sentence of justification."

Scoring: STRONG = 3, MODERATE = 2, WEAK = 1. Average across personas.
- Score >= 2.5 = STRONG consensus
- Score 1.5-2.4 = MODERATE consensus
- Score < 1.5 = WEAK consensus (shelve)

---

## Phase 0 DISCOVER — Perspective Agents

Brief all 5 agents with the same baseline data from `references/diagnose-scripts.md` § DISCOVER Step 1. Dispatch all 5 simultaneously:

| Agent | Perspective | What it looks for |
|-------|------------|-------------------|
| **The User** | Reads routing telemetry for weak and error-prone routes (`python3 scripts/learning-db.py route-health`, then `route-stats --by errors`) and requests that landed on `general-purpose` with no specialist. "What did users ask for that we couldn't handle?" |
| **The Operator** | Examines the active projects (check git repos in `~/`) for repeated manual workflows that could be skills. "What am I doing by hand that should be automated?" |
| **The Strategist** | Uses the business-ops skill's (formerly csuite) EVALUATION mode thinking: what decision-support, content, or process skills would make the owner more effective? Reads `skills/business/business-ops/SKILL.md` for framework. "What high-leverage skills are we missing?" |
| **The Community** | Web-searches for what people are building and requesting in AI coding communities (Claude Code GitHub issues, Reddit, X/Twitter). "What does the market want?" |
| **The Architect** | Examines current skill categories (from `skills/INDEX.json`) for structural gaps. Cross-references with `agents/INDEX.json`. "Where are the architectural blind spots?" E.g., "we have 23 process skills but 0 decision skills." |

Each agent produces 2-3 skill proposals in this format:

```
PROPOSAL: {skill-name}
Category: {category}
Triggers: {3-5 routing triggers}
Justification: {1-2 sentences on why this is needed}
Evidence: {what data supports this -- routing gaps, user patterns, market signals}
```

---

## Scheduling (Cron Setup)

Runs nightly at 3:07 AM, after auto-dream (2:07 AM) finishes consolidating learnings:

```bash
python3 ~/.claude/scripts/crontab-manager.py add \
  --tag "toolkit-evolution" \
  --schedule "7 3 * * *" \
  --command "/home/feedgen/vexjoy-agent/scripts/toolkit-evolution-cron.sh --execute >> /home/feedgen/vexjoy-agent/cron-logs/toolkit-evolution/cron.log 2>&1"
```

Schedule uses 3:07 AM (off-minute per cron best practice, 1 hour after auto-dream). Budget set to $5.00 per run.

Manual invocation:
```
/evolve
/evolve routing
/do evolve toolkit
/do evolve hooks
```
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 470kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall