Skip to content

notque/vexjoy-agent / skills/meta/toolkit

toolkit

Verified

Toolkit management: create and evaluate skills and agents, manage routing tables, generate Claude.md.

FileKindSize
SKILL.mdskill md15.5 KB
agents/skill-creator/analyzer.mdmarkdown4.8 KB
agents/skill-creator/comparator.mdmarkdown4.7 KB
agents/skill-creator/grader.mdmarkdown3.9 KB
assets/skill-creator/eval_viewer.htmlother57.1 KB
references/agent-comparison.mdmarkdown11.5 KB
references/agent-comparison/benchmark-tasks.mdmarkdown6.8 KB
references/agent-comparison/do-creation-compliance-tasks.jsonconfig8.5 KB
references/agent-comparison/examples-and-errors.mdmarkdown1.0 KB
references/agent-comparison/grading-rubric.mdmarkdown4.3 KB
references/agent-comparison/methodology.mdmarkdown7.4 KB
references/agent-comparison/optimization-guide.mdmarkdown12.0 KB
references/agent-comparison/optimization-tasks.example.jsonconfig853 B
references/agent-comparison/optimize-phase.mdmarkdown9.3 KB
references/agent-comparison/read-only-ops-short-tasks.jsonconfig375 B
references/agent-comparison/report-template.mdmarkdown3.8 KB
references/agent-comparison/socratic-debugging-body-short-tasks.jsonconfig372 B
references/agent-comparison/socratic-debugging-trigger-tasks.jsonconfig2.8 KB
references/agent-creator.mdmarkdown10.9 KB
references/agent-creator/agent-design-patterns.mdmarkdown12.4 KB
references/agent-creator/agent-eval-design.mdmarkdown7.7 KB
references/agent-creator/agent-frontmatter-template.mdmarkdown9.3 KB
references/agent-evaluation.mdmarkdown10.6 KB
references/agent-evaluation/batch-evaluation.mdmarkdown1.6 KB
references/agent-evaluation/common-issues.mdmarkdown2.1 KB
references/agent-evaluation/report-templates.mdmarkdown2.2 KB
references/agent-evaluation/scoring-rubric.mdmarkdown2.4 KB
references/blind-ab-tests.mdmarkdown7.8 KB
references/generate-claudemd.mdmarkdown11.0 KB
references/generate-claudemd/CLAUDEMD_TEMPLATE.mdmarkdown4.1 KB
references/generate-claudemd/examples-and-errors.mdmarkdown7.4 KB
references/github-profile-rules.mdmarkdown7.5 KB
references/github-profile-rules/github-api-patterns.mdmarkdown8.2 KB
references/github-profile-rules/rule-categories.mdmarkdown4.8 KB
references/hook-development.mdmarkdown10.9 KB
references/hook-development/architecture.mdmarkdown3.7 KB
references/hook-development/code-examples.mdmarkdown9.4 KB
references/hook-development/preferred-patterns.mdmarkdown6.5 KB
references/hook-development/telemetry-database.mdmarkdown6.5 KB
references/routing-table-updater.mdmarkdown10.2 KB
references/routing-table-updater/batch-mode.mdmarkdown2.1 KB
references/routing-table-updater/conflict-resolution.mdmarkdown3.8 KB
references/routing-table-updater/error-handling.mdmarkdown1.9 KB
references/routing-table-updater/examples.mdmarkdown4.8 KB
references/routing-table-updater/extraction-patterns.mdmarkdown2.2 KB
references/routing-table-updater/routing-format.mdmarkdown3.6 KB
references/routing-table-updater/skill-examples.mdmarkdown2.7 KB
references/skill-composer.mdmarkdown9.1 KB
references/skill-composer/compatibility-matrix.mdmarkdown13.0 KB
references/skill-composer/composition-patterns.mdmarkdown4.9 KB
references/skill-composer/examples.mdmarkdown17.1 KB
references/skill-composer/skill-patterns.mdmarkdown12.5 KB
references/skill-creator.mdmarkdown28.9 KB
references/skill-creator/agent-template.mdmarkdown18.5 KB
references/skill-creator/artifact-schemas.mdmarkdown8.2 KB
references/skill-creator/bundled-components.mdmarkdown2.3 KB
references/skill-creator/complexity-tiers.mdmarkdown8.0 KB
references/skill-creator/domain-research-targets.mdmarkdown13.2 KB
references/skill-creator/enrichment-workflow.mdmarkdown9.9 KB
references/skill-creator/error-catalog.mdmarkdown9.9 KB
references/skill-creator/preferred-patterns.mdmarkdown11.7 KB
references/skill-creator/progressive-disclosure.mdmarkdown8.5 KB
references/skill-creator/skill-template.mdmarkdown14.6 KB
references/skill-creator/workflow-patterns.mdmarkdown7.6 KB
references/system-upgrade.mdmarkdown11.0 KB
references/system-upgrade/component-audit-checklists.mdmarkdown8.4 KB
references/system-upgrade/upgrade-failure-modes.mdmarkdown8.7 KB
references/system-upgrade/upgrade-signal-parsing.mdmarkdown7.3 KB
references/toolkit-evolution.mdmarkdown14.5 KB
references/toolkit-evolution/diagnose-scripts.mdmarkdown9.5 KB
references/toolkit-evolution/evolution-history.mdmarkdown5.5 KB
references/toolkit-evolution/evolution-report-template.mdmarkdown716 B
references/toolkit-evolution/evolve-preferred-patterns.mdmarkdown6.9 KB
references/toolkit-evolution/evolve-scripts.mdmarkdown3.7 KB
references/toolkit-governance.mdmarkdown16.0 KB
references/toolkit-governance/adr-lifecycle.mdmarkdown10.9 KB
references/toolkit-governance/frontmatter-compliance.mdmarkdown11.1 KB
references/toolkit-governance/hook-standardization.mdmarkdown21.5 KB
references/toolkit-governance/routing-table-patterns.mdmarkdown11.6 KB
references/weak-model-uplift.mdmarkdown10.3 KB
scripts/agent-comparison/compare.pyscript2.3 KB
scripts/agent-comparison/generate_variant.pyscript19.6 KB
scripts/agent-comparison/optimize_loop.pyscript89.3 KB
scripts/agent-comparison/trigger_eval.pyscript19.5 KB
scripts/routing-table-updater/extract_metadata.pyscript10.8 KB
scripts/routing-table-updater/generate_routes.pyscript10.5 KB
scripts/routing-table-updater/scan.pyscript4.7 KB
scripts/routing-table-updater/update_routing.pyscript12.4 KB
scripts/routing-table-updater/validate.pyscript7.3 KB
scripts/skill-composer/build_dag.pyscript12.5 KB
scripts/skill-composer/discover_skills.pyscript12.1 KB
scripts/skill-composer/validate.pyscript12.3 KB
scripts/skill-creator/aggregate_benchmark.pyscript10.0 KB
scripts/skill-creator/eval_compare.pyscript10.2 KB
scripts/skill-creator/optimize_description.pyscript12.5 KB
scripts/skill-creator/package_results.pyscript7.8 KB
scripts/skill-creator/run_eval.pyscript6.3 KB

references/system-upgrade/upgrade-failure-modes.md

View on GitHub
# Upgrade Orchestration Failure Modes Reference

> **Scope**: Common orchestration failures in the 6-phase system-upgrade workflow: premature implementation, approval gate bypass, inline edits, and scope creep. Covers detection and remediation.
> **Version range**: system-upgrade reference (formerly system-upgrade-engineer), all versions
> **Generated**: 2026-04-15

---

## Overview

The system-upgrade workflow has mandatory gates and specialist dispatch rules. The most damaging
failures are silent: implementing outside authorized scope, making domain changes inline
instead of delegating to specialists, and running full-repo audits for scoped changes. These
produce either unauthorized bulk edits or subtly incorrect results that bypass domain validation.

---

<!-- no-pair-required: section header, not a standalone failure mode block -->
## Failure Mode Catalog

### Implementing Without Authorized Scope

**What it looks like**: Applying changes outside the user’s request or skipping the ranked PLAN presentation.

**Why wrong**: The user must control which upgrades happen. Requiring a second approval for an accepted plan also blocks requested work.

Do instead: Follow `skills/process/workflow/references/system-upgrade.md` Phase 3. Present the ranked table, continue within existing authorization, and ask about uncovered changes. Honor interactive and plan-only requests.

---

### Implementing Domain Changes Inline

**What it looks like**: Directly editing `hooks/posttool-rename-sweep.py` instead of
dispatching a hook worker that loads `hook-development.md`. Writing new skill frontmatter inline instead of
calling the `toolkit` skill.

Do instead: dispatch a `general-purpose` hook worker (prompt starts "Read ~/.claude/skills/toolkit/references/hook-development.md and follow it") for hook changes, `toolkit` for skill
and reference changes, and `toolkit` for routing table changes. Details follow.

**Detection**:
```bash
# Look for direct file edits to domain files in agent output logs
grep -rn "Edit\|Write" hooks/*.py agents/*.md --include="*.py" --include="*.md" 2>/dev/null
# the upgrade orchestrator should only create task_plan.md and branch setup files
```

**Why wrong**: Domain references (`hook-development.md`, the `toolkit` skill) carry template
conventions, event schema knowledge, and frontmatter validation that inline edits bypass.
A hook written inline without `hook-development.md`'s exit code contract knowledge will
likely use wrong exit codes. A skill written inline will miss required frontmatter fields.

Do instead:
- Hook changes → dispatch a `general-purpose` hook worker (prompt starts "Read ~/.claude/skills/toolkit/references/hook-development.md and follow it")
- Skill/reference changes → call the `toolkit` skill
- Routing table changes → dispatch `toolkit` skill
- Only create `task_plan.md` and branch setup files directly

---

### Not Scoping the Audit to Signal-Identified Components

**Detection**:
```bash
# Check if audit scanned all components for a targeted change
grep -c "Scanned\|checked\|audited" task_plan.md
# An audit for 2-hook changes should reference < 20 components, not 120+

# Check signal column presence in Change Manifest
grep "Component Types\|component type" task_plan.md
# Should be present — if missing, audit had no scope
```

**Why wrong**: Auditing all 120+ skills for a 2-hook change produces noise proportional to
scope. When every component appears in the audit, the PLAN phase cannot distinguish affected
from unaffected. Tier assignment degrades to noise.

Do instead: Build the Change Manifest with a "Component Types" column first. Default scope
is 10 most-recently-modified agents + all hooks + affected routing tables. Comprehensive
audit only with the explicit "comprehensive" keyword from the user.

---

### Skipping Required Validation

**What it looks like**: Shipping directly after IMPLEMENT without checking changed behavior and affected contracts.

**Why wrong**: Neither a smaller file nor a model score proves correctness.

Do instead: Follow the pipeline’s VALIDATE phase and report real checks and findings. Scoring is optional; required checks and blocking findings still govern delivery.

---

### Force-Pushing or Committing to Main

**Detection**:
```bash
# Confirm current branch is not main before any writes
git branch --show-current
# Should NEVER be main during an upgrade run

# Check no force-push flags in recent git commands
history | grep "push --force\|push -f"
```

**Why wrong**: Commits to main bypass branch protection and review. Force-push to main
overwrites upstream state and is unrecoverable without a backup. The branch naming
convention (`chore/system-upgrade-YYYY-MM-DD`) exists to ensure all changes go through PR.

Do instead: Run `git checkout -b chore/system-upgrade-YYYY-MM-DD` before Phase 4. Never
use `--force` or `-f` on push. If already on main, stash and create branch before proceeding.

---

### Reporting Regression as "Necessary"

**Detection**:
```bash
# Check VALIDATE section for score drops paired with justification phrases
grep -A 3 "score.*lower\|regressed\|dropped" task_plan.md | grep -i "necessary\|intentional\|expected\|trade-off"
# These phrases in the same context indicate a rationalized regression
```

**Why wrong**: Regressions are user decisions, not agent decisions. When a component
scores lower after modification, the agent's job is to surface it clearly, not to
rationalize it away. The user may have context that makes the tradeoff acceptable;
the agent does not have that context.

Do instead: Report the regression factually: "Component X scored N before, M after (delta -K).
Cause: [specific change]. Recommend: revert or acknowledge." Then wait.

---

## Parallel Dispatch Patterns

### When to Fan Out

Dispatch parallel Agent calls when:
- 3 or more independent changes of the same type (e.g., 4 hooks need the same upgrade)
- Changes target different component types (hook + agent + routing table, no interdependency)
- A/B comparison of two upgrade approaches

```markdown
## Parallel Group Assignment (Phase 3 output format)

| Tier | Component | Change Type | Effort | Group |
|------|-----------|------------|--------|-------|
| Critical | hooks/rename-sweep.py | upgrade | S | Group A |
| Critical | hooks/branch-safety.py | upgrade | S | Group A |
| Important | skills/meta/toolkit/references/hook-development.md | inject-pattern | M | Group B |
| Important | skills/toolkit | upgrade | M | Group B |
| Minor | routing-tables.md | update | S | Group C |
```

Dispatch Group A in a single message (parallel). Wait for completion. Then Group B. Then C.

### When NOT to Fan Out

Use a single agent when:
- Change B depends on output from Change A (sequential dependency)
- Both agents would edit the same file (race condition)
- The approval gate has not yet been cleared

---

## Error-Fix Mappings

| Symptom | Root Cause | Fix |
|---------|-----------|-----|
| Session deadlock after hook deploy | Hook deployed before syntax verification | `python3 -m py_compile hook.py` before `cp` to `~/.claude/hooks/` |
| PR merge fails — CI checks failed | Ruff lint or format error in .py files created during upgrade | `ruff check . && ruff format --check .` before push |
| Agent dispatch returns empty output | Dispatched specialist had insufficient context | Re-dispatch with narrower scope and explicit file paths |
| Routing gap after upgrade | New agent not added to routing tables | Call the Skill tool with `toolkit`. Do this after every new agent. |
| Component scores unchanged (no improvement) | Upgrade changed names but not behavior | Verify upgrade touched functional content, not just formatting |

---

## Phase Gate Summary

| Phase | Gate | Consequence of Skipping |
|-------|------|------------------------|
| Phase 1 → Phase 2 | 0 signals check | Audit scans everything with no focus |
| Phase 2 → Phase 3 | All components opened and checked | Plan tier assignments are wrong |
| Phase 3 → Phase 4 | Plan presented and scope authorized | Unauthorized bulk edits |
| Phase 4 → Phase 5 | Branch exists, not main | Risk of main commit or force push |
| Phase 5 → Phase 6 | Required checks pass and blocking findings resolved | Regressions ship without user awareness |

---

## Detection Commands Reference

```bash
# Confirm not on main before writes
git branch --show-current

# Check Phase 3 plan was presented before writes
grep "PLAN\|approval\|proceed" task_plan.md

# Verify toolkit was run (Phase 5)
grep "toolkit\|score.*before\|score.*after" task_plan.md

# Check for regression rationalization phrases
grep -i "necessary\|intentional\|expected trade" task_plan.md

# Verify no inline domain edits (the upgrade orchestrator should not edit hook files directly)
git diff --name-only HEAD | grep "^hooks/" | head -5
# Should be empty — hooks should only be touched by the hook worker
```
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall