oliver-kriska/claude-elixir-phoenix / lab/autoresearch
lab:autoresearch
VerifiedSelf-improving loop for plugin skills. Reads program.md, proposes one mutation per iteration, evaluates against deterministic scorer, keeps improvements via git, reverts failures. Targets weakest skill+dimension. Use with /loop for overnight runs.
| File | Kind | Size |
|---|---|---|
| .gitignore | other | 130 B |
| SKILL.md | skill md | 4.6 KB |
| program.md | markdown | 3.9 KB |
| references/mutation-strategies.md | markdown | 2.4 KB |
| references/state-management.md | markdown | 1.1 KB |
| retention.py | script | 5.7 KB |
| scripts/checks.shexecutable | script | 3.2 KB |
| scripts/protected_sections.py | script | 3.4 KB |
| scripts/run-iteration.py | script | 25.0 KB |
| scripts/score-skill.py | script | 1.2 KB |
| tests/__init__.py | script | 0 B |
| tests/test_deviation_dispatch.py | script | 3.1 KB |
| tests/test_protected_sections.py | script | 3.3 KB |
SKILL.md
View on GitHub---
name: lab:autoresearch
description: >
Self-improving loop for plugin skills. Reads program.md, proposes one
mutation per iteration, evaluates against deterministic scorer, keeps
improvements via git, reverts failures. Targets weakest skill+dimension.
Use with /loop for overnight runs.
effort: high
argument-hint: "[--skill NAME] [--strategy targeted|sweep|random] [--dry-run] [--max-iterations N]"
disable-model-invocation: true
---
# Autoresearch — Plugin Skill Self-Improvement
Iteratively improve plugin skills via the autoresearch pattern:
propose one mutation -> eval -> keep/revert -> repeat.
## Usage
```
/lab:autoresearch # Targeted: attack weakest skill+dimension
/lab:autoresearch --skill review # Focus on one skill
/lab:autoresearch --strategy sweep # Process all skills alphabetically
/lab:autoresearch --dry-run # Show what would change, don't commit
```
For overnight runs:
```
/loop 5m /lab:autoresearch --strategy sweep --max-iterations 200
```
## Iron Laws
1. **ONE mutation per iteration** — if description needs "and", split into two
2. **NEVER mutate read-only files** — check program.md before every write
3. **EVAL is deterministic** — always use the wrapper script, never LLM-judge
4. **REVERT on regression OR checks failure** — no exceptions
5. **LOG every iteration** — use `keep` or `revert` command (never skip)
6. **CHECK ideas.md before proposing** — don't rediscover known optimizations
## Wrapper Script Commands
All eval/git/journal operations go through ONE script. Do NOT run these manually.
```bash
# Find the weakest skill+dimension
python3 lab/autoresearch/scripts/run-iteration.py target --strategy targeted
# Score a skill (before mutation, to get baseline)
python3 lab/autoresearch/scripts/run-iteration.py score <skill-name>
# After mutation: score + checks + compare → verdict (KEEP or REVERT)
python3 lab/autoresearch/scripts/run-iteration.py eval <skill-name>
# Act on verdict:
python3 lab/autoresearch/scripts/run-iteration.py keep <skill> <dim> <old> <new> \
--desc "what changed" --asi '{"hypothesis": "why", "mechanism": "how"}'
python3 lab/autoresearch/scripts/run-iteration.py revert <skill> <dim> <old> <new> \
--desc "what was attempted" --asi '{"hypothesis": "why", "regression": "what broke", "avoid": "do not retry this"}'
# Check overall progress
python3 lab/autoresearch/scripts/run-iteration.py status
```
## Core Loop (ONE iteration)
### Step 1: Read State
1. Read `lab/autoresearch/program.md` (goals, mutable surface, rules)
2. Read `lab/autoresearch/ideas.md` if it exists (deferred optimizations)
3. Run: `python3 lab/autoresearch/scripts/run-iteration.py status`
### Step 2: Select Target
Run: `python3 lab/autoresearch/scripts/run-iteration.py target --strategy targeted`
Parse the JSON: `skill`, `dimension`, `failing_checks`. If `all_perfect` → STOP.
### Step 3: Read + Propose
1. Read target SKILL.md and its references/ listing
2. Read eval definition from `lab/eval/evals/{skill}.json`
3. Check `ideas.md` for deferred ideas about this skill
4. Check recent journal entries for prior failures on this skill (avoid repeats)
5. Consult `${CLAUDE_SKILL_DIR}/references/mutation-strategies.md`
6. Propose exactly ONE change targeting the failing checks
### Step 4: Apply + Evaluate
1. Apply the mutation via Edit tool
2. Run: `python3 lab/autoresearch/scripts/run-iteration.py eval <skill-name>`
3. Parse JSON → check `verdict` field
### Step 5: Keep or Revert
**If verdict is KEEP**:
```bash
python3 lab/autoresearch/scripts/run-iteration.py keep <skill> <dim> <old> <new> \
--desc "..." --asi '{"hypothesis": "...", "mechanism": "..."}'
```
**If verdict is REVERT**:
```bash
python3 lab/autoresearch/scripts/run-iteration.py revert <skill> <dim> <old> <new> \
--desc "..." --asi '{"hypothesis": "...", "regression": "...", "avoid": "..."}'
```
### Step 6: Ideas Backlog
If during analysis you discovered a promising optimization you can't act on now:
- Append it to `lab/autoresearch/ideas.md` as a bullet
- On next resume: prune stale/tried ideas, experiment with the rest
### Step 7: Continue or Stop
- All targets >= 0.95? Print "AUTORESEARCH_COMPLETE"
- Max iterations reached? Print "AUTORESEARCH_COMPLETE"
- 50 consecutive discards? Print "AUTORESEARCH_STUCK"
- Otherwise: immediately start Step 1 again
## References
- `${CLAUDE_SKILL_DIR}/references/mutation-strategies.md` — mutation type catalog
- `${CLAUDE_SKILL_DIR}/references/state-management.md` — git protocol, journaling
- `lab/autoresearch/program.md` — research agenda (read every iteration)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |