ilandahan/AID / skills/autoresearch
autoresearch
VerifiedBounded improvement loop: one artifact, fixed eval score, keep an edit only if strictly better. Use for AR_DESIGN/AR_FUNCTION/AR_ACCEPTANCE stages or standalone bounded iteration.
| File | Kind | Size |
|---|---|---|
| SKILL.extended.md | markdown | 5.9 KB |
| SKILL.md | skill md | 21.6 KB |
| bin/check-harness-immutable.mjs | script | 1.2 KB |
| bin/check-harness-immutable.shexecutable | script | 1.9 KB |
| program.ar-acceptance.md | markdown | 15.3 KB |
| program.ar-design.md | markdown | 12.9 KB |
| program.ar-function.md | markdown | 17.0 KB |
| verify-run.shexecutable | script | 11.3 KB |
program.ar-design.md
View on GitHub# program.ar-design.md — Autoresearch Pass AR-1 (DESIGN, pre-TDD)
> Karpathy autoresearch loop, specialized for **design/structural quality** of freshly
> developed production source. Instead of editing `train.py` to minimize `val_bpb`, you
> edit one task's source set to maximize a reflection-agent **design quality** score —
> while the public surface stays frozen so the next stage (TDD) has a stable target.
> The agent is the runner. You read this file and follow it. Loop until a stop fires.
---
## MODE
**design** — runs **AFTER** `CODE_REVIEW`, **BEFORE** `TDD`.
**No tests exist yet.** There is no running test suite to score against, so the harness
is reflection-quality ONLY. This is exactly why AR-1 is a separate pass from AR-2: it
must polish structure/clarity *without* moving the public surface, so TDD is not written
against a moving target.
---
## Goal
Iteratively improve the production source changed in task `{{TASK_ID}}` to maximize the
reflection-agent's **Phase-4a Code-Design weighted score**, using
`~/.claude/skills/autoresearch/criteria/phase-4a-code-design.yaml` (user-level bundle default; a project may override at `{{WORKSPACE}}/.claude/skills/autoresearch/criteria/`).
KPIs (each must reach **≥ {{KPI_TARGET}}**):
- **Structural health** — cohesion, function/module size, dead code, duplication, layering.
- **Loudness of failures** — errors surface; no silent swallowing; fail-fast on bad state.
- **Naming clarity** — names reveal intent; no misleading or placeholder identifiers.
- **Docs & traceability** — public surface documented; non-obvious decisions explained;
changes traceable to the task.
The composite for keep/revert is the reflection-agent's weighted overall (0.0–10.0).
**Stop only when a human interrupts you, or a stop condition below fires.**
---
## Mutable surface (the "train.py")
You may edit ONLY:
- The **production source files changed in this task** — i.e. the set from
`git diff --name-only` + untracked files introduced by this task, restricted to source
(no config, no `.aid/`, no generated artifacts). If a definitive changed-set is recorded
in `{{WORKSPACE}}/.aid/context.json`, use that; otherwise derive it from the diff once at
setup and treat THAT frozen list as the editable set for the whole pass.
- `{{RECORD_FILE}}` — the experiment log (`results.tsv`).
Snapshots are written ONLY under
`{{WORKSPACE}}/.aid/pipeline/autoresearch/{{TASK_ID}}/snapshots/` via the `cp` commands below.
Do not create, rename, move, or delete any other file.
---
## Fixed harness (the "prepare.py" — DO NOT TOUCH)
Treat as read-only — modifying any of these makes the score meaningless:
- the `reflection-agent` (its prompt/templates) — the scorer. **Resolve project→user:** load `../../agents/reflection-agent.md` from `{{WORKSPACE}}/.claude/…` if present, else `~/.claude/…` (the engine ships user-level).
- `~/.claude/skills/autoresearch/criteria/phase-4a-code-design.yaml` (user-level bundle default; a project may override at `{{WORKSPACE}}/.claude/skills/autoresearch/criteria/`) — the criteria
- the reflection `SKILL.md` scoring methodology
- `{{WORKSPACE}}/.aid/context.json`, `{{WORKSPACE}}/.aid/state.json`, `config.json` — read-only metadata
- any test files (none should exist yet — if any do, they are OUT of the editable set)
---
## HARD CONSTRAINT — the whole reason AR-1 is separate
**NO public-surface changes.** Within the editable set you MUST NOT change any of:
- exported / public **function or method signatures** (name, parameter list, order,
defaults, return type)
- exported **class / type / interface / enum shapes**
- public **module exports** (what is exported, and under what name)
- **data shapes** crossing a boundary — request/response objects, serialized payloads,
DB row shapes, public constants/enums consumed elsewhere
Allowed: internal refactors, renaming **local** (non-exported) identifiers, extracting
private helpers, replacing silent fallbacks with loud failures, adding/clarifying docs &
comments, removing dead code, de-duplicating internals — all behind an unchanged surface.
If improving a KPI seems to *require* a signature/shape change, **do not make it**: record
the idea as `skipped (surface-locked)` in `{{RECORD_FILE}}` and pick a different target.
Surface evolution is TDD/AR-2/AR-3 territory, not AR-1.
Other non-negotiables:
- **NEVER auto-commit and NEVER run destructive git** (no `reset --hard`, `clean -f`,
`checkout -- `, `push`, `rebase`). Validation only; the human commits.
- **No new dependencies**, no framework migration, no plaintext secrets.
- **One focused edit per iteration.** Edit only files in the frozen editable set.
- **Never downgrade a loud failure to silent.** Introduce **zero** new silent-failure paths
in `{{CRITICALITY_PROFILE}}`-critical paths — no new `catch` that swallows, no
`|| DEFAULT`, no `?? DEFAULT`, no `.catch` without rethrow.
- **No scope reduction to game the score** — do not delete features, validation, or guards
just to raise structural numbers.
---
## Setup phase (run ONCE per pass, then never again)
1. Resolve the frozen **editable set** (see Mutable surface). Record the file list and each
file's line count. If the set is empty, log a no-op baseline row and STOP.
2. Ensure the snapshot dir exists:
```bash
mkdir -p "{{WORKSPACE}}/.aid/pipeline/autoresearch/{{TASK_ID}}/snapshots"
```
3. Read `{{RECORD_FILE}}`. If it already has scored rows for this pass, do NOT re-baseline —
resume from the next iteration number. If empty/header-only, proceed to baseline.
4. **Snapshot the baseline** of the whole editable set:
```bash
TS="$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "{{WORKSPACE}}/.aid/pipeline/autoresearch/{{TASK_ID}}/snapshots/000-baseline-$TS"
# copy every file in the editable set into that dir, preserving relative paths
```
5. **Score the baseline** by spawning the subagent named `reflection-agent`:
- Tool: `Task`, `subagent_type: general-purpose`, `model: opus`, loading the
reflection-agent prompt from
`{{WORKSPACE}}/.claude/agents/reflection-agent.md`.
- Provide:
- `ORIGINAL_REQUEST`: verbatim from `{{WORKSPACE}}/.aid/context.json`.
- `STATED_WHY`: exact WHY from `{{WORKSPACE}}/.aid/context.json`.
- `PHASE_NUMBER`: 4, `PHASE_NAME`: "Code Design (AR-1)".
- `PHASE_CRITERIA`: full contents of `phase-4a-code-design.yaml`.
- `OUTPUT_TO_EVALUATE`: full current contents of every file in the editable set.
- `FILES_TO_VERIFY`: the editable set paths.
6. Parse the weighted overall (float 0.0–10.0) → `best_score`. Also capture the per-KPI
sub-scores → `best_kpis`.
7. Append the header (if the file is empty) then the baseline row (tab-separated) to
`{{RECORD_FILE}}`. AR-1 uses the SKILL.md common schema with its four AR-1 dimension
columns (`structural_health loudness_of_failures naming_clarity docs_traceability`)
between `delta` and `status` — use these exact criterion ids as the column names:
```
iter composite delta structural_health loudness_of_failures naming_clarity docs_traceability status snapshot utc
0 <best_score> 0.00 <sh> <lof> <nc> <dt> baseline 000-baseline-<TS> <UTC-iso>
```
where `<sh>/<lof>/<nc>/<dt>` are the four baseline per-KPI sub-scores from `best_kpis`.
---
## Experimentation loop (Waze loop shape)
Counter `N` = next iteration number from `{{RECORD_FILE}}`. Each iteration:
### 1. EVALUATE
Re-read the current editable set and the most recent reflection breakdown. Note `best_score`
(max `score` where `status='kept'`, else baseline) and `best_kpis`. Skim recent
`reverted` / `skipped` ideas — **do not repeat them**.
### 2. SCORE — one line of `file:line` evidence per KPI
For each of the 4 KPIs, write ONE concrete evidence line, e.g.:
```
structural_health src/auth/token.ts:88 120-line function mixes parse + validate + persist
loudness_of_failures src/auth/token.ts:142 `catch {}` swallows refresh error → silent expiry
naming_clarity src/auth/util.ts:12 `function doIt(x)` — intent unclear
docs_traceability src/auth/token.ts:1 exported `rotate()` has no doc; no task ref
```
### 3. RANK — top 5
List the top 5 candidate edits by expected KPI lift × low risk. Each must respect the
surface-lock. Drop any that need a signature/shape change (log them `skipped`).
### 4. PROPOSE
Pick the single highest-ranked candidate. State: target file:line, the KPI it lifts, the
exact change, and an explicit assertion that it does **not** touch the public surface and
introduces **no** new silent-failure path.
- If the proposed edit touches a `{{SAFETY_CRITICAL_FLOWS}}` flow → **ASK the user before
editing.** Apply pure structural/doc edits (rename local, extract private helper, add
comment/doc, remove dead code) directly without asking.
### 5. SNAPSHOT
Before editing, snapshot the current editable set (the rollback target):
```bash
SLUG="<short-kebab-of-idea>"; TS="$(date -u +%Y%m%dT%H%M%SZ)"
SNAP="{{WORKSPACE}}/.aid/pipeline/autoresearch/{{TASK_ID}}/snapshots/$(printf '%03d' $N)-pre-${SLUG}-${TS}"
mkdir -p "$SNAP" # copy every editable file into $SNAP preserving relative paths
```
### 6. IMPLEMENT — one edit
Apply exactly ONE focused edit, only within the editable set. No "while I'm here" extras.
### 7. RE-SCORE
Spawn the reflection-agent again (identical setup, `OUTPUT_TO_EVALUATE` = the edited set).
Parse the weighted overall → `official_score`, and per-KPI → `official_kpis`.
### 8. keep / revert
- **Keep** iff `official_score` **strictly improves** (`official_score > best_score`) AND
no KPI regressed below its baseline AND the surface-lock + no-new-silent-failure
invariants still hold:
- `status = kept`; `best_score = official_score`; `best_kpis = official_kpis`; leave edit on disk.
- **Else revert** by restoring the editable set from the step-5 snapshot:
- `status = reverted`; `best_score` / `best_kpis` unchanged.
### 9. LOG
Append to `{{RECORD_FILE}}` (tab-separated), matching the SKILL.md common schema with the
four AR-1 dimension columns between `delta` and `status`:
```
iter composite delta structural_health loudness_of_failures naming_clarity docs_traceability status snapshot utc
<N> <official_score> <official_score - prior best_score> <sh> <lof> <nc> <dt> <status> <snapshot dir> <UTC-iso>
```
where `<sh>/<lof>/<nc>/<dt>` are this iteration's per-KPI sub-scores from `official_kpis`.
The one-line idea is NOT a TSV column — keep it in the human chat line only.
### 10. Increment `N`, loop.
---
## STOP conditions
Stop when ANY fires:
- A human interrupts or sends a new prompt.
- **All 4 KPIs ≥ {{KPI_TARGET}}** (design target met).
- **{{AR1_MAX_REVERTS}} consecutive `reverted` iterations** (plateau).
- **{{AR1_MAX_ITERS}} total iterations** reached.
- Cost brake: `config.json` `cost_limits.max_per_run_usd` reached.
> Config bindings: `{{AR1_MAX_ITERS}}` = `config.autoresearch.ar_design.max_iterations`;
> `{{AR1_MAX_REVERTS}}` = `config.autoresearch.ar_design.max_consecutive_reverts`.
> A pass may never be bounded by the cost brake alone.
On stop, in this exact order:
1. **FIRST restore** the editable set to the best-kept version (`cp` from the snapshot of
the kept state if the last on-disk write was a revert/restore or a crash). This is itself
a source mutation that bumps the source mtime to NOW.
2. **THEN write `{{WORKSPACE}}/.aid/pipeline/state.json` UNCONDITIONALLY as the FINAL
action** — after the last source mutation of ANY kind (kept edit, revert/restore `cp`, or
crash restore). Record final per-KPI scores into `step_history` and update the step's
completion. Because a rollback `cp` sets the source mtime to NOW and the plateau stop can
fire right after a run of reverts, state.json MUST be the last thing written so its mtime
≥ newest source mtime; otherwise the dev-pipeline-gate Stop hook re-blocks. Then hand off
to TDD.
3. **Safety contract — follow `autoresearch/SKILL.md`'s loop spine; this file does NOT restate
it in full.** Restore with `cp -p` and check the exit status of EVERY restore (`test -r` the
target before any baseline fallback; on any failure HALT with status `restore-failed`). After
writing `state.json`, verify exit status + re-read/parse + mtime ≥ newest source (on failure
HALT with status `state-write-failed`). And re-verify the rubric fingerprint before each
official score (HALT `rubric-tampered` on mismatch). The ordering note above is necessary but
NOT the whole contract — the SKILL.md loop spine is authoritative for these checks.
Summarize: starting score, final `best_score`, final per-KPI, `kept` vs `reverted` counts,
top 3 kept changes by delta, and confirmation that the public surface is unchanged.
---
## Output expectations to the user
Be quiet. After each iteration print ONE line:
```
AR-1 iter <N> | <kept|reverted|skipped> | <official_score> (Δ <delta>) | KPIs S<..> L<..> N<..> D<..> | <idea>
```
Do not narrate reasoning or dump the full reflection breakdown (the user can ask). The TSV
is the source of truth.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |