Skip to content

mxyhi/ok-skills / autoresearch

autoresearch

Flagged

Autonomous iteration loop: modify, verify, keep/discard against any metric

FileKindSize
LICENSEother1.0 KB
SKILL.mdskill md7.8 KB
agents/openai.yamlconfig250 B
autoresearch.mdmarkdown4.8 KB
debug.mdmarkdown4.0 KB
evals.mdmarkdown5.0 KB
fix.mdmarkdown4.3 KB
improve.mdmarkdown6.0 KB
learn.mdmarkdown7.9 KB
plan.mdmarkdown2.8 KB
predict.mdmarkdown3.7 KB
probe.mdmarkdown4.8 KB
reason.mdmarkdown4.7 KB
references/orchestrator-routing.mdmarkdown6.8 KB
references/predict-personas.mdmarkdown3.1 KB
references/reason-judge-protocol.mdmarkdown3.4 KB
references/security-checklist.mdmarkdown3.4 KB
regression.mdmarkdown9.3 KB
scenario.mdmarkdown4.1 KB
scripts/orchestrate.shexecutablescript18.5 KB
scripts/score-regression.shexecutablescript7.5 KB
security.mdmarkdown4.6 KB
ship.mdmarkdown3.8 KB
---
name: autoresearch:evals
description: "Analyze iteration results: trends, plateaus, regressions, recommendations"
argument-hint: "[path/to/results.tsv] [--format text|json|md]"
---

EXECUTE IMMEDIATELY.

## Parse Arguments

Extract from $ARGUMENTS:
- Positional path to a specific TSV file
- `--format` — output format: text (default console), json, md (markdown file)
- `--compare <path>` — (v2.2.0 placeholder, not yet implemented)

## Input Discovery

1. If path provided → use that TSV directly
2. If no path → scan current directory + `autoresearch/*/` for `*-results.tsv` files
3. If multiple found → request_user_input: "Which results to analyze?" — list found files
4. If none found → request_user_input: "Provide path to results TSV"
5. Also scan project root for v2.0.03 legacy TSV files (backward compat)

## Parse TSV

1. Read line 1: extract `# metric_direction: higher_is_better|lower_is_better` comment
   - If missing → infer from column names (metric/error_count → guess, or ask user)
2. Read line 2: header row → detect available columns
3. Read remaining lines: data rows
4. Handle missing `timestamp` column gracefully (v2.0.03 compat)

## Column Detection & Analysis

Activate analysis based on columns present in header:

| Column | Analysis |
|---|---|
| `metric` | Trend direction, plateau detection (3+ flat iterations), diminishing returns, biggest single-iteration jumps |
| `delta` | Per-iteration efficiency, cumulative improvement, effort-to-gain ratio |
| `status` | Keep/discard rate, crash frequency, success streaks, failure clusters, longest winning streak |
| `guard` + `guard-metric` | Guard failure rate, metric-improved-but-guard-failed analysis |
| `severity` | Severity distribution (critical/high/medium/low/info), critical discovery rate per iteration |
| `hypothesis` + `status` | Confirmation rate, investigation efficiency, most productive techniques |
| `commit` | File hotspot analysis (cross-ref with `git diff` for kept commits), change size correlation |
| `technique` | Technique effectiveness ranking |
| `dimension` | Dimension coverage completeness (X/12) |
| `candidate_label` + `judge_verdict` | Convergence speed, oscillation count |
| `error_type` | Error category distribution, fix rate per category |
| `classification` | New vs extension vs duplicate ratio, saturation curve |
| `convergence_count` | Convergence trajectory |

Unknown columns: report presence but skip analysis. Forward-compatible with future subcommands.

## Report Structure

```
## Evals Summary — {subcommand} ({N} iterations)

### Key Metrics
- Total iterations: N | Kept: X | Reverted: Y | Revert rate: Z%
- Starting metric: A | Final metric: B | Improvement: C%

### Trend Analysis
- Metric progression: [description of trajectory]
- Plateau detected at iteration N (metric stable for M iterations)
- Biggest win: iteration X (+delta, description)
- Biggest loss: iteration Y (-delta, description)
- Diminishing returns: [after iteration N, average delta dropped below threshold]

### Patterns
- What types of changes succeeded: [extracted from descriptions of kept iterations]
- What types of changes failed: [extracted from descriptions of discarded iterations]
- File hotspots: [files changed most in kept iterations, if commit data available]
- Technique effectiveness: [ranked by confirmation rate, if technique column present]

### Recommendation
- [continue / stop / change strategy — based on trend, plateau, revert rate]
- [specific actionable suggestion based on pattern analysis]
```

## Output

- Console: structured report (30-50 lines)
- If `--format md` → write `evals-summary.md` in same directory as input TSV
- If `--format json` → write `evals-summary.json` with structured data

## Mid-Loop Checkpoint Protocol (for --evals flag in other commands)

This section documents the checkpoint protocol that looping commands embed:

- **Adaptive interval:** `floor(max_iterations / 3)`, minimum 1. Fixed 10 for unbounded. Override: `--evals-interval N`.
- **Checkpoint format (5 lines max):**
  ```
  --- Eval Checkpoint (iterations {X}-{Y}) ---
  Metric: {start} → {end} ({delta}) | Kept: {n}/{total} | Trend: {up/flat/down}
  {one-line recommendation}
  ---
  ```
- **Early stop recommendation:** if plateau detected for 3+ consecutive checkpoints
- **Final summary:** at loop end, produce full evals report to console + evals-summary.md

### Adaptive Interval Examples

| Subcommand | Default Iterations | Interval | Checkpoints At |
|---|---|---|---|
| reason | 8 | 2 | 2, 4, 6, 8 |
| learn | 10 | 3 | 3, 6, 9, final |
| debug/security | 15 | 5 | 5, 10, 15 |
| fix/scenario | 20 | 6 | 6, 12, 18, final |
| core | 25 | 8 | 8, 16, 24, final |
| unbounded | unlimited | 10 | every 10 |

## Backward Compatibility

- v2.0.03 TSV files: column names preserved, `timestamp` absence handled gracefully
- Fuzzy column matching: `metric_value` → `metric`, `error_count` → `metric`
- Files in project root (not `autoresearch/` subdirectory) → discovered during scan
- v2.0.03 status values all supported: baseline, keep, keep (reworked), discard, crash, no-op, hook-blocked, metric-error
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 196kUpdated Claude CodeCodexVerifiedInstall