LeoYeAI/agent-master-skills / skills/autoforge
autoforge
FlaggedAutoForge is a production-grade autonomous optimization framework for AI agents. It replaces subjective "reflection" with mathematically rigorous convergence loops — tracking every iteration in TSV, cross-validating with multiple models, and stopping only when pass rates confirm real improvement. Four specialized modes: prompt (skill & doc optimization via scenario simulation), code (sandboxed test execution with measurable criteria), audit (CLI verification against live tool behavior), and project (whole-repo cross-file consistency analysis). Battle-tested across 50+ iterations on production skills. Use when: user says "autoforge", "forge", "optimize skill", "improve", "run autoforge", "optimize code", "improve script", "optimize repo", "forge project", "check project", "repo audit".
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 16.1 KB |
| SKILL.md | skill md | 20.7 KB |
| _meta.json | config | 273 B |
| examples/example-config.json | config | 241 B |
| references/eval-examples.md | markdown | 8.8 KB |
| references/ml-mode.md | markdown | 2.9 KB |
| results/email-prompt-proposed.md | markdown | 427 B |
| scripts/report.sh | script | 6.0 KB |
| scripts/visualize.py | script | 4.7 KB |
references/ml-mode.md
View on GitHub# ML Mode — Advanced Usage
Autonomous ML training: the agent modifies `train.py`, trains for N minutes, checks `val_bpb`, keeps or discards.
This mode extends autoforge beyond prompt/code/audit into real machine learning experimentation. The same TSV tracking, stop conditions, and reporting infrastructure applies.
## Prerequisites
- macOS with Apple Silicon (M1/M2/M3/M4) or Linux with GPU
- Python 3.10+ with `uv` package manager
- Git for experiment version control
## Setup (one-time)
```bash
# 1. Clone the macOS-optimized fork
git clone https://github.com/miolini/autoresearch-macos
cd autoresearch-macos
# 2. Install uv (if not present)
curl -LsSf https://astral.sh/uv/install.sh | sh
# 3. Install dependencies
uv sync
# 4. Prepare data (~2 min, one-time)
uv run prepare.py
```
## Starting an Experiment
```bash
# Create branch for this run
git checkout -b autoresearch/$(date +%b%d | tr '[:upper:]' '[:lower:]')
# Initialize results tracking
echo -e "commit\tval_bpb\tmemory_gb\tstatus\tdescription" > results.tsv
# Point your coding agent at the repo:
# "Read program.md and start a new experiment loop."
```
## Agent Instructions
```
Read program.md in this repo and start the experiment loop.
- Only modify train.py
- Execute each run with: uv run train.py > run.log 2>&1
- Extract metric: grep "^val_bpb:" run.log
- Log result in results.tsv
- On improvement (lower val_bpb): keep git commit
- On equal or worse result: git reset back
- LOOP FOREVER — do not ask whether to continue
```
## Expected Performance
| Hardware | Experiments/Hour |
|----------|-----------------|
| H100 | ~12 |
| M2/M3/M4 Max | ~2–4 |
| M1/M2 | ~1–2 |
## Hyperparameter Tips for Mac (smaller models)
For Apple Silicon, start with conservative settings:
- Dataset: TinyStories instead of full dataset
- `vocab_size`: 2048–4096 instead of 8192
- `DEPTH`: 4 instead of 8
- `MAX_SEQ_LEN`: 256–512
- `WINDOW_PATTERN`: only "L" (no "SSSL")
- `TOTAL_BATCH_SIZE`: 2^14 (~16K)
## Viewing Results
```bash
# Pretty-print results
cat results.tsv | column -t -s $'\t'
# Generate progress chart
python3 scripts/visualize.py results.tsv --title "ML Experiment"
```
## Preventing Sleep During Training
```bash
# macOS: prevent sleep while loop runs
caffeinate -i &
CAFE_PID=$!
# After the run: kill $CAFE_PID
# Linux: use systemd-inhibit or screen/tmux
systemd-inhibit --what=idle uv run train.py
```
## Integration with AutoForge
ML mode integrates with the standard autoforge infrastructure:
1. **TSV tracking** — Same format, `val_bpb` maps to `pass_rate` (inverted: lower is better)
2. **Reporting** — `report.sh` works unchanged, showing progress bars
3. **Stop conditions** — Same convergence rules apply (adapt for minimization)
4. **Visualization** — `visualize.py` charts the training curve
To adapt stop conditions for minimization (lower = better):
- `improved` = val_bpb is **lower** than previous best
- `retained` = val_bpb is equal
- `discard` = val_bpb is higher
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |