Skip to content

LeoYeAI/agent-master-skills / skills/autoforge

autoforge

Flagged

AutoForge is a production-grade autonomous optimization framework for AI agents. It replaces subjective "reflection" with mathematically rigorous convergence loops β€” tracking every iteration in TSV, cross-validating with multiple models, and stopping only when pass rates confirm real improvement. Four specialized modes: prompt (skill & doc optimization via scenario simulation), code (sandboxed test execution with measurable criteria), audit (CLI verification against live tool behavior), and project (whole-repo cross-file consistency analysis). Battle-tested across 50+ iterations on production skills. Use when: user says "autoforge", "forge", "optimize skill", "improve", "run autoforge", "optimize code", "improve script", "optimize repo", "forge project", "check project", "repo audit".

FileKindSize
README.mdmarkdown16.1 KB
SKILL.mdskill md20.7 KB
_meta.jsonconfig273 B
examples/example-config.jsonconfig241 B
references/eval-examples.mdmarkdown8.8 KB
references/ml-mode.mdmarkdown2.9 KB
results/email-prompt-proposed.mdmarkdown427 B
scripts/report.shscript6.0 KB
scripts/visualize.pyscript4.7 KB
<div align="center">

# πŸ”¨ AutoForge

### *Stop vibing. Start converging.*

**Iterative optimization loops for AI agent skills, code, docs & entire repos.**

Mathematical convergence Β· Multi-model cross-validation Β· Live Unicode reporting

[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
[![OpenClaw Skill](https://img.shields.io/badge/OpenClaw-skill-brightgreen.svg)](https://openclaw.ai)
[![ClawHub](https://img.shields.io/badge/ClawHub-autoforge-8B5CF6.svg)](https://clawhub.com/skills/autoforge)

</div>

---

Most "self-improving agent" approaches boil down to *"reflect on your output."*
That's a vibe check, not optimization. AutoForge is different.

|  | Typical "Reflect" | AutoForge |
|---|---|---|
| **When to stop** | "Looks good to me" | 3Γ— 100% pass, 5Γ— retained, or 3Γ— discard |
| **Progress** | Chat history | TSV with pass rates & diffs per iteration |
| **Validation** | Same model checks itself | Multi-model cross-validation |
| **Reporting** | Final summary | Live Unicode bars after every iteration |
| **Modes** | One generic loop | 4 specialized modes |
| **Track record** | Demo | 50+ iterations across 6 production skills |

---

## πŸ—οΈ Architecture

### Core Loop

```
                         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                         β”‚  Define target +     β”‚
                         β”‚  evals               β”‚
                         β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                   β”‚
                                   β–Ό
                         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                         β”‚  Baseline scan       β”‚
                         β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                   β”‚
                    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                    β”‚
                    β–Ό
           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”     proposed     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
       β”Œβ”€β”€β–Άβ”‚   Optimizer   │────────────────▢│   Validator   β”‚
       β”‚   β”‚  (Claude Opus)β”‚                  β”‚    (GPT-5)    β”‚
       β”‚   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜                  β””β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜
       β”‚                                              β”‚
       β”‚                              β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
       β”‚                              β”‚                               β”‚
       β”‚                              β–Ό                               β–Ό
       β”‚                     β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”                 β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
       β”‚                     β”‚ βœ… improved  β”‚                 β”‚ ❌ discarded β”‚
       β”‚                     β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜                 β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜
       β”‚                            β”‚                               β”‚
       β”‚                            β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”˜
       β”‚                                        β–Ό             β”‚
       β”‚                               β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”       β”‚
       β”‚                               β”‚  Log to TSV  β”‚       β”‚
       β”‚                               β”‚  + report.sh β”‚       β”‚
       β”‚                               β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”˜       β”‚
       β”‚                                      β”‚              β”‚
       β”‚                                      β–Ό              β”‚
       β”‚                              β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”       β”‚
       β”‚          No                  β”‚  Converged?  β”‚       β”‚
       └───────────────────────────────              β”‚       β”‚
                                      β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜       β”‚
                                             β”‚               β”‚
                         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                         β”‚                   β”‚               β”‚
                         β–Ό                   β–Ό               β–Ό
                  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”   β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                  β”‚ 3Γ— 100% βœ…  β”‚   β”‚ 5Γ— retained  β”‚  β”‚ 3Γ— discardβ”‚
                  β”‚   Deploy!   β”‚   β”‚  Converged   β”‚  β”‚   Stop ⚠️  β”‚
                  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜   β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
```

### Multi-Model Validation

```
  Iter 1 (Optimizer)          Iter 2 (Validator)         Iter 3 (Optimizer)
  ─────────────────           ──────────────────         ─────────────────

  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”            β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  β”‚ Claude Opus  β”‚            β”‚    GPT-5     β”‚           β”‚ Claude Opus  β”‚
  β”‚              β”‚            β”‚              β”‚           β”‚              β”‚
  β”‚ Analyze      β”‚            β”‚ Blind review β”‚           β”‚ Fix findings β”‚
  β”‚ Find issues  β”‚            β”‚ of output    β”‚           β”‚ from GPT-5   β”‚
  β”‚ Write fixes  β”‚            β”‚ (no context) β”‚           β”‚              β”‚
  β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜            β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜           β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜
         β”‚                           β”‚                          β”‚
         β–Ό                           β–Ό                          β–Ό
  pass_rate: 62%              pass_rate: 78%              pass_rate: 95%
  status: improved            status: improved            status: improved

         └────── TSV ──────────────── TSV ──────────────── TSV β”€β”€β”€β”€β”€β”€β”˜

  Different model validates β†’ no "grading your own homework" blind spot
```

### Project Mode β€” Three Phases

```
  Phase 1                    Phase 2                      Phase 3
  SCAN & PLAN                CROSS-FILE ANALYSIS          ITERATIVE FIX LOOP
  ─────────────              ───────────────────          ──────────────────

  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”           β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”         β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  β”‚ Walk repo    β”‚           β”‚ README ↔ CLI     β”‚    β”Œβ”€β”€β”€β–Άβ”‚ Surgical fix β”‚
  β”‚ tree         β”‚           β”‚ Dockerfile ↔ depsβ”‚    β”‚    β”‚ across files β”‚
  β”‚              │──────────▢│ CI ↔ scripts     │───▢│    β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜
  β”‚ Build file   β”‚           β”‚ .env ↔ code refs β”‚    β”‚           β”‚
  β”‚ priority map β”‚           β”‚ imports ↔ reqs   β”‚    β”‚           β–Ό
  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜           β”‚ .gitignore ↔ out β”‚    β”‚    β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                             β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜    β”‚    β”‚ Validate     β”‚
                                                     β”‚    β”‚ consistency  β”‚
                                                     β”‚    β””β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”˜
                                                     β”‚           β”‚
                                                     β”‚     Not   β”‚  Done?
                                                     └─── yet β—€β”€β”€β”˜
```

---

## πŸš€ Quick Start

**Install**

```bash
# Via ClawHub
clawhub install autoforge

# Or clone
git clone https://github.com/akrimm702/autoforge.git
cp -r autoforge ~/.openclaw/workspace/skills/autoforge
```

**Configure reporting** *(optional)*

```bash
export AF_CHANNEL="telegram"        # telegram | discord | slack
export AF_CHAT_ID="-100XXXXXXXXXX"  # chat/group ID
export AF_TOPIC_ID="1234"           # thread ID (optional)
```

No env vars? Reports print to stdout with ANSI colors.

**Tell your agent**

```
Start autoforge mode: prompt for the coding-agent skill.
Evals: PTY handling correct? Workspace protection enforced? Clear structure?
```

The agent reads the skill, runs the loop, tracks everything in TSV, reports live, and stops when convergence math says it's done.

---

## πŸ”§ Four Modes

### `prompt` β€” Mental Simulation

Simulates 5 realistic scenarios per iteration, evaluates Yes/No against defined evals, calculates pass rate mathematically. No code execution.

**Best for:** SKILL.md files, prompt engineering, documentation, briefing templates.

### `code` β€” Real Execution

Runs code in a sandbox, measures exit codes, stdout, stderr, runtime. Evaluates against concrete test criteria.

**Best for:** Shell scripts, Python tools, data pipelines, build systems.

### `audit` β€” CLI Testing

Tests documented commands against actual CLI behavior (`--help`, read-only). Catches docs-vs-reality drift. Two variants: Simple (2 iterations) or Deep (iterative with multi-model).

**Best for:** Verifying skill documentation matches real CLI behavior.

### `project` β€” Whole Repository ⭐

Scans an entire repo, builds a file-map with priorities, runs **cross-file consistency checks**, and iteratively fixes issues across multiple files.

**Best for:** README ↔ CLI drift, Dockerfile ↔ dependency mismatches, CI ↔ project structure gaps.

Cross-file checks include:

- README documents what the CLI actually does
- Dockerfile installs the right dependency versions
- CI workflows reference correct paths and scripts
- `.env.example` covers all env vars used in code
- Every import has a matching dependency declaration
- `.gitignore` excludes build artifacts and secrets

---

## πŸ“Š Live Reporting

After each iteration, `report.sh` sends live updates:

```
πŸ“Š AutoForge: coding-agent

πŸ“ Iter 1   β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘  45%
βœ… Iter 2   β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘β–‘β–‘β–‘  62%
βœ… Iter 3   β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘β–‘β–‘  78%
βœ… Iter 4   β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘β–‘  85%
βœ… Iter 5   β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘  90%
βœ… Iter 6   β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘  92%
βœ… Iter 7   β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘β–‘  92%
βœ… Iter 8   β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘  95%
βœ… Iter 9   β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–‘  95%
βœ… Iter 10  β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ  100%

──────────────────────
Iterations: 10  βœ… Keep: 10  ❌ Discard: 0
πŸ† Best pass rate: 100% (Iter 10)

βœ… Loop converged β€” improvement found
```

Every iteration is tracked in TSV:

```
iteration  prompt_version_summary       pass_rate  change_description              status
1          Baseline                     45%        Original SKILL.md baseline      baseline
2          Add missing subcommands      62%        16 Codex subcommands added      improved
3          Fix approval flags           78%        Scoped flags by context         improved
...
10         Validation pass              100%       All checks green                improved
```

---

## πŸ“ Convergence Rules

No vibes. No "looks good." Mathematical stop conditions:

| Condition | Rule | Purpose |
|-----------|------|---------|
| ⬇️ Minimum iters | Must reach N before any stop | Prevents premature convergence |
| πŸ›‘ Max 30 iters | Hard safety cap | Cost protection |
| ❌ 3Γ— discard streak | Stop + analyze | Detects structural problems |
| βœ… 3Γ— 100% pass | Confirmed perfect | After minimum reached |
| ➑️ 5Γ— retained streak | Fully converged | No further improvement possible |

**Validator noise detection:** In multi-model setups, validators can produce false positives. AutoForge recognizes config/path confusion, inverted checks, normal English flagged as forbidden references, and over-counting. After all real fixes, if >3 discards stem from non-reproducible complaints β†’ declare convergence.

---

## πŸ”€ Multi-Model Cross-Validation

For complex audits, split optimizer and validator across different models:

| Role | Example Models | Task |
|------|---------------|------|
| **Optimizer** | Claude Opus, GPT-4.1 | Finds issues, writes fixes |
| **Validator** | GPT-5, Gemini | Checks against ground truth independently |

The validator doesn't see the optimizer's reasoning β€” just the output. This prevents the "same model validates its own work" blind spot.

---

## πŸ† Real-World Results

*Production runs, not demos.*

**coding-agent SKILL.md** β€” 553 lines rewritten across 10 iterations. 16 Codex subcommands + 40 Claude CLI flags documented. 45% β†’ 100%. Discovered `--yolo` was never a real flag.

**ACP Router** β€” 90% β†’ 100% in 9 iterations. Agent coverage doubled from 6 to 12 harnesses. Thread spawn recovery policy written from scratch.

**Sub-Agents Documentation** β€” 70% β†’ 100% in 14 iterations with multi-model validation. 6 real bugs found in upstream docs. Identified 4 categories of validator false positives.

**backup.sh** β€” Added rsync support, validation checks, restore-test. Code mode with sandboxed test runs. 3 iterations to stable, 2 more to polish.

**AutoForge on itself 🀯** β€” Self-forged in project mode: 67% β†’ 100% in 8 iterations. Fixed 2 script bugs, cleaned config/doc inconsistencies across 7 files.

---

## πŸ—‚οΈ Directory Structure

```
autoforge/
β”œβ”€β”€ SKILL.md                ← OpenClaw skill definition
β”œβ”€β”€ README.md               ← You are here
β”œβ”€β”€ LICENSE                 ← MIT
β”œβ”€β”€ .gitignore
β”œβ”€β”€ scripts/
β”‚   β”œβ”€β”€ report.sh           ← Live reporting (channel or stdout)
β”‚   └── visualize.py        ← PNG progress chart generator
β”œβ”€β”€ references/
β”‚   β”œβ”€β”€ eval-examples.md    ← 200+ pre-built evals by category
β”‚   └── ml-mode.md          ← ML training integration guide
β”œβ”€β”€ examples/
β”‚   β”œβ”€β”€ demo-results.tsv    ← Sample iteration data
β”‚   └── example-config.json ← Reference template
└── results/                ← Your run data (gitignored)
    └── .gitkeep
```

---

## βš™οΈ Configuration

AutoForge is configured entirely via environment variables. No config file needed.

| Variable | Default | Description |
|----------|---------|-------------|
| `AF_CHANNEL` | `telegram` | Report delivery channel |
| `AF_CHAT_ID` | *(none)* | Chat/group ID. Unset = stdout |
| `AF_TOPIC_ID` | *(none)* | Thread/topic ID |

| Flag | Behavior |
|------|----------|
| `--dry-run` *(default)* | Only TSV + proposed files. Target unchanged. |
| `--live` | Overwrites target. Auto-backup to `results/backups/`. |
| `--resume` | Continue from existing TSV. |

---

## 🀝 Contributing

1. Fork β†’ feature branch β†’ PR
2. Run `shellcheck scripts/report.sh` and `python3 -m py_compile scripts/visualize.py`
3. Include real-world results from your own runs if possible

**Good contributions:** new eval templates, additional channel support, bug fixes with repro steps, production run case studies.

---

## License

MIT β€” see [LICENSE](LICENSE).

---

<div align="center">

Built by [Alexander Krimm](https://github.com/akrimm702).

Battle-tested across **50+ iterations** on **6 production skills**.

*Stop reflecting. Start forging.* πŸ”¨

</div>
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall