LeoYeAI/agent-master-skills / skills/autoforge
autoforge
FlaggedAutoForge is a production-grade autonomous optimization framework for AI agents. It replaces subjective "reflection" with mathematically rigorous convergence loops β tracking every iteration in TSV, cross-validating with multiple models, and stopping only when pass rates confirm real improvement. Four specialized modes: prompt (skill & doc optimization via scenario simulation), code (sandboxed test execution with measurable criteria), audit (CLI verification against live tool behavior), and project (whole-repo cross-file consistency analysis). Battle-tested across 50+ iterations on production skills. Use when: user says "autoforge", "forge", "optimize skill", "improve", "run autoforge", "optimize code", "improve script", "optimize repo", "forge project", "check project", "repo audit".
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 16.1 KB |
| SKILL.md | skill md | 20.7 KB |
| _meta.json | config | 273 B |
| examples/example-config.json | config | 241 B |
| references/eval-examples.md | markdown | 8.8 KB |
| references/ml-mode.md | markdown | 2.9 KB |
| results/email-prompt-proposed.md | markdown | 427 B |
| scripts/report.sh | script | 6.0 KB |
| scripts/visualize.py | script | 4.7 KB |
README.md
View on GitHub<div align="center">
# π¨ AutoForge
### *Stop vibing. Start converging.*
**Iterative optimization loops for AI agent skills, code, docs & entire repos.**
Mathematical convergence Β· Multi-model cross-validation Β· Live Unicode reporting
[](LICENSE)
[](https://openclaw.ai)
[](https://clawhub.com/skills/autoforge)
</div>
---
Most "self-improving agent" approaches boil down to *"reflect on your output."*
That's a vibe check, not optimization. AutoForge is different.
| | Typical "Reflect" | AutoForge |
|---|---|---|
| **When to stop** | "Looks good to me" | 3Γ 100% pass, 5Γ retained, or 3Γ discard |
| **Progress** | Chat history | TSV with pass rates & diffs per iteration |
| **Validation** | Same model checks itself | Multi-model cross-validation |
| **Reporting** | Final summary | Live Unicode bars after every iteration |
| **Modes** | One generic loop | 4 specialized modes |
| **Track record** | Demo | 50+ iterations across 6 production skills |
---
## ποΈ Architecture
### Core Loop
```
βββββββββββββββββββββββ
β Define target + β
β evals β
βββββββββββ¬ββββββββββββ
β
βΌ
βββββββββββββββββββββββ
β Baseline scan β
βββββββββββ¬ββββββββββββ
β
ββββββββββββββββ
β
βΌ
βββββββββββββββββ proposed βββββββββββββββββ
ββββΆβ Optimizer ββββββββββββββββββΆβ Validator β
β β (Claude Opus)β β (GPT-5) β
β βββββββββββββββββ βββββββββ¬ββββββββ
β β
β βββββββββββββββββ΄ββββββββββββββββ
β β β
β βΌ βΌ
β βββββββββββββββ βββββββββββββββ
β β β
improved β β β discarded β
β ββββββββ¬βββββββ ββββββββ¬βββββββ
β β β
β βββββββββββββ¬ββββββββββββββ¬ββββββ
β βΌ β
β βββββββββββββββ β
β β Log to TSV β β
β β + report.sh β β
β ββββββββ¬βββββββ β
β β β
β βΌ β
β ββββββββββββββββ β
β No β Converged? β β
ββββββββββββββββββββββββββββββββ€ β β
ββββββββ¬ββββββββ β
β β
βββββββββββββββββββββΌββββββββββββββββ
β β β
βΌ βΌ βΌ
βββββββββββββββ ββββββββββββββββ βββββββββββββ
β 3Γ 100% β
β β 5Γ retained β β 3Γ discardβ
β Deploy! β β Converged β β Stop β οΈ β
βββββββββββββββ ββββββββββββββββ βββββββββββββ
```
### Multi-Model Validation
```
Iter 1 (Optimizer) Iter 2 (Validator) Iter 3 (Optimizer)
βββββββββββββββββ ββββββββββββββββββ βββββββββββββββββ
ββββββββββββββββ ββββββββββββββββ ββββββββββββββββ
β Claude Opus β β GPT-5 β β Claude Opus β
β β β β β β
β Analyze β β Blind review β β Fix findings β
β Find issues β β of output β β from GPT-5 β
β Write fixes β β (no context) β β β
ββββββββ¬ββββββββ ββββββββ¬ββββββββ ββββββββ¬ββββββββ
β β β
βΌ βΌ βΌ
pass_rate: 62% pass_rate: 78% pass_rate: 95%
status: improved status: improved status: improved
βββββββ TSV ββββββββββββββββ TSV ββββββββββββββββ TSV βββββββ
Different model validates β no "grading your own homework" blind spot
```
### Project Mode β Three Phases
```
Phase 1 Phase 2 Phase 3
SCAN & PLAN CROSS-FILE ANALYSIS ITERATIVE FIX LOOP
βββββββββββββ βββββββββββββββββββ ββββββββββββββββββ
ββββββββββββββββ ββββββββββββββββββββ ββββββββββββββββ
β Walk repo β β README β CLI β βββββΆβ Surgical fix β
β tree β β Dockerfile β depsβ β β across files β
β ββββββββββββΆβ CI β scripts βββββΆβ ββββββββ¬ββββββββ
β Build file β β .env β code refs β β β
β priority map β β imports β reqs β β βΌ
ββββββββββββββββ β .gitignore β out β β ββββββββββββββββ
ββββββββββββββββββββ β β Validate β
β β consistency β
β ββββββββ¬ββββββββ
β β
β Not β Done?
ββββ yet ββββ
```
---
## π Quick Start
**Install**
```bash
# Via ClawHub
clawhub install autoforge
# Or clone
git clone https://github.com/akrimm702/autoforge.git
cp -r autoforge ~/.openclaw/workspace/skills/autoforge
```
**Configure reporting** *(optional)*
```bash
export AF_CHANNEL="telegram" # telegram | discord | slack
export AF_CHAT_ID="-100XXXXXXXXXX" # chat/group ID
export AF_TOPIC_ID="1234" # thread ID (optional)
```
No env vars? Reports print to stdout with ANSI colors.
**Tell your agent**
```
Start autoforge mode: prompt for the coding-agent skill.
Evals: PTY handling correct? Workspace protection enforced? Clear structure?
```
The agent reads the skill, runs the loop, tracks everything in TSV, reports live, and stops when convergence math says it's done.
---
## π§ Four Modes
### `prompt` β Mental Simulation
Simulates 5 realistic scenarios per iteration, evaluates Yes/No against defined evals, calculates pass rate mathematically. No code execution.
**Best for:** SKILL.md files, prompt engineering, documentation, briefing templates.
### `code` β Real Execution
Runs code in a sandbox, measures exit codes, stdout, stderr, runtime. Evaluates against concrete test criteria.
**Best for:** Shell scripts, Python tools, data pipelines, build systems.
### `audit` β CLI Testing
Tests documented commands against actual CLI behavior (`--help`, read-only). Catches docs-vs-reality drift. Two variants: Simple (2 iterations) or Deep (iterative with multi-model).
**Best for:** Verifying skill documentation matches real CLI behavior.
### `project` β Whole Repository β
Scans an entire repo, builds a file-map with priorities, runs **cross-file consistency checks**, and iteratively fixes issues across multiple files.
**Best for:** README β CLI drift, Dockerfile β dependency mismatches, CI β project structure gaps.
Cross-file checks include:
- README documents what the CLI actually does
- Dockerfile installs the right dependency versions
- CI workflows reference correct paths and scripts
- `.env.example` covers all env vars used in code
- Every import has a matching dependency declaration
- `.gitignore` excludes build artifacts and secrets
---
## π Live Reporting
After each iteration, `report.sh` sends live updates:
```
π AutoForge: coding-agent
π Iter 1 ββββββββββββββββββββ 45%
β
Iter 2 ββββββββββββββββββββ 62%
β
Iter 3 ββββββββββββββββββββ 78%
β
Iter 4 ββββββββββββββββββββ 85%
β
Iter 5 ββββββββββββββββββββ 90%
β
Iter 6 ββββββββββββββββββββ 92%
β
Iter 7 ββββββββββββββββββββ 92%
β
Iter 8 ββββββββββββββββββββ 95%
β
Iter 9 ββββββββββββββββββββ 95%
β
Iter 10 ββββββββββββββββββββ 100%
ββββββββββββββββββββββ
Iterations: 10 β
Keep: 10 β Discard: 0
π Best pass rate: 100% (Iter 10)
β
Loop converged β improvement found
```
Every iteration is tracked in TSV:
```
iteration prompt_version_summary pass_rate change_description status
1 Baseline 45% Original SKILL.md baseline baseline
2 Add missing subcommands 62% 16 Codex subcommands added improved
3 Fix approval flags 78% Scoped flags by context improved
...
10 Validation pass 100% All checks green improved
```
---
## π Convergence Rules
No vibes. No "looks good." Mathematical stop conditions:
| Condition | Rule | Purpose |
|-----------|------|---------|
| β¬οΈ Minimum iters | Must reach N before any stop | Prevents premature convergence |
| π Max 30 iters | Hard safety cap | Cost protection |
| β 3Γ discard streak | Stop + analyze | Detects structural problems |
| β
3Γ 100% pass | Confirmed perfect | After minimum reached |
| β‘οΈ 5Γ retained streak | Fully converged | No further improvement possible |
**Validator noise detection:** In multi-model setups, validators can produce false positives. AutoForge recognizes config/path confusion, inverted checks, normal English flagged as forbidden references, and over-counting. After all real fixes, if >3 discards stem from non-reproducible complaints β declare convergence.
---
## π Multi-Model Cross-Validation
For complex audits, split optimizer and validator across different models:
| Role | Example Models | Task |
|------|---------------|------|
| **Optimizer** | Claude Opus, GPT-4.1 | Finds issues, writes fixes |
| **Validator** | GPT-5, Gemini | Checks against ground truth independently |
The validator doesn't see the optimizer's reasoning β just the output. This prevents the "same model validates its own work" blind spot.
---
## π Real-World Results
*Production runs, not demos.*
**coding-agent SKILL.md** β 553 lines rewritten across 10 iterations. 16 Codex subcommands + 40 Claude CLI flags documented. 45% β 100%. Discovered `--yolo` was never a real flag.
**ACP Router** β 90% β 100% in 9 iterations. Agent coverage doubled from 6 to 12 harnesses. Thread spawn recovery policy written from scratch.
**Sub-Agents Documentation** β 70% β 100% in 14 iterations with multi-model validation. 6 real bugs found in upstream docs. Identified 4 categories of validator false positives.
**backup.sh** β Added rsync support, validation checks, restore-test. Code mode with sandboxed test runs. 3 iterations to stable, 2 more to polish.
**AutoForge on itself π€―** β Self-forged in project mode: 67% β 100% in 8 iterations. Fixed 2 script bugs, cleaned config/doc inconsistencies across 7 files.
---
## ποΈ Directory Structure
```
autoforge/
βββ SKILL.md β OpenClaw skill definition
βββ README.md β You are here
βββ LICENSE β MIT
βββ .gitignore
βββ scripts/
β βββ report.sh β Live reporting (channel or stdout)
β βββ visualize.py β PNG progress chart generator
βββ references/
β βββ eval-examples.md β 200+ pre-built evals by category
β βββ ml-mode.md β ML training integration guide
βββ examples/
β βββ demo-results.tsv β Sample iteration data
β βββ example-config.json β Reference template
βββ results/ β Your run data (gitignored)
βββ .gitkeep
```
---
## βοΈ Configuration
AutoForge is configured entirely via environment variables. No config file needed.
| Variable | Default | Description |
|----------|---------|-------------|
| `AF_CHANNEL` | `telegram` | Report delivery channel |
| `AF_CHAT_ID` | *(none)* | Chat/group ID. Unset = stdout |
| `AF_TOPIC_ID` | *(none)* | Thread/topic ID |
| Flag | Behavior |
|------|----------|
| `--dry-run` *(default)* | Only TSV + proposed files. Target unchanged. |
| `--live` | Overwrites target. Auto-backup to `results/backups/`. |
| `--resume` | Continue from existing TSV. |
---
## π€ Contributing
1. Fork β feature branch β PR
2. Run `shellcheck scripts/report.sh` and `python3 -m py_compile scripts/visualize.py`
3. Include real-world results from your own runs if possible
**Good contributions:** new eval templates, additional channel support, bug fixes with repro steps, production run case studies.
---
## License
MIT β see [LICENSE](LICENSE).
---
<div align="center">
Built by [Alexander Krimm](https://github.com/akrimm702).
Battle-tested across **50+ iterations** on **6 production skills**.
*Stop reflecting. Start forging.* π¨
</div>
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |