gepa-ai/gepa / .claude/skills/gepa-optimize-anything
gepa-optimize-anything
FlaggedAutomatically improve any text artifact that can be scored — prompts, programs/code, configs, specs, regex/SQL/schemas, agent scaffolds, or encoded search solutions — with optimize_anything, an LLM-driven optimization API: a search backend repeatedly proposes better versions from execution feedback and an evaluator you provide assigns the score (an objective metric or an LLM-as-judge for subjective tasks). optimize_anything is one interface across several optimizer backends — GEPA (reflective evolutionary search, the default), AutoResearch, MetaHarness — plus a best-of-N baseline: write the evaluator once and switch backend with a single argument. Use whenever a task needs auto-optimizing, tuning, or searching over text with any quality signal (accuracy, pass rate, latency, cost, judge rating); when building an evaluator + proposer loop; when running GEPA; or when comparing optimizers. The candidate can be any string an evaluator can grade, scored on one problem or generalized across many.
| File | Kind | Size |
|---|---|---|
| .claude-plugin/plugin.json | config | 686 B |
| SKILL.md | skill md | 18.0 KB |
| references/api.md | markdown | 17.8 KB |
| references/gotchas.md | markdown | 8.1 KB |
| references/tracking.md | markdown | 2.8 KB |
| references/writing_evaluators.md | markdown | 7.4 KB |
| scripts/preflight.py | script | 4.8 KB |
SKILL.md
View on GitHub---
name: gepa-optimize-anything
description: >-
Automatically improve any text artifact that can be scored — prompts, programs/code, configs,
specs, regex/SQL/schemas, agent scaffolds, or encoded search solutions — with optimize_anything,
an LLM-driven optimization API: a search backend repeatedly proposes better versions from
execution feedback and an evaluator you provide assigns the score (an objective metric or an
LLM-as-judge for subjective tasks). optimize_anything is one interface across several optimizer
backends — GEPA (reflective evolutionary search, the default), AutoResearch, MetaHarness — plus a
best-of-N baseline: write the evaluator once and switch backend with a single argument.
Use whenever a task needs auto-optimizing, tuning, or searching over text with any quality signal
(accuracy, pass rate, latency, cost, judge rating); when building an evaluator + proposer loop;
when running GEPA; or when comparing optimizers. The candidate can be any string an evaluator can
grade, scored on one problem or generalized across many.
---
# `optimize_anything`
**Naming, precisely.** `optimize_anything` is the tool: a general API for optimizing text
artifacts. **GEPA** is one specific optimizer behind it — reflective evolutionary search, the
default backend (`engine="gepa"`) — and, for legacy reasons, also the name of the Python package
that ships all of this. In this skill, "the gepa backend" always means the optimizer; statements
about "the optimizer" or "the backend" apply to whichever engine you chose.
`optimize_anything` does **black-box optimization**: you provide (1) a seed artifact, (2) an
**evaluator** that scores any artifact and returns feedback, and (3) a backend, which repeatedly
proposes improved artifacts and scores them through your evaluator. "Black-box" refers to the
**evaluator**, not the artifact: the backend never sees how the score is computed — no gradients,
no metric internals — only the scalar score and the feedback text you emit. The candidate itself
*is* visible: the proposer reads and rewrites it, applying the LLM's understanding of your artifact.
The framework just imposes no structure on it — any string an evaluator can score works. The
leverage is in your score and your feedback.
**You write the task and evaluator once, then choose the search algorithm with one `engine`
argument** — and the same code runs under any of them:
- **`gepa`** — the GEPA optimizer: reflective evolutionary search, in-process (an LLM reflects on
feedback and mutates candidates; keeps a Pareto frontier). The default; strongest when feedback
is rich.
- **`autoresearch`** — an agentic optimizer: one Claude Code subprocess iterates like a researcher in
a work dir, scoring candidates through an HTTP eval server.
- **`meta_harness`** — an agentic proposer (Claude subprocess) that reads the frontier/history each
iteration and writes new candidates for the engine to benchmark.
(There is also a `best_of_n` engine — sample N independent candidates, keep the best. It is
deliberately naive: use it as a **baseline** to compare an optimizer against, not as the optimizer.)
This makes it easy to start with one backend and benchmark others on the identical task/evaluator.
There are also **composition/pipeline helpers** that combine backends over the same task:
`optimize_sequential` (a pipeline — each stage's best seeds the next), `optimize_parallel`,
`optimize_best_of`, `optimize_vote` (re-score each branch's best for a fair pick), and an adaptive
scheduler that rotates backends on score plateaus — see `references/api.md`.
## What can be a candidate
A candidate is **any string your evaluator can score**. "Text in → a number out (higher is better),
plus optional feedback" is the entire contract, which covers a wide range of artifacts:
- **prompts** — system/user prompts, instruction templates, rubrics, few-shot exemplars
- **programs / code** — functions, whole files, CUDA kernels, scored by compiling + running + benchmarking
- **configs / specs / schemas / regex / SQL** — any text whose effect you can measure
- **agent scaffolds** — tool instructions, planner/critic prompts, orchestration text
- **pure search artifacts** — a mathematical construction, a packing layout, a plan, encoded as text
The score can be an **objective metric** (accuracy, pass rate, runtime, cost) **or an LLM-as-judge**
rating for subjective tasks (writing quality, helpfulness, style). At this API the candidate is a
single string (`seed_candidate: str | None`; `None` = seedless — the engine bootstraps from
`objective`/`background`). Multi-component dict candidates exist only in the lower-level
`gepa.gepa_launcher.optimize_anything` API, not here.
## Three optimization modes (choose by how you pass data)
The mode is determined by whether you provide `dataset` and `valset`:
1. **Single-task** (`dataset=None, valset=None`) — solve one hard problem; the candidate *is* the
solution; the evaluator is called with no example. *E.g. one CUDA kernel, a circle-packing layout.*
2. **Multi-task** (`dataset=<list>, valset=None`) — solve a batch of related problems with one shared
candidate, transferring insight across them; evaluator called per example. *E.g. a single prompt
that works across many tasks.*
3. **Generalization** (`dataset=<list>, valset=<list>`) — build a candidate that transfers to
**unseen** problems; optimize on `dataset`, select on `valset`. *E.g. a prompt tuned to generalize.*
*Note:* GEPA is the algorithm designed around this mode, and `valset`-based held-out selection is
currently implemented only by the gepa backend — the other backends fold `valset` into the
training pool (they can still generalize; there's just no separate selection split).
`test_set` is **separate from the modes and reporting-only**: the seed and the final candidate are
scored on it after optimization for an unbiased number — it never enters the search, selection, or
budget (for the agentic backends it is sealed at the eval server's HTTP layer, so the agent cannot
even see it). See `references/api.md` for details and when to use each mode.
## Install
```bash
pip install "gepa[full]" # [full] pulls cloudpickle — needed to pickle closure evaluators for
# parallel workers / opt-in evaluation caching; plain `pip install gepa`
# can fail there when your evaluator closes over data.
# Proposer LLM: the gepa backend's reflection LM defaults to "openai/gpt-5.1" (a LiteLLM id) — set
# that provider's key (OPENAI_API_KEY), or pass your own id (e.g. "anthropic/claude-sonnet-4-6" with
# ANTHROPIC_API_KEY, or a Bedrock ARN with AWS creds). You can also pass any callable implementing
# GEPA's LM protocol — a self-hosted / custom inference engine — instead of a model-id string.
# Agentic backends (autoresearch, meta_harness) additionally need the `claude` CLI on PATH (plus
# `jq` for the generated eval.sh):
npm install -g @anthropic-ai/claude-code # or: curl -fsSL https://claude.ai/install.sh | bash
# ...then run `claude` once to authenticate.
# On Linux they also need bubblewrap: the default sandbox=True jails the claude subprocess with
# bwrap (`sudo apt install bubblewrap` / `sudo dnf install bubblewrap`) and the run aborts at
# launch if it's missing. sandbox=False opts out — the agent then runs unsandboxed (loud warning).
```
## Mental model (4 pieces)
1. **Candidate** — the text you optimize (a `str`). The seed is the start; `None` = seedless.
2. **`evaluate(candidate, example) -> (score, info)`** — `score` is a float (higher is better);
`info` is a free-form dict of **feedback the backend's proposer reads** to make better candidates
(the gepa backend's reflection LM, or the agentic backends' agent; the `best_of_n` baseline
ignores feedback). When evals batch better than they stream (e.g. a provider batch API), pass
`batch_evaluator=` instead — all pending `(candidate, example)` pairs in one call; see
`references/api.md`.
3. **Engine (backend)** — `"gepa"` (default), `"autoresearch"`, `"meta_harness"`, the `"best_of_n"`
baseline, or a constructed `Engine` instance.
4. **Budget** — `max_evals` (server-side eval-call cap, **default 100**) and/or `max_token_cost`
(USD cap on the backend's own proposer-LLM spend). Setting both to `None` makes the run unbounded
(only a warning). **Size `max_evals` for many proposal rounds, not one** (see below) — this is the
most common way agents misuse this API.
## Sizing the valset and the budget (read this — the #1 mistake)
`max_evals` is the *main* control over how long the optimizer runs. Leave it at the default (100)
with a large valset, or set it too low, and the run stops after a **single proposal**, then reports
a "best candidate" that looks fine but is barely optimized.
- **valset** — on the gepa backend the best candidate is selected by its scores on the valset, so
make it a **representative subset of your data**. There is no fixed size: pick what represents the task (a
handful for small/expensive tasks, more when data is cheap and plentiful — anywhere from a few to
hundreds). Bigger valset = less noisy selection but more eval calls per candidate.
- **budget** — every proposed candidate is scored on the **whole selection set**, so size
`max_evals` off that set, not off a single proposal. Which set that is depends on the mode:
```
generalization (dataset + valset): max_evals ≳ 15–20 × len(valset) # scored & selected on valset
multi-task (dataset only): max_evals ≳ 15–20 × len(dataset) # scored & selected on dataset
single-task (no dataset/valset): max_evals ≳ 15–20 # 1 eval per candidate, so this
# IS the number of proposals
```
The constant is the same everywhere — **let the backend propose AND evaluate ~15–20 candidates**
(more if you can afford it). Anything much less and the run tries only a couple of candidates —
i.e. it's barely optimizing. (This arithmetic is exact for the gepa backend — and the best_of_n
baseline — which score every candidate on the full selection set; the agentic backends decide
themselves how to spend eval calls, so treat it as a floor.)
After the run, check how many proposals actually happened (on the gepa backend,
`result.candidates` is the full pool; with `engine.write_agent_state=True` the
`run_dir/iterations/` tree shows each one). **If it stopped after one proposal, the budget was too
low** — raise it and rerun.
### Give every run a real stop condition
`max_evals` bounds eval calls, but add explicit stops so runs end at the right moment:
- **`stop_at_score`** — set it whenever your metric has a known ceiling (e.g. `1.0` for a pass rate /
accuracy). The backend stops the moment a candidate reaches it instead of burning the rest of the
budget at the optimum.
- **`max_token_cost`** — a hard USD cap on the backend's own proposer/agent LLM spend. Especially
important for the agentic backends (`autoresearch`, `meta_harness`), whose Claude subprocesses
spend tokens between eval calls.
- **a wall-clock `timeout`** on the process you launch (e.g. `timeout 1200 python run.py`) as a
backstop.
- If you **opt in** to evaluation caching (`engine_config={"engine": {"cache_evaluation": True}}` on
the gepa backend — it is **off by default**), be aware `max_evals` then counts only cache *misses*:
a converged search can keep proposing cache-hitting candidates without consuming eval budget, so
`stop_at_score`/`max_token_cost` become mandatory, not optional. A distinct `valset` is cached
separately from the trainset; `valset=None` still reuses minibatch rollouts.
## Minimal working example
The example optimizes a system prompt for concreteness, but the **shape is identical** for any
candidate — swap `SEED` for a code file / config / etc. and have `evaluate` compile/run/measure it.
```python
from gepa.optimize_anything import optimize_anything, OptimizeAnythingConfig
SEED = "You are an expert. Solve the task. Output only the final answer."
def evaluate(candidate: str, example) -> tuple[float, dict]:
output = run_my_model(system_prompt=candidate, user_prompt=example["prompt"]) # your call
score = grade(output, example) # float, higher=better
return score, { # everything here is shown to the proposer LLM
"score": score,
"output": output,
"error": example.get("error"), # concrete, actionable feedback drives good proposals
}
result = optimize_anything(
seed_candidate=SEED,
evaluator=evaluate,
dataset=trainset, # optimize on these (multi-task/generalization mode)
valset=valset, # select the best candidate on these (generalization mode)
test_set=testset, # OPTIONAL, reporting-only: seed + final candidate scored here at the end
objective="Produce a prompt that maximizes task accuracy.",
background="Domain rules, constraints, output format the model must follow.",
config=OptimizeAnythingConfig(
engine="gepa", # swap to "autoresearch" / "meta_harness" — same code
# ("best_of_n" runs the same way, as a comparison baseline)
name="my_run",
max_evals=300, # ≳ 15-20 × len(valset): enough for ~15-20 proposals (see above)
stop_at_score=1.0, # stop at the optimum (set when your metric has a known ceiling)
max_concurrency=16,
run_dir="runs/my_run", # engine workspace (gepa run dir / agent work dir)
output_dir="outputs/my_run", # eval server: per-eval JSON, progress_log.jsonl, summary.json
engine_config={ # gepa backend: a GEPAConfig-shaped dict, validated strictly —
"reflection": { # an unknown key raises TypeError immediately (fail fast)
# a LiteLLM id (set the provider key) OR any callable implementing the LM protocol
"reflection_lm": "anthropic/claude-sonnet-4-6",
"reflection_minibatch_size": 5,
},
"engine": {"max_workers": 32, "seed": 0}, # seed = reproducibility
},
),
)
print(result.best_candidate, result.best_score)
# held-out (only present if you passed test_set): "test_score" = average, "test_scores" = per-example
print("held-out:", result.metadata.get("test_score"),
"seed held-out:", result.metadata.get("baseline_test_score"))
```
## Standard workflow
1. **Pick the mode** (single-task / multi-task / generalization) by which of `dataset`/`valset` you pass.
2. **Define the score deliberately.** The optimizer optimizes exactly what you measure — gate the
score on what you actually care about (see `references/gotchas.md`, reward hacking).
3. **Write a feedback-rich `evaluate`.** The `info` dict is the proposer's signal — return errors,
diffs, partial credit, not just a number (`oa.log()` and `capture_stdio` can route diagnostics in
automatically). See `references/writing_evaluators.md`.
4. **Pick a proposer LLM** — a LiteLLM id (set the provider key) or a custom LM-protocol callable.
Validate it with a 1-call test before a long run.
5. **Set a budget** (`max_evals` sized per above, and/or `max_token_cost`) plus `stop_at_score` when
the metric has a ceiling.
6. **Run `python scripts/preflight.py`** to fail fast on missing creds / CLI before a long run.
7. **Launch**, watch the first 1-2 evals (the eval→model→score chain), then let it run.
8. **Read `result.best_candidate` and `run_dir/`** (and `result.metadata["test_score"]` if you
passed a `test_set`).
## Critical gotchas (read before a real run)
These silently degrade *results* — skim before launching:
- **Reward hacking.** Every backend optimizes exactly what you score; a weak proxy gets gamed (e.g. a
"correct"-only score → a do-nothing wrapper). Gate the score on the real goal. → `references/gotchas.md`.
- **Selection bias.** In generalization mode the best candidate is the max over many scored on
`valset` — an optimistic estimate. Use enough `valset` examples (and N>1 for stochastic models), and
report on a `test_set` for an unbiased number. → `references/gotchas.md`.
- **Stochastic models default to N=1 per eval** → noisy selection. Average N samples *inside*
`evaluate`. → `references/writing_evaluators.md`.
- **Saturated signal → the gepa backend returns the seed unchanged.** If the seed already aces the
training examples, every proposal looks "not better" and is rejected (many proposals, ~0 accepted).
Reflection needs examples the seed gets *wrong* to learn from — ensure `dataset` has real
failures. → `references/gotchas.md`.
- **`engine_config` is validated strictly per backend** — an unknown key (including a leftover key
from a different backend after swapping `engine=`) raises `TypeError` at construction. Swapping
`engine=` means swapping the `engine_config` block. → `references/api.md`.
- **Agentic backends (`autoresearch`, `meta_harness`) shell out to the `claude` CLI** and abort at
launch with install instructions if it's missing — likewise for `bwrap` (bubblewrap) on Linux,
which the default `sandbox=True` needs. `sandbox=False` runs the agent unconfined (loud warning).
Run `scripts/preflight.py` first; details in `references/api.md`.
## Reference files (load as needed)
- `references/api.md` — `OptimizeAnythingConfig`, the backends and their typed `engine_config`
options, the three modes, the LM protocol, budget/cost semantics, `GEPAResult` shape, and the
composition/pipeline helpers.
- `references/writing_evaluators.md` — the `(score, info)` contract, `oa.log()`/`capture_stdio`,
LLM-as-judge scoring, multi-objective via `info["scores"]`, N>1 averaging, feedback design.
- `references/tracking.md` — enabling wandb / mlflow experiment tracking and what gets logged.
- `references/gotchas.md` — reward hacking, selection bias, the three modes, backend prerequisites.
- `scripts/preflight.py` — validate creds / proposer LM / `claude` CLI before launching.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |