gepa-ai/gepa / .claude/skills/gepa-optimize-anything
gepa-optimize-anything
FlaggedAutomatically improve any text artifact that can be scored — prompts, programs/code, configs, specs, regex/SQL/schemas, agent scaffolds, or encoded search solutions — with optimize_anything, an LLM-driven optimization API: a search backend repeatedly proposes better versions from execution feedback and an evaluator you provide assigns the score (an objective metric or an LLM-as-judge for subjective tasks). optimize_anything is one interface across several optimizer backends — GEPA (reflective evolutionary search, the default), AutoResearch, MetaHarness — plus a best-of-N baseline: write the evaluator once and switch backend with a single argument. Use whenever a task needs auto-optimizing, tuning, or searching over text with any quality signal (accuracy, pass rate, latency, cost, judge rating); when building an evaluator + proposer loop; when running GEPA; or when comparing optimizers. The candidate can be any string an evaluator can grade, scored on one problem or generalized across many.
| File | Kind | Size |
|---|---|---|
| .claude-plugin/plugin.json | config | 686 B |
| SKILL.md | skill md | 18.0 KB |
| references/api.md | markdown | 17.8 KB |
| references/gotchas.md | markdown | 8.1 KB |
| references/tracking.md | markdown | 2.8 KB |
| references/writing_evaluators.md | markdown | 7.4 KB |
| scripts/preflight.py | script | 4.8 KB |
references/tracking.md
View on GitHub# Experiment tracking
The gepa backend logs to **Weights & Biases** and **MLflow** via `TrackingConfig` — configure it
under `engine_config["tracking"]` (a valid `GEPAConfig` field, coerced to `TrackingConfig`). The
other backends have no built-in tracker; for them, rely on the eval server's `output_dir` records
(always written, whatever the backend):
```python
result = optimize_anything(
seed_candidate=SEED, evaluator=evaluate, dataset=trainset, valset=valset,
config=OptimizeAnythingConfig(
engine="gepa", max_evals=300,
engine_config={
"reflection": {"reflection_lm": "anthropic/claude-sonnet-4-6"},
"tracking": { # -> TrackingConfig
"use_wandb": True,
"wandb_init_kwargs": {"project": "my-gepa-run", "name": "experiment-1"},
},
},
),
)
```
## `TrackingConfig` fields
- `use_wandb`, `wandb_api_key`, `wandb_init_kwargs`, `wandb_attach_existing`, `wandb_step_metric`
- `use_mlflow`, `mlflow_tracking_uri`, `mlflow_experiment_name`, `mlflow_attach_existing`
- `key_prefix` — prepended to every logged key/name (e.g. `"gepa/"` → `gepa/val_score`)
- `logger` — a custom `LoggerProtocol` for plain-text log lines
## What gets logged
- **scalars** — `val_program_average`, `best_score_on_valset`, `total_metric_calls`, …
- **tables** — `candidates`, `proposals`, `valset_scores`, `valset_pareto_front`
- **run summary** — seed and best-found component text
- **interactive candidate-tree HTML**
## Attaching to an existing run
If you've already called `wandb.init()` (or started an MLflow run) yourself — e.g. a parent script
that also logs other things — set `wandb_attach_existing=True` (or `mlflow_attach_existing=True`).
The tracker then logs **into your active run** without calling `init()`/`finish()`, so it won't
disrupt the run lifecycle. Combine with `key_prefix="gepa/"` to keep its metrics in their own
namespace.
When the host loop manages its own wandb step counter, also set `wandb_step_metric` (e.g.
`"gepa/iteration"`): the tracker then declares its own x-axis via `wandb.define_metric` instead of
passing `step=`, so its `1, 2, 3, …` steps don't collide with the host's counter (a collision makes
wandb silently drop the data).
## Custom hooks
For programmatic observation beyond metric logging, pass `engine_config={"callbacks": [...]}` —
`GEPACallback` objects receive events like `on_optimization_start`, `on_iteration_end`,
`on_candidate_accepted`, `on_proposal_end`. The eval server also always writes a flat record
(per-eval JSON + `progress_log.jsonl` + `summary.json`) under `output_dir`, whatever tracking is
configured.
See the official guide for the authoritative list of fields and behaviors:
<https://gepa-ai.github.io/gepa/guides/experiment-tracking/>.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |