Skip to content

gepa-ai/gepa / .claude/skills/gepa-optimize-anything

gepa-optimize-anything

Flagged

Automatically improve any text artifact that can be scored — prompts, programs/code, configs, specs, regex/SQL/schemas, agent scaffolds, or encoded search solutions — with optimize_anything, an LLM-driven optimization API: a search backend repeatedly proposes better versions from execution feedback and an evaluator you provide assigns the score (an objective metric or an LLM-as-judge for subjective tasks). optimize_anything is one interface across several optimizer backends — GEPA (reflective evolutionary search, the default), AutoResearch, MetaHarness — plus a best-of-N baseline: write the evaluator once and switch backend with a single argument. Use whenever a task needs auto-optimizing, tuning, or searching over text with any quality signal (accuracy, pass rate, latency, cost, judge rating); when building an evaluator + proposer loop; when running GEPA; or when comparing optimizers. The candidate can be any string an evaluator can grade, scored on one problem or generalized across many.

FileKindSize
.claude-plugin/plugin.jsonconfig686 B
SKILL.mdskill md18.0 KB
references/api.mdmarkdown17.8 KB
references/gotchas.mdmarkdown8.1 KB
references/tracking.mdmarkdown2.8 KB
references/writing_evaluators.mdmarkdown7.4 KB
scripts/preflight.pyscript4.8 KB

references/tracking.md

View on GitHub
# Experiment tracking

The gepa backend logs to **Weights & Biases** and **MLflow** via `TrackingConfig` — configure it
under `engine_config["tracking"]` (a valid `GEPAConfig` field, coerced to `TrackingConfig`). The
other backends have no built-in tracker; for them, rely on the eval server's `output_dir` records
(always written, whatever the backend):

```python
result = optimize_anything(
    seed_candidate=SEED, evaluator=evaluate, dataset=trainset, valset=valset,
    config=OptimizeAnythingConfig(
        engine="gepa", max_evals=300,
        engine_config={
            "reflection": {"reflection_lm": "anthropic/claude-sonnet-4-6"},
            "tracking": {                       # -> TrackingConfig
                "use_wandb": True,
                "wandb_init_kwargs": {"project": "my-gepa-run", "name": "experiment-1"},
            },
        },
    ),
)
```

## `TrackingConfig` fields
- `use_wandb`, `wandb_api_key`, `wandb_init_kwargs`, `wandb_attach_existing`, `wandb_step_metric`
- `use_mlflow`, `mlflow_tracking_uri`, `mlflow_experiment_name`, `mlflow_attach_existing`
- `key_prefix` — prepended to every logged key/name (e.g. `"gepa/"` → `gepa/val_score`)
- `logger` — a custom `LoggerProtocol` for plain-text log lines

## What gets logged
- **scalars** — `val_program_average`, `best_score_on_valset`, `total_metric_calls`, …
- **tables** — `candidates`, `proposals`, `valset_scores`, `valset_pareto_front`
- **run summary** — seed and best-found component text
- **interactive candidate-tree HTML**

## Attaching to an existing run
If you've already called `wandb.init()` (or started an MLflow run) yourself — e.g. a parent script
that also logs other things — set `wandb_attach_existing=True` (or `mlflow_attach_existing=True`).
The tracker then logs **into your active run** without calling `init()`/`finish()`, so it won't
disrupt the run lifecycle. Combine with `key_prefix="gepa/"` to keep its metrics in their own
namespace.

When the host loop manages its own wandb step counter, also set `wandb_step_metric` (e.g.
`"gepa/iteration"`): the tracker then declares its own x-axis via `wandb.define_metric` instead of
passing `step=`, so its `1, 2, 3, …` steps don't collide with the host's counter (a collision makes
wandb silently drop the data).

## Custom hooks
For programmatic observation beyond metric logging, pass `engine_config={"callbacks": [...]}` —
`GEPACallback` objects receive events like `on_optimization_start`, `on_iteration_end`,
`on_candidate_accepted`, `on_proposal_end`. The eval server also always writes a flat record
(per-eval JSON + `progress_log.jsonl` + `summary.json`) under `output_dir`, whatever tracking is
configured.

See the official guide for the authoritative list of fields and behaviors:
<https://gepa-ai.github.io/gepa/guides/experiment-tracking/>.
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall