Skip to content

ObneyAI/orc / .claude/skills/orc-optimize

orc-optimize

Verified

Set up GEPA prompt optimization for an ORC workflow

FileKindSize
SKILL.mdskill md3.0 KB
---
name: orc-optimize
description: Set up GEPA prompt optimization for an ORC workflow
---

# GEPA Prompt Optimization

Optimize LLM instructions in an ORC workflow using GEPA (Genetic-Pareto Prompt Optimizer). Read `docs/GEPA-GUIDE.md` for the full reference.

## Require

```clojure
(require '[ai.obney.orc.gepa.interface :as gepa])
```

## Overview

GEPA improves LLM node instructions through:
1. **Reflective mutation** — LLM analyzes failures and proposes better instructions
2. **Pareto selection** — Maintains diversity by tracking per-example best candidates
3. **Iterative generations** — Each generation proposes and evaluates instruction variants

## Setup

### 1. Define Metrics

Metrics score how well a candidate instruction performs on each training example.

```clojure
;; Exact match (binary: 0 or 1)
(gepa/make-exact-match-metric "answer")

;; Contains check
(gepa/make-contains-metric "answer")

;; Judge-based (LLM evaluates quality)
(gepa/make-judge-metric
  {:grounding 0.35
   :completeness 0.25
   :instruction-following 0.25
   :reasoning 0.15})
```

### 2. Define Training Examples

```clojure
(def examples
  [{"question" "What is 2+2?" "expected-answer" "4"}
   {"question" "Capital of France?" "expected-answer" "Paris"}
   {"question" "Largest ocean?" "expected-answer" "Pacific"}])
```

### 3. Start Optimization

```clojure
(gepa/optimize! ctx
  {:sheet-id sheet-id
   :node-name "answer-node"          ;; which LLM node to optimize
   :trainset examples
   :valset examples
   :metric-fn (gepa/make-exact-match-metric "answer")
   :config {:max-metric-calls 30}
   :block? false})
```

### 4. Check Results

```clojure
;; Get the best candidate
(gepa/get-best-candidate ctx optimization-id)

;; Get the Pareto frontier
(gepa/get-pareto-frontier ctx optimization-id)

;; Get optimization progress
(gepa/get-progress ctx optimization-id)
```

If the process stops during an optimization, reconstruct the same Grain context
and advance one durable missing transition at a time:

```clojure
(gepa/resume! ctx optimization-id)
;; => {:status :resumed :boundary ...}
;; or {:status :already-terminal ...}
```

### 5. Apply and Publish the Winner

```clojure
;; The exact immutable source version is mandatory.
(gepa/apply-winner! ctx optimization-id 3)
;; => {:source-version 3 :target-version 4
;;     :source-fingerprint ... :target-fingerprint ...}
```

Optimization never silently mutates a published workflow. Applying a completed
winner updates the draft and publishes a new immutable version with source and
target fingerprints.

## Key Concepts

- **Candidate** — An instruction variant with scores per training example
- **Pareto frontier** — Set of non-dominated candidates (no single candidate beats another on all examples)
- **Generation** — One round of propose → evaluate → select
- **Budget** — Controls how many generations and candidates per generation

## Reference
- `docs/GEPA-GUIDE.md` — Full GEPA integration guide
- `docs/SELF-IMPROVING-LOOP.md` — How evaluation feeds continuous improvement
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 322kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 322kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 322kUpdated Claude CodeCodexVerifiedInstall