Skip to content

agentsope/SkillAlchemy / skills/agentsop-dspy

agentsop-dspy

Verified

Operating SOP for DSPy (Stanford NLP) — the declarative framework for "programming, not prompting" language models. Activate when the user says any of: "use DSPy", "compile a prompt", "optimize prompts/programs", "MIPRO/MIPROv2", "BootstrapFewShot", "GEPA", "Signatures + Modules", "teleprompter", "auto-tune prompts for a different LM", or whenever a brittle hand-crafted prompt pipeline needs to be turned into a *compiled*, measurable, swappable program. Do NOT activate for one-shot prompt tweaks, no-metric exploratory work, or pipelines where prompts must remain human-authored verbatim — use raw prompting or LangChain templates instead.

references/R2-sop-workflow.md

View on GitHub
# R2 — SOP Workflow

DSPy docs are explicit about a **three-stage gate** [dspy.ai/learn/]:

> "It's unproductive to launch optimization runs using a poorly designed program or a bad metric."

The stages are not optional. Each has an exit criterion.

---

## Stage 1 — Programming

**Goal**: a *working but un-optimized* program that produces plausible outputs.

### Steps

1. **Write the signature.**
   - Start inline: `"question -> answer"`.
   - Promote to class-based when types matter or fields need disambiguation.
   - Field names carry semantic load — they are the only hint the optimizer sees before data.
2. **Pick the module.**
   - Default: `dspy.ChainOfThought(Sig)`.
   - `Predict` for trivial; `ReAct` for tools; `ProgramOfThought` for math.
3. **Compose with Python**, subclassing `dspy.Module`.
4. **Configure the LM**: `dspy.configure(lm=dspy.LM('openai/gpt-4o-mini'))`.
5. **Hand-eyeball 5–10 examples** via `dspy.inspect_history(n=3)`.

### Exit criterion

Un-optimized program produces *plausible* (not great) outputs on 5+ examples. If outputs are nonsensical, the **program structure** is wrong — fix decomposition, not optimizer.

---

## Stage 2 — Evaluation

**Goal**: a metric the team trusts, plus a measured baseline.

### Steps

6. **Build a dev set.** Documented thresholds [dspy.ai/learn/optimization/overview/]:
   - 30 examples = minimum useful.
   - 300 examples = recommended.
   - 200+ = required floor for MIPROv2 to avoid overfit.
7. **Write a metric**:

   ```python
   def metric(example, pred, trace=None):
       return example.answer.lower() == pred.answer.lower()
   ```

   For multi-criteria: decompose into sub-judges, return `bool` when `trace is not None` (compile-time), `float` otherwise (eval-time).
8. **Spot-check the metric** against a human on 10–20 examples. >20% disagreement = fix the metric first.
9. **Measure baseline** with `dspy.Evaluate(devset=dev, metric=metric, num_threads=16)`.

### Exit criterion

Baseline is stable across two cache-free runs AND metric agrees with human on ≥80% of spot-checks.

---

## Stage 3 — Optimization (compile)

**Goal**: a compiled program that beats baseline on held-out test data.

### Steps

10. **Pick optimizer** (see R3 Case A + SKILL §4.1 table).
11. **Use unusual 20/80 split** for prompt-based optimizers [dspy.ai/learn/optimization/overview/]. GEPA uses standard ML splits.
12. **Pick optimizer LM ≠ task LM if budget tight**. Community evidence: gpt-4o-mini optimizing for gpt-4o gives comparable results at fraction of cost [github.com/stanfordnlp/dspy/issues/1596].
13. **Start `auto="light"`.** Never `heavy` first.
14. **Save with metadata**:

    ```python
    compiled.save("./v1/", save_program=True)   # whole program with deps
    # OR
    compiled.save("v1.json")                     # state only
    ```
15. **Evaluate on a held-out test set** (distinct from val).
16. **Deploy** [dspy.ai/tutorials/deployment/]:
    - FastAPI for lightweight: `dspy.asyncify(program)` then uvicorn.
    - MLflow for production: `mlflow.dspy.log_model(program, task="llm/v1/chat")`.
17. **Track usage in production**: `dspy.configure(track_usage=True)` and log `program.get_lm_usage()`.

### Exit criterion

Compiled program beats baseline on the held-out test set by task-relevant delta. Held-out set must NOT be the val set used by the optimizer.

---

## Decision points (one screen)

```
[Stage 1] Plausible un-optimized output?
              No  → fix signature / module / decomposition
              Yes ↓
[Stage 2] Metric agrees with human ≥ 80%?
              No  → decompose metric, sub-judges, length penalty
              Yes ↓
          Have ≥ 30 examples?
              No  → collect more, or use LabeledFewShot as floor
              Yes ↓
[Stage 3] Pick optimizer (see R3 Case A):
              ≤10 examples  → BootstrapFewShot
              30–50         → BootstrapFewShotWithRandomSearch
              200+          → MIPROv2(auto="light")
              Textual fb    → GEPA
          ↓
          auto="light" run. Gains > 5%?
              No  → return to Stage 1/2, the optimizer is not the bottleneck
              Yes → escalate to "medium"; if 300+ data, "heavy"
          ↓
          Evaluate on held-out test. Better than baseline?
              No  → return to Stage 1, fix program graph
              Yes → save, deploy
```

---

## What to do when optimization plateaus

The docs explicitly list the iteration questions [dspy.ai/learn/optimization/overview/]:

1. Is the task well-defined? (Signature too vague?)
2. Do you need more data?
3. Should your evaluation metric change?
4. Would a more sophisticated optimizer help? (← typically the **last** thing to try)
5. Does your program structure need revision? (Module choices, decomposition.)

The instinct to "try a different optimizer" is usually wrong. Go back to Stage 1.

---

## Cost-aware sub-protocol

Typical costs [dspy.ai/faqs/]:
- BootstrapFewShotWithRandomSearch on a small task: ~$3 USD, 6 minutes, 3200 API calls.
- MIPROv2 `auto="light"`: order of $2.
- MIPROv2 `auto="heavy"` on 1000+ examples: tens of dollars, hours.

Cost-control checklist before any compile:
- [ ] Use cheap LM (e.g. gpt-4o-mini) as `prompt_model` even if `task_model` is expensive.
- [ ] `auto="light"` first.
- [ ] Cap `max_bootstrapped_demos` and `max_labeled_demos` (defaults: 4 each).
- [ ] Set `num_threads` to a value your rate limits tolerate.
- [ ] Disable cache only if you actually need fresh runs: `dspy.LM(..., cache=False)`.
- [ ] In Lambda/serverless, set both `DSP_CACHEDIR` and `DSPY_CACHEDIR` to `/tmp` or disable.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall