yiouli/pixie-qa / skills/eval-driven-dev
eval-driven-dev
UnverifiedImprove AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user asks to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviors, improve quality, or do quality assurance for any Python project that calls an LLM model.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 17.4 KB |
| references/1-a-project-analysis.md | markdown | 5.7 KB |
| references/1-b-entry-point.md | markdown | 2.3 KB |
| references/1-c-eval-criteria.md | markdown | 7.5 KB |
| references/2a-instrumentation.md | markdown | 7.3 KB |
| references/2b-implement-runnable.md | markdown | 7.4 KB |
| references/2c-capture-and-verify-trace.md | markdown | 5.7 KB |
| references/3-define-evaluators.md | markdown | 10.3 KB |
| references/4-build-dataset.md | markdown | 21.6 KB |
| references/5-run-tests.md | markdown | 6.0 KB |
| references/6-analyze-outcomes.md | markdown | 16.2 KB |
| references/evaluators.md | markdown | 17.6 KB |
| references/runnable-examples/cli-app.md | markdown | 2.0 KB |
| references/runnable-examples/fastapi-web-server.md | markdown | 4.1 KB |
| references/runnable-examples/standalone-function.md | markdown | 1.8 KB |
| references/testing-api.md | markdown | 16.3 KB |
| references/wrap-api.md | markdown | 8.8 KB |
| resources/setup.shexecutable | script | 3.0 KB |
| resources/verify_step6_completion.py | script | 4.4 KB |
references/5-run-tests.md
View on GitHub# Step 5: Run `pixie test` and Fix Mechanical Issues
**Why this step**: Run `pixie test` and fix mechanical issues in your QA components — dataset format problems, runnable implementation bugs, and custom evaluator errors — until every entry produces real scores. This step is NOT about assessing result quality or fixing the application itself.
---
## 5a. Run tests
```bash
uv run pixie test
```
For verbose output with per-case scores and evaluator reasoning:
```bash
uv run pixie test -v
```
`pixie test` automatically loads the `.env` file before running tests.
The evaluation harness:
1. Resolves the `Runnable` class from the dataset's `runnable` field
2. Calls `Runnable.create()` to construct an instance, then `setup()` once
3. Runs all dataset entries **concurrently** (up to 4 in parallel):
a. Reads `input_data` and `eval_input` from the entry
b. Populates the wrap input registry with `eval_input` data
c. Initialises the capture registry
d. Validates `input_data` into the Pydantic model and calls `Runnable.run(args)`
e. `wrap(purpose="input")` calls in the app return registry values instead of calling external services
f. `wrap(purpose="output"/"state")` calls capture data for evaluation
g. Builds `Evaluable` from captured data
h. Runs evaluators
4. Calls `Runnable.teardown()` once
Because entries run concurrently, the Runnable's `run()` method must be concurrency-safe. If you see `sqlite3.OperationalError`, `"database is locked"`, or similar errors, add a `Semaphore(1)` to your Runnable (see the concurrency section in Step 2 reference).
## 5b. Fix mechanical issues only
This step is strictly about fixing what you built in previous steps — the dataset, the runnable, and any custom evaluators. You are fixing mechanical problems that prevent the pipeline from running, NOT assessing or improving the application's output quality.
**What counts as a mechanical issue** (fix these):
| Error | Cause | Fix |
| ------------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| `WrapRegistryMissError: name='<key>'` | Dataset entry missing an `eval_input` item with the `name` that the app's `wrap(purpose="input", name="<key>")` expects | Add the missing `{"name": "<key>", "value": ...}` to `eval_input` in every affected entry |
| `WrapTypeMismatchError` | Deserialized type doesn't match what the app expects | Fix the value in the dataset |
| Runnable resolution failure | `runnable` path or class name is wrong, or the class doesn't implement the `Runnable` protocol | Fix `filepath:ClassName` in the dataset; ensure the class has `create()` and `run()` methods |
| Import error | Module path or syntax error in runnable/evaluator | Fix the referenced file |
| `ModuleNotFoundError: pixie_qa` | `pixie_qa/` directory missing `__init__.py` | Run `pixie init` to recreate it |
| `TypeError: ... is not callable` | Evaluator name points to a non-callable attribute | Evaluators must be functions, classes, or callable instances |
| `sqlite3.OperationalError` | Concurrent `run()` calls sharing a SQLite connection | Add `asyncio.Semaphore(1)` to the Runnable (see Step 2 concurrency section) |
| Custom evaluator crashes | Bug in your custom evaluator implementation | Fix the evaluator code |
**What is NOT a mechanical issue** (do NOT fix these here):
- Application produces wrong/low-quality output → that's the application's behavior, analyzed in Step 6
- Evaluator scores are low → that's a quality signal, analyzed in Step 6
- LLM calls fail inside the application → report in Step 6, do not mock or work around
- Evaluator scores fluctuate between runs → normal LLM non-determinism, not a bug
Iterate — fix errors, re-run, fix the next error — until `pixie test` runs to completion with real evaluator scores for all entries.
## Output
After `pixie test` completes successfully, results are stored in the per-entry directory structure:
```
{PIXIE_ROOT}/results/<test_id>/
meta.json # test run metadata
dataset-{idx}/
metadata.json # dataset name, path, runnable
entry-{idx}/
config.json # evaluators, description, expectation
eval-input.jsonl # input data fed to evaluators
eval-output.jsonl # output data captured from app
evaluations.jsonl # evaluation results (scored + pending)
trace.jsonl # LLM call traces (if captured)
```
The `<test_id>` is printed in console output. You will reference this directory in Step 6.
---
> **If you hit an unexpected error** when running tests (wrong parameter names, import failures, API mismatch), read `wrap-api.md`, `evaluators.md`, or `testing-api.md` for the authoritative API reference before guessing at a fix.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |