yiouli/pixie-qa / skills/eval-driven-dev
eval-driven-dev
UnverifiedImprove AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user asks to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviors, improve quality, or do quality assurance for any Python project that calls an LLM model.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 17.4 KB |
| references/1-a-project-analysis.md | markdown | 5.7 KB |
| references/1-b-entry-point.md | markdown | 2.3 KB |
| references/1-c-eval-criteria.md | markdown | 7.5 KB |
| references/2a-instrumentation.md | markdown | 7.3 KB |
| references/2b-implement-runnable.md | markdown | 7.4 KB |
| references/2c-capture-and-verify-trace.md | markdown | 5.7 KB |
| references/3-define-evaluators.md | markdown | 10.3 KB |
| references/4-build-dataset.md | markdown | 21.6 KB |
| references/5-run-tests.md | markdown | 6.0 KB |
| references/6-analyze-outcomes.md | markdown | 16.2 KB |
| references/evaluators.md | markdown | 17.6 KB |
| references/runnable-examples/cli-app.md | markdown | 2.0 KB |
| references/runnable-examples/fastapi-web-server.md | markdown | 4.1 KB |
| references/runnable-examples/standalone-function.md | markdown | 1.8 KB |
| references/testing-api.md | markdown | 16.3 KB |
| references/wrap-api.md | markdown | 8.8 KB |
| resources/setup.shexecutable | script | 3.0 KB |
| resources/verify_step6_completion.py | script | 4.4 KB |
references/runnable-examples/standalone-function.md
View on GitHub# Runnable Example: Standalone Function (No Server)
**When the app is a plain Python function or module** — no web framework, no server, no infrastructure.
**Approach**: Import and call the function directly from `run()`. This is the simplest case.
```python
# pixie_qa/run_app.py
from pydantic import BaseModel
import pixie
class AppArgs(BaseModel):
question: str
class AppRunnable(pixie.Runnable[AppArgs]):
"""Drives a standalone function for tracing and evaluation."""
@classmethod
def create(cls) -> "AppRunnable":
return cls()
async def run(self, args: AppArgs) -> None:
from myapp.agent import answer_question
await answer_question(args.question)
```
If the function is synchronous, wrap it with `asyncio.to_thread`:
```python
import asyncio
async def run(self, args: AppArgs) -> None:
from myapp.agent import answer_question
await asyncio.to_thread(answer_question, args.question)
```
If the function depends on an external service (e.g., a vector store), the `wrap(purpose="input")` calls you added in Step 2a handle it automatically — the registry injects test data in eval mode.
### When to use `setup()` / `teardown()`
Most standalone functions don't need lifecycle methods. Use them only when the function requires a shared resource (e.g., a pre-loaded embedding model, a database connection):
```python
class AppRunnable(pixie.Runnable[AppArgs]):
_model: SomeModel
@classmethod
def create(cls) -> "AppRunnable":
return cls()
async def setup(self) -> None:
from myapp.models import load_model
self._model = load_model()
async def run(self, args: AppArgs) -> None:
from myapp.agent import answer_question
await answer_question(args.question, model=self._model)
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |