yiouli/pixie-qa / skills/eval-driven-dev
eval-driven-dev
UnverifiedImprove AI application with evaluation-driven development. Define eval criteria, instrument the application, build golden datasets, observe and evaluate application runs, analyze results, and produce a concrete action plan for improvements. ALWAYS USE THIS SKILL when the user asks to set up QA, add tests, add evals, evaluate, benchmark, fix wrong behaviors, improve quality, or do quality assurance for any Python project that calls an LLM model.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 17.4 KB |
| references/1-a-project-analysis.md | markdown | 5.7 KB |
| references/1-b-entry-point.md | markdown | 2.3 KB |
| references/1-c-eval-criteria.md | markdown | 7.5 KB |
| references/2a-instrumentation.md | markdown | 7.3 KB |
| references/2b-implement-runnable.md | markdown | 7.4 KB |
| references/2c-capture-and-verify-trace.md | markdown | 5.7 KB |
| references/3-define-evaluators.md | markdown | 10.3 KB |
| references/4-build-dataset.md | markdown | 21.6 KB |
| references/5-run-tests.md | markdown | 6.0 KB |
| references/6-analyze-outcomes.md | markdown | 16.2 KB |
| references/evaluators.md | markdown | 17.6 KB |
| references/runnable-examples/cli-app.md | markdown | 2.0 KB |
| references/runnable-examples/fastapi-web-server.md | markdown | 4.1 KB |
| references/runnable-examples/standalone-function.md | markdown | 1.8 KB |
| references/testing-api.md | markdown | 16.3 KB |
| references/wrap-api.md | markdown | 8.8 KB |
| resources/setup.shexecutable | script | 3.0 KB |
| resources/verify_step6_completion.py | script | 4.4 KB |
references/runnable-examples/fastapi-web-server.md
View on GitHub# Runnable Example: FastAPI / Web Server
**When the app is a web server** (FastAPI, Flask, Starlette) and you need to exercise the full HTTP request pipeline.
**Approach**: Use `httpx.AsyncClient` with `ASGITransport` to run the ASGI app in-process. This is the fastest and most reliable approach — no subprocess, no port management.
```python
# pixie_qa/run_app.py
import httpx
from pydantic import BaseModel
import pixie
class AppArgs(BaseModel):
user_message: str
class AppRunnable(pixie.Runnable[AppArgs]):
"""Drives a FastAPI app via in-process ASGI transport."""
_client: httpx.AsyncClient
@classmethod
def create(cls) -> "AppRunnable":
return cls()
async def setup(self) -> None:
from myapp.main import app # your FastAPI/Starlette app instance
transport = httpx.ASGITransport(app=app)
self._client = httpx.AsyncClient(transport=transport, base_url="http://test")
async def run(self, args: AppArgs) -> None:
await self._client.post("/chat", json={"message": args.user_message})
async def teardown(self) -> None:
await self._client.aclose()
```
## ASGITransport skips lifespan events
`httpx.ASGITransport` does **not** trigger ASGI lifespan events (`startup` / `shutdown`). If the app initializes resources in its lifespan (database connections, caches, service clients), you must replicate that initialization manually in `setup()`:
```python
async def setup(self) -> None:
# Manually replicate what the app's lifespan does
from myapp.db import get_connection, init_db, seed_data
import myapp.main as app_module
conn = get_connection()
init_db(conn)
seed_data(conn)
app_module.db_conn = conn # set the module-level global the app expects
transport = httpx.ASGITransport(app=app_module.app)
self._client = httpx.AsyncClient(transport=transport, base_url="http://test")
async def teardown(self) -> None:
await self._client.aclose()
# Clean up the manually-initialized resources
import myapp.main as app_module
if hasattr(app_module, "db_conn") and app_module.db_conn:
app_module.db_conn.close()
```
## Concurrency with shared mutable state
If the app uses shared mutable state (in-memory SQLite, file-based DB, global caches), add a semaphore to serialise access:
```python
import asyncio
class AppRunnable(pixie.Runnable[AppArgs]):
_client: httpx.AsyncClient
_sem: asyncio.Semaphore
@classmethod
def create(cls) -> "AppRunnable":
inst = cls()
inst._sem = asyncio.Semaphore(1)
return inst
async def setup(self) -> None:
from myapp.main import app
transport = httpx.ASGITransport(app=app)
self._client = httpx.AsyncClient(transport=transport, base_url="http://test")
async def run(self, args: AppArgs) -> None:
async with self._sem:
await self._client.post("/chat", json={"message": args.user_message})
async def teardown(self) -> None:
await self._client.aclose()
```
Only use the semaphore when needed — if the app uses per-session state keyed by unique IDs (call_sid, session_id), concurrent calls are naturally isolated and no lock is needed.
## Alternative: External server with httpx
When the app can't be imported directly (complex startup, `uvicorn.run()` in `__main__`), start it as a subprocess and hit it with HTTP:
```python
class AppRunnable(pixie.Runnable[AppArgs]):
_client: httpx.AsyncClient
@classmethod
def create(cls) -> "AppRunnable":
return cls()
async def setup(self) -> None:
# Assumes the server is already running (started via run-with-timeout.sh)
self._client = httpx.AsyncClient(base_url="http://localhost:8000")
async def run(self, args: AppArgs) -> None:
await self._client.post("/chat", json={"message": args.user_message})
async def teardown(self) -> None:
await self._client.aclose()
```
Start the server before running `pixie trace` or `pixie test`:
```bash
bash resources/run-with-timeout.sh 120 uv run python -m myapp.server
sleep 3 # wait for readiness
```
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 196k | Updated | Claude CodeCodex | Verified | Install |