Skip to content

github/awesome-copilot / skills/phoenix-evals

phoenix-evals

Verified

Build and run evaluators for AI/LLM applications using Phoenix.

FileKindSize
SKILL.mdskill md4.5 KB
references/axial-coding.mdmarkdown2.3 KB
references/common-mistakes-python.mdmarkdown7.0 KB
references/error-analysis-multi-turn.mdmarkdown1.3 KB
references/error-analysis.mdmarkdown4.2 KB
references/evaluate-dataframe-python.mdmarkdown4.4 KB
references/evaluators-code-python.mdmarkdown3.0 KB
references/evaluators-code-typescript.mdmarkdown1.2 KB
references/evaluators-custom-templates.mdmarkdown1.2 KB
references/evaluators-llm-python.mdmarkdown2.6 KB
references/evaluators-llm-typescript.mdmarkdown1.4 KB
references/evaluators-overview.mdmarkdown1.1 KB
references/evaluators-pre-built.mdmarkdown2.4 KB
references/evaluators-rag.mdmarkdown3.1 KB
references/experiments-datasets-python.mdmarkdown4.8 KB
references/experiments-datasets-typescript.mdmarkdown2.7 KB
references/experiments-overview.mdmarkdown1.7 KB
references/experiments-running-python.mdmarkdown2.9 KB
references/experiments-running-typescript.mdmarkdown3.1 KB
references/experiments-synthetic-python.mdmarkdown1.8 KB
references/experiments-synthetic-typescript.mdmarkdown2.2 KB
references/fundamentals-anti-patterns.mdmarkdown1.6 KB
references/fundamentals-model-selection.mdmarkdown1.4 KB
references/fundamentals.mdmarkdown1.8 KB
references/observe-sampling-python.mdmarkdown2.4 KB
references/observe-sampling-typescript.mdmarkdown3.9 KB
references/observe-tracing-setup.mdmarkdown3.3 KB
references/production-continuous.mdmarkdown3.6 KB
references/production-guardrails.mdmarkdown1.3 KB
references/production-overview.mdmarkdown2.3 KB
references/setup-python.mdmarkdown1.9 KB
references/setup-typescript.mdmarkdown981 B
references/validation-evaluators-python.mdmarkdown1.0 KB
references/validation-evaluators-typescript.mdmarkdown4.4 KB
references/validation.mdmarkdown1.7 KB

references/production-overview.md

View on GitHub
# Production: Overview

CI/CD evals vs production monitoring - complementary approaches.

## Two Evaluation Modes

| Aspect | CI/CD Evals | Production Monitoring |
| ------ | ----------- | -------------------- |
| **When** | Pre-deployment | Post-deployment, ongoing |
| **Data** | Fixed dataset | Sampled traffic |
| **Goal** | Prevent regression | Detect drift |
| **Response** | Block deploy | Alert & analyze |

## CI/CD Evaluations

```python
from phoenix.client import Client

client = Client()

# Fast, deterministic checks
ci_evaluators = [
    has_required_format,
    no_pii_leak,
    safety_check,
    regression_test_suite,
]

# Small but representative dataset (~100 examples)
client.experiments.run_experiment(dataset=ci_dataset, task=task, evaluators=ci_evaluators)
```

Set thresholds: regression=0.95, safety=1.0, format=0.98.

## Production Monitoring

### Python

```python
from phoenix.client import Client
from datetime import datetime, timedelta

client = Client()

# Sample recent traces (last hour)
traces = client.traces.get_traces(
    project_identifier="my-app",
    start_time=datetime.now() - timedelta(hours=1),
    include_spans=True,
    limit=100,
)

# Run evaluators on sampled traffic
for trace in traces:
    results = run_evaluators_async(trace, production_evaluators)
    if any(r["score"] < 0.5 for r in results):
        alert_on_failure(trace, results)
```

### TypeScript

```typescript
import { getTraces } from "@arizeai/phoenix-client/traces";
import { getSpans } from "@arizeai/phoenix-client/spans";

// Sample recent traces (last hour)
const { traces } = await getTraces({
  project: { projectName: "my-app" },
  startTime: new Date(Date.now() - 60 * 60 * 1000),
  includeSpans: true,
  limit: 100,
});

// Or sample spans directly for evaluation
const { spans } = await getSpans({
  project: { projectName: "my-app" },
  startTime: new Date(Date.now() - 60 * 60 * 1000),
  limit: 100,
});

// Run evaluators on sampled traffic
for (const span of spans) {
  const results = await runEvaluators(span, productionEvaluators);
  if (results.some((r) => r.score < 0.5)) {
    await alertOnFailure(span, results);
  }
}
```

Prioritize: errors → negative feedback → random sample.

## Feedback Loop

```
Production finds failure → Error analysis → Add to CI dataset → Prevents future regression
```
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall