Skip to content

github/awesome-copilot / skills/phoenix-evals

phoenix-evals

Verified

Build and run evaluators for AI/LLM applications using Phoenix.

FileKindSize
SKILL.mdskill md4.5 KB
references/axial-coding.mdmarkdown2.3 KB
references/common-mistakes-python.mdmarkdown7.0 KB
references/error-analysis-multi-turn.mdmarkdown1.3 KB
references/error-analysis.mdmarkdown4.2 KB
references/evaluate-dataframe-python.mdmarkdown4.4 KB
references/evaluators-code-python.mdmarkdown3.0 KB
references/evaluators-code-typescript.mdmarkdown1.2 KB
references/evaluators-custom-templates.mdmarkdown1.2 KB
references/evaluators-llm-python.mdmarkdown2.6 KB
references/evaluators-llm-typescript.mdmarkdown1.4 KB
references/evaluators-overview.mdmarkdown1.1 KB
references/evaluators-pre-built.mdmarkdown2.4 KB
references/evaluators-rag.mdmarkdown3.1 KB
references/experiments-datasets-python.mdmarkdown4.8 KB
references/experiments-datasets-typescript.mdmarkdown2.7 KB
references/experiments-overview.mdmarkdown1.7 KB
references/experiments-running-python.mdmarkdown2.9 KB
references/experiments-running-typescript.mdmarkdown3.1 KB
references/experiments-synthetic-python.mdmarkdown1.8 KB
references/experiments-synthetic-typescript.mdmarkdown2.2 KB
references/fundamentals-anti-patterns.mdmarkdown1.6 KB
references/fundamentals-model-selection.mdmarkdown1.4 KB
references/fundamentals.mdmarkdown1.8 KB
references/observe-sampling-python.mdmarkdown2.4 KB
references/observe-sampling-typescript.mdmarkdown3.9 KB
references/observe-tracing-setup.mdmarkdown3.3 KB
references/production-continuous.mdmarkdown3.6 KB
references/production-guardrails.mdmarkdown1.3 KB
references/production-overview.mdmarkdown2.3 KB
references/setup-python.mdmarkdown1.9 KB
references/setup-typescript.mdmarkdown981 B
references/validation-evaluators-python.mdmarkdown1.0 KB
references/validation-evaluators-typescript.mdmarkdown4.4 KB
references/validation.mdmarkdown1.7 KB

references/fundamentals-anti-patterns.md

View on GitHub
# Anti-Patterns

Common mistakes and fixes.

| Anti-Pattern | Problem | Fix |
| ------------ | ------- | --- |
| Generic metrics | Pre-built scores don't match your failures | Build from error analysis |
| Vibe-based | No quantification | Measure with experiments |
| Ignoring humans | Uncalibrated LLM judges | Validate >80% TPR/TNR |
| Premature automation | Evaluators for imagined problems | Let observed failures drive |
| Saturation blindness | 100% pass = no signal | Keep capability evals at 50-80% |
| Similarity metrics | BERTScore/ROUGE for generation | Use for retrieval only |
| Model switching | Hoping a model works better | Error analysis first |
| Single-run scoring | LLM judges and non-deterministic tasks add per-run noise that can drown the signal from a prompt change on a small dataset | Set `repetitions` on `runExperiment` (or grow the dataset) when the task or judge is an LLM call |

## Quantify Changes

```python
from phoenix.client import Client

client = Client()
baseline = client.experiments.run_experiment(dataset=dataset, task=old_prompt, evaluators=evaluators)
improved = client.experiments.run_experiment(dataset=dataset, task=new_prompt, evaluators=evaluators)
print(f"Improvement: {improved.pass_rate - baseline.pass_rate:+.1%}")
```

## Don't Use Similarity for Generation

```python
# BAD
score = bertscore(output, reference)

# GOOD
correct_facts = check_facts_against_source(output, context)
```

## Error Analysis Before Model Change

```python
# BAD
for model in models:
    results = test(model)

# GOOD
failures = analyze_errors(results)
# Then decide if model change is warranted
```
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall