github/awesome-copilot / skills/phoenix-evals
phoenix-evals
VerifiedBuild and run evaluators for AI/LLM applications using Phoenix.
references/fundamentals-model-selection.md
View on GitHub# Model Selection
Error analysis first, model changes last.
## Decision Tree
```
Performance Issue?
│
▼
Error analysis suggests model problem?
NO → Fix prompts, retrieval, tools
YES → Is it a capability gap?
YES → Consider model change
NO → Fix the actual problem
```
## Judge Model Selection
| Principle | Action |
| --------- | ------ |
| Start capable | Use gpt-4o first |
| Optimize later | Test cheaper after criteria stable |
| Same model OK | Judge does different task |
```python
# Start with capable model
judge = ClassificationEvaluator(
llm=LLM(provider="openai", model="gpt-4o"),
...
)
# After validation, test cheaper
judge_cheap = ClassificationEvaluator(
llm=LLM(provider="openai", model="gpt-4o-mini"),
...
)
# Compare TPR/TNR on same test set
```
## Don't Model Shop
```python
from phoenix.client import Client
client = Client()
# BAD
for model in ["gpt-4o", "claude-3", "gemini-pro"]:
results = client.experiments.run_experiment(
dataset=dataset,
task=lambda input, _model=model: task(input, model=_model),
evaluators=evaluators,
)
# GOOD
failures = analyze_errors(results)
# "Ignores context" → Fix prompt
# "Can't do math" → Maybe try better model
```
## When Model Change Is Warranted
- Failures persist after prompt optimization
- Capability gaps (reasoning, math, code)
- Error analysis confirms model limitation
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |