github/awesome-copilot / skills/phoenix-evals
phoenix-evals
VerifiedBuild and run evaluators for AI/LLM applications using Phoenix.
references/validation-evaluators-python.md
View on GitHub# Validating Evaluators (Python)
Validate LLM evaluators against human-labeled examples. Target >80% TPR/TNR/Accuracy.
## Calculate Metrics
```python
from sklearn.metrics import classification_report, confusion_matrix
print(classification_report(human_labels, evaluator_predictions))
cm = confusion_matrix(human_labels, evaluator_predictions)
tn, fp, fn, tp = cm.ravel()
tpr = tp / (tp + fn)
tnr = tn / (tn + fp)
print(f"TPR: {tpr:.2f}, TNR: {tnr:.2f}")
```
## Correct Production Estimates
```python
def correct_estimate(observed, tpr, tnr):
"""Adjust observed pass rate using known TPR/TNR."""
return (observed - (1 - tnr)) / (tpr - (1 - tnr))
```
## Find Misclassified
```python
# False Positives: Evaluator pass, human fail
fp_mask = (evaluator_predictions == 1) & (human_labels == 0)
false_positives = dataset[fp_mask]
# False Negatives: Evaluator fail, human pass
fn_mask = (evaluator_predictions == 0) & (human_labels == 1)
false_negatives = dataset[fn_mask]
```
## Red Flags
- TPR or TNR < 70%
- Large gap between TPR and TNR
- Kappa < 0.6
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |