Skip to content

github/awesome-copilot / skills/phoenix-evals

phoenix-evals

Verified

Build and run evaluators for AI/LLM applications using Phoenix.

FileKindSize
SKILL.mdskill md4.5 KB
references/axial-coding.mdmarkdown2.3 KB
references/common-mistakes-python.mdmarkdown7.0 KB
references/error-analysis-multi-turn.mdmarkdown1.3 KB
references/error-analysis.mdmarkdown4.2 KB
references/evaluate-dataframe-python.mdmarkdown4.4 KB
references/evaluators-code-python.mdmarkdown3.0 KB
references/evaluators-code-typescript.mdmarkdown1.2 KB
references/evaluators-custom-templates.mdmarkdown1.2 KB
references/evaluators-llm-python.mdmarkdown2.6 KB
references/evaluators-llm-typescript.mdmarkdown1.4 KB
references/evaluators-overview.mdmarkdown1.1 KB
references/evaluators-pre-built.mdmarkdown2.4 KB
references/evaluators-rag.mdmarkdown3.1 KB
references/experiments-datasets-python.mdmarkdown4.8 KB
references/experiments-datasets-typescript.mdmarkdown2.7 KB
references/experiments-overview.mdmarkdown1.7 KB
references/experiments-running-python.mdmarkdown2.9 KB
references/experiments-running-typescript.mdmarkdown3.1 KB
references/experiments-synthetic-python.mdmarkdown1.8 KB
references/experiments-synthetic-typescript.mdmarkdown2.2 KB
references/fundamentals-anti-patterns.mdmarkdown1.6 KB
references/fundamentals-model-selection.mdmarkdown1.4 KB
references/fundamentals.mdmarkdown1.8 KB
references/observe-sampling-python.mdmarkdown2.4 KB
references/observe-sampling-typescript.mdmarkdown3.9 KB
references/observe-tracing-setup.mdmarkdown3.3 KB
references/production-continuous.mdmarkdown3.6 KB
references/production-guardrails.mdmarkdown1.3 KB
references/production-overview.mdmarkdown2.3 KB
references/setup-python.mdmarkdown1.9 KB
references/setup-typescript.mdmarkdown981 B
references/validation-evaluators-python.mdmarkdown1.0 KB
references/validation-evaluators-typescript.mdmarkdown4.4 KB
references/validation.mdmarkdown1.7 KB

references/evaluators-pre-built.md

View on GitHub
# Evaluators: Pre-Built

Use for exploration only. Validate before production.

## Python

```python
from phoenix.evals import LLM
from phoenix.evals.metrics import FaithfulnessEvaluator

llm = LLM(provider="openai", model="gpt-4o")
faithfulness_eval = FaithfulnessEvaluator(llm=llm)
```

**Note**: `HallucinationEvaluator` is deprecated. Use `FaithfulnessEvaluator` instead.
It uses "faithful"/"unfaithful" labels with score 1.0 = faithful.

## TypeScript

```typescript
import { createHallucinationEvaluator } from "@arizeai/phoenix-evals";
import { openai } from "@ai-sdk/openai";

const hallucinationEval = createHallucinationEvaluator({ model: openai("gpt-4o") });
```

## Available (2.0)

| Evaluator | Type | Description |
| --------- | ---- | ----------- |
| `FaithfulnessEvaluator` | LLM | Is the response faithful to the context? |
| `CorrectnessEvaluator` | LLM | Is the response correct? |
| `DocumentRelevanceEvaluator` | LLM | Are retrieved documents relevant? |
| `ToolSelectionEvaluator` | LLM | Did the agent select the right tool? |
| `ToolInvocationEvaluator` | LLM | Did the agent invoke the tool correctly? |
| `ToolResponseHandlingEvaluator` | LLM | Did the agent handle the tool response well? |
| `MatchesRegex` | Code | Does output match a regex pattern? |
| `PrecisionRecallFScore` | Code | Precision/recall/F-score metrics |
| `exact_match` | Code | Exact string match |

Legacy evaluators (`HallucinationEvaluator`, `QAEvaluator`, `RelevanceEvaluator`,
`ToxicityEvaluator`, `SummarizationEvaluator`) are in `phoenix.evals.legacy` and deprecated.

## When to Use

| Situation | Recommendation |
| --------- | -------------- |
| Exploration | Find traces to review |
| Find outliers | Sort by scores |
| Production | Validate first (>80% human agreement) |
| Domain-specific | Build custom |

## Exploration Pattern

```python
from phoenix.evals import evaluate_dataframe

results_df = evaluate_dataframe(dataframe=traces, evaluators=[faithfulness_eval])

# Score columns contain dicts — extract numeric scores
scores = results_df["faithfulness_score"].apply(
    lambda x: x.get("score", 0.0) if isinstance(x, dict) else 0.0
)
low_scores = results_df[scores < 0.5]   # Review these
high_scores = results_df[scores > 0.9]  # Also sample
```

## Validation Required

```python
from sklearn.metrics import classification_report

print(classification_report(human_labels, evaluator_results["label"]))
# Target: >80% agreement
```
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall