Skip to content

github/awesome-copilot / skills/phoenix-evals

phoenix-evals

Verified

Build and run evaluators for AI/LLM applications using Phoenix.

FileKindSize
SKILL.mdskill md4.5 KB
references/axial-coding.mdmarkdown2.3 KB
references/common-mistakes-python.mdmarkdown7.0 KB
references/error-analysis-multi-turn.mdmarkdown1.3 KB
references/error-analysis.mdmarkdown4.2 KB
references/evaluate-dataframe-python.mdmarkdown4.4 KB
references/evaluators-code-python.mdmarkdown3.0 KB
references/evaluators-code-typescript.mdmarkdown1.2 KB
references/evaluators-custom-templates.mdmarkdown1.2 KB
references/evaluators-llm-python.mdmarkdown2.6 KB
references/evaluators-llm-typescript.mdmarkdown1.4 KB
references/evaluators-overview.mdmarkdown1.1 KB
references/evaluators-pre-built.mdmarkdown2.4 KB
references/evaluators-rag.mdmarkdown3.1 KB
references/experiments-datasets-python.mdmarkdown4.8 KB
references/experiments-datasets-typescript.mdmarkdown2.7 KB
references/experiments-overview.mdmarkdown1.7 KB
references/experiments-running-python.mdmarkdown2.9 KB
references/experiments-running-typescript.mdmarkdown3.1 KB
references/experiments-synthetic-python.mdmarkdown1.8 KB
references/experiments-synthetic-typescript.mdmarkdown2.2 KB
references/fundamentals-anti-patterns.mdmarkdown1.6 KB
references/fundamentals-model-selection.mdmarkdown1.4 KB
references/fundamentals.mdmarkdown1.8 KB
references/observe-sampling-python.mdmarkdown2.4 KB
references/observe-sampling-typescript.mdmarkdown3.9 KB
references/observe-tracing-setup.mdmarkdown3.3 KB
references/production-continuous.mdmarkdown3.6 KB
references/production-guardrails.mdmarkdown1.3 KB
references/production-overview.mdmarkdown2.3 KB
references/setup-python.mdmarkdown1.9 KB
references/setup-typescript.mdmarkdown981 B
references/validation-evaluators-python.mdmarkdown1.0 KB
references/validation-evaluators-typescript.mdmarkdown4.4 KB
references/validation.mdmarkdown1.7 KB

references/fundamentals.md

View on GitHub
# Fundamentals

Application-specific tests for AI systems. Code first, LLM for nuance, human for truth.

## Evaluator Types

| Type | Speed | Cost | Use Case |
| ---- | ----- | ---- | -------- |
| **Code** | Fast | Cheap | Regex, JSON, format, exact match |
| **LLM** | Medium | Medium | Subjective quality, complex criteria |
| **Human** | Slow | Expensive | Ground truth, calibration |

**Decision:** Code first → LLM only when code can't capture criteria → Human for calibration.

## Score Structure

| Property | Required | Description |
| -------- | -------- | ----------- |
| `name` | Yes | Evaluator name |
| `kind` | Yes | `"code"`, `"llm"`, `"human"` |
| `score` | No* | 0-1 numeric |
| `label` | No* | `"pass"`, `"fail"` |
| `explanation` | No | Rationale |

*One of `score` or `label` required.

## Binary > Likert

Use pass/fail, not 1-5 scales. Clearer criteria, easier calibration.

```python
# Multiple binary checks instead of one Likert scale
evaluators = [
    AnswersQuestion(),    # Yes/No
    UsesContext(),        # Yes/No
    NoHallucination(),    # Yes/No
]
```

## Quick Patterns

### Code Evaluator

```python
from phoenix.evals import create_evaluator

@create_evaluator(name="has_citation", kind="code")
def has_citation(output: str) -> bool:
    return bool(re.search(r'\[\d+\]', output))
```

### LLM Evaluator

```python
from phoenix.evals import ClassificationEvaluator, LLM

evaluator = ClassificationEvaluator(
    name="helpfulness",
    prompt_template="...",
    llm=LLM(provider="openai", model="gpt-4o"),
    choices={"not_helpful": 0, "helpful": 1}
)
```

### Run Experiment

```python
from phoenix.client.experiments import run_experiment

experiment = run_experiment(
    dataset=dataset,
    task=my_task,
    evaluators=[evaluator1, evaluator2],
)
print(experiment.aggregate_scores)
```
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 197kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall