Skip to content

BuilderIO/agent-native / packages/core/docs

agent-native-docs

Unverified

How to look up version-matched Agent-Native framework docs and source in node_modules. Use before coding against @agent-native/core APIs or advanced features.

FileKindSize
AGENTS.mdmarkdown6.0 KB
SKILL.mdskill md2.2 KB
content/a2a-protocol.mdxmarkdown37.6 KB
content/actions-access-control.mdxmarkdown28.9 KB
content/actions-advanced.mdxmarkdown13.1 KB
content/actions-agent-tools.mdxmarkdown5.6 KB
content/actions-defining.mdxmarkdown31.2 KB
content/actions-other-surfaces.mdxmarkdown10.3 KB
content/actions-overview.mdxmarkdown8.3 KB
content/actions-run-context.mdxmarkdown9.1 KB
content/add-ui-to-your-agent.mdxmarkdown5.0 KB
content/administered-deployments.mdxmarkdown7.7 KB
content/agent-mentions.mdxmarkdown10.2 KB
content/agent-native-config.mdxmarkdown34.9 KB
content/agent-native-toolkit.mdxmarkdown17.3 KB
content/agent-resources.mdxmarkdown27.0 KB
content/agent-surfaces.mdxmarkdown33.8 KB
content/agent-teams.mdxmarkdown13.4 KB
content/agent-web-surfaces.mdxmarkdown10.7 KB
content/agentkit.mdxmarkdown14.0 KB
content/audit-log.mdxmarkdown14.3 KB
content/authentication.mdxmarkdown30.7 KB
content/automation-connectors.mdxmarkdown7.8 KB
content/automations.mdxmarkdown26.7 KB
content/aws-amplify.mdxmarkdown10.7 KB
content/aws-lambda.mdxmarkdown13.9 KB
content/aws-rds.mdxmarkdown1.7 KB
content/azure-postgresql.mdxmarkdown1.8 KB
content/azure-static-web-apps.mdxmarkdown9.1 KB
content/blueprint-installer.mdxmarkdown6.3 KB
content/cli-adapters.mdxmarkdown5.7 KB
content/client-advanced.mdxmarkdown13.1 KB
content/client-agent-chat.mdxmarkdown17.7 KB
content/client-data.mdxmarkdown19.3 KB
content/client-entry-points.mdxmarkdown1.2 KB
content/client-overview.mdxmarkdown10.6 KB
content/client-routing.mdxmarkdown5.7 KB
content/client-sync-internals.mdxmarkdown5.2 KB
content/cloneable-saas.mdxmarkdown16.6 KB
content/cloud-sql.mdxmarkdown2.3 KB
content/cloudflare.mdxmarkdown8.3 KB
content/code-agents-ui.mdxmarkdown26.6 KB
content/components.mdxmarkdown42.9 KB
content/context-awareness.mdxmarkdown24.9 KB
content/create-with-an-agent.mdxmarkdown8.8 KB
content/creating-templates.mdxmarkdown21.6 KB
content/cross-app-connect-design.mdxmarkdown8.4 KB
content/cross-app-sso.mdxmarkdown8.7 KB
content/custom-design-system.mdxmarkdown17.3 KB
content/data-programs.mdxmarkdown25.0 KB
content/decisions.mdxmarkdown7.9 KB
content/deno-deploy.mdxmarkdown7.2 KB
content/deploy-an-app.mdxmarkdown5.0 KB
content/deployment-environment-variables.mdxmarkdown16.3 KB
content/deployment.mdxmarkdown11.3 KB
content/dispatch.mdxmarkdown18.6 KB
content/docker.mdxmarkdown8.1 KB
content/docs-components.mdxmarkdown28.9 KB
content/doctor.mdxmarkdown21.3 KB
content/drafts/clips-experimental-features.mdmarkdown5.4 KB
content/drop-in-agent.mdxmarkdown16.0 KB
content/durable-background-runs.mdxmarkdown13.8 KB
content/durable-resume.mdxmarkdown10.0 KB
content/embedding-sdk.mdxmarkdown31.3 KB
content/environment-variables.mdxmarkdown29.9 KB
content/evals.mdxmarkdown12.4 KB
content/extensions.mdxmarkdown34.8 KB
content/external-agents-catalog.mdxmarkdown40.5 KB
content/external-agents.mdxmarkdown29.3 KB
content/file-uploads.mdxmarkdown10.7 KB
content/generative-ui.mdxmarkdown10.8 KB
content/getting-started-actions.mdxmarkdown2.7 KB
content/getting-started-pages.mdxmarkdown2.6 KB
content/getting-started.mdxmarkdown7.2 KB
content/harness-agents.mdxmarkdown16.9 KB
content/http-api.mdxmarkdown20.9 KB
content/integrations.mdxmarkdown33.8 KB
content/internationalization.mdxmarkdown6.2 KB
content/key-concepts.mdxmarkdown30.6 KB
content/koyeb.mdxmarkdown7.3 KB
content/locales/ar-SA/a2a-protocol.mdxmarkdown48.5 KB
content/locales/ar-SA/actions-access-control.mdxmarkdown40.8 KB
content/locales/ar-SA/actions-advanced.mdxmarkdown16.3 KB
content/locales/ar-SA/actions-agent-tools.mdxmarkdown8.1 KB
content/locales/ar-SA/actions-defining.mdxmarkdown42.2 KB
content/locales/ar-SA/actions-other-surfaces.mdxmarkdown14.5 KB
content/locales/ar-SA/actions-overview.mdxmarkdown11.3 KB
content/locales/ar-SA/actions-run-context.mdxmarkdown11.1 KB
content/locales/ar-SA/add-ui-to-your-agent.mdxmarkdown3.7 KB
content/locales/ar-SA/administered-deployments.mdxmarkdown6.4 KB
content/locales/ar-SA/agent-mentions.mdxmarkdown11.8 KB
content/locales/ar-SA/agent-native-toolkit.mdxmarkdown17.3 KB
content/locales/ar-SA/agent-resources.mdxmarkdown36.8 KB
content/locales/ar-SA/agent-surfaces.mdxmarkdown48.4 KB
content/locales/ar-SA/agent-teams.mdxmarkdown18.5 KB
content/locales/ar-SA/agent-web-surfaces.mdxmarkdown14.3 KB
content/locales/ar-SA/audit-log.mdxmarkdown15.9 KB
content/locales/ar-SA/authentication.mdxmarkdown39.8 KB
content/locales/ar-SA/automations.mdxmarkdown31.3 KB
content/locales/ar-SA/aws-amplify.mdxmarkdown14.9 KB
content/locales/ar-SA/aws-lambda.mdxmarkdown18.9 KB
content/locales/ar-SA/aws-rds.mdxmarkdown319 B
content/locales/ar-SA/azure-postgresql.mdxmarkdown344 B
content/locales/ar-SA/azure-static-web-apps.mdxmarkdown12.4 KB
content/locales/ar-SA/blueprint-installer.mdxmarkdown8.4 KB
content/locales/ar-SA/cli-adapters.mdxmarkdown6.7 KB
content/locales/ar-SA/client-advanced.mdxmarkdown17.8 KB
content/locales/ar-SA/client-agent-chat.mdxmarkdown23.0 KB
content/locales/ar-SA/client-data.mdxmarkdown23.8 KB
content/locales/ar-SA/client-entry-points.mdxmarkdown1.6 KB
content/locales/ar-SA/client-overview.mdxmarkdown13.9 KB
content/locales/ar-SA/client-routing.mdxmarkdown7.2 KB
content/locales/ar-SA/client-sync-internals.mdxmarkdown7.3 KB
content/locales/ar-SA/cloneable-saas.mdxmarkdown23.2 KB
content/locales/ar-SA/cloud-sql.mdxmarkdown317 B
content/locales/ar-SA/cloudflare.mdxmarkdown11.9 KB
content/locales/ar-SA/code-agents-ui.mdxmarkdown34.2 KB
content/locales/ar-SA/components.mdxmarkdown51.5 KB
content/locales/ar-SA/context-awareness.mdxmarkdown33.1 KB
content/locales/ar-SA/create-with-an-agent.mdxmarkdown12.2 KB
content/locales/ar-SA/creating-templates.mdxmarkdown26.8 KB
content/locales/ar-SA/cross-app-connect-design.mdxmarkdown1.5 KB
content/locales/ar-SA/cross-app-sso.mdxmarkdown9.7 KB
content/locales/ar-SA/custom-design-system.mdxmarkdown21.6 KB
content/locales/ar-SA/data-programs.mdxmarkdown33.3 KB
content/locales/ar-SA/decisions.mdxmarkdown9.1 KB
content/locales/ar-SA/deno-deploy.mdxmarkdown10.1 KB
content/locales/ar-SA/deploy-an-app.mdxmarkdown7.1 KB
content/locales/ar-SA/deployment-environment-variables.mdxmarkdown21.3 KB
content/locales/ar-SA/deployment.mdxmarkdown13.9 KB
content/locales/ar-SA/dispatch.mdxmarkdown27.3 KB
content/locales/ar-SA/docker.mdxmarkdown11.3 KB
content/locales/ar-SA/doctor.mdxmarkdown3.4 KB
content/locales/ar-SA/drop-in-agent.mdxmarkdown20.8 KB
content/locales/ar-SA/durable-resume.mdxmarkdown15.4 KB
content/locales/ar-SA/embedding-sdk.mdxmarkdown41.3 KB
content/locales/ar-SA/evals.mdxmarkdown13.3 KB
content/locales/ar-SA/extensions.mdxmarkdown34.0 KB
content/locales/ar-SA/external-agents.mdxmarkdown89.3 KB
content/locales/ar-SA/file-uploads.mdxmarkdown13.4 KB
content/locales/ar-SA/getting-started-actions.mdxmarkdown3.2 KB
content/locales/ar-SA/getting-started-pages.mdxmarkdown3.2 KB
content/locales/ar-SA/getting-started.mdxmarkdown9.7 KB
content/locales/ar-SA/harness-agents.mdxmarkdown22.3 KB
content/locales/ar-SA/internationalization.mdxmarkdown8.2 KB
content/locales/ar-SA/key-concepts.mdxmarkdown42.0 KB
content/locales/ar-SA/koyeb.mdxmarkdown10.2 KB
content/locales/ar-SA/mcp-apps.mdxmarkdown33.7 KB
content/locales/ar-SA/mcp-clients.mdxmarkdown35.0 KB
content/locales/ar-SA/mcp-protocol.mdxmarkdown27.6 KB
content/locales/ar-SA/messaging.mdxmarkdown59.2 KB
content/locales/ar-SA/multi-app-workspace.mdxmarkdown36.2 KB
content/locales/ar-SA/multi-tenancy.mdxmarkdown7.5 KB
content/locales/ar-SA/native-chat-ui.mdxmarkdown21.3 KB
content/locales/ar-SA/neon.mdxmarkdown244 B
content/locales/ar-SA/netlify.mdxmarkdown15.2 KB
content/locales/ar-SA/node-docker.mdxmarkdown878 B
content/locales/ar-SA/node-js.mdxmarkdown8.4 KB
content/locales/ar-SA/notifications.mdxmarkdown16.2 KB
content/locales/ar-SA/observability.mdxmarkdown45.2 KB
content/locales/ar-SA/observational-memory.mdxmarkdown10.9 KB
content/locales/ar-SA/onboarding.mdxmarkdown16.9 KB
content/locales/ar-SA/organizations-teams-permissions.mdxmarkdown5.9 KB
content/locales/ar-SA/package-lifecycle.mdxmarkdown3.0 KB
content/locales/ar-SA/plan-plugin.mdxmarkdown23.0 KB
content/locales/ar-SA/portal.mdxmarkdown636 B
content/locales/ar-SA/postgres.mdxmarkdown487 B
content/locales/ar-SA/pr-visual-recap.mdxmarkdown53.9 KB
content/locales/ar-SA/privacy-and-data-rights.mdxmarkdown6.2 KB
content/locales/ar-SA/processors.mdxmarkdown8.7 KB
content/locales/ar-SA/progress.mdxmarkdown10.7 KB
content/locales/ar-SA/pure-agent-apps.mdxmarkdown5.0 KB
content/locales/ar-SA/real-time-collaboration.mdxmarkdown55.6 KB
content/locales/ar-SA/recorder.mdxmarkdown16.7 KB
content/locales/ar-SA/recurring-jobs.mdxmarkdown23.8 KB
content/locales/ar-SA/render.mdxmarkdown11.2 KB
content/locales/ar-SA/sandbox-adapters.mdxmarkdown18.0 KB
content/locales/ar-SA/security.mdxmarkdown31.5 KB
content/locales/ar-SA/server-database.mdxmarkdown27.6 KB
content/locales/ar-SA/server-middleware.mdxmarkdown7.9 KB
content/locales/ar-SA/server-overview.mdxmarkdown12.2 KB
content/locales/ar-SA/server-plugins.mdxmarkdown9.1 KB
content/locales/ar-SA/server-routes.mdxmarkdown14.3 KB
content/locales/ar-SA/sharing.mdxmarkdown28.6 KB
content/locales/ar-SA/skills-guide.mdxmarkdown27.0 KB
content/locales/ar-SA/ssr-caching.mdxmarkdown7.4 KB
content/locales/ar-SA/supabase.mdxmarkdown264 B
content/locales/ar-SA/syncing-template-changes.mdxmarkdown11.7 KB
content/locales/ar-SA/telemetry.mdxmarkdown21.9 KB
content/locales/ar-SA/template-analytics.mdxmarkdown26.4 KB
content/locales/ar-SA/template-assets.mdxmarkdown31.9 KB
content/locales/ar-SA/template-brain.mdxmarkdown29.7 KB
content/locales/ar-SA/template-calendar-agent.mdxmarkdown10.2 KB
content/locales/ar-SA/template-calendar-developers.mdxmarkdown33.4 KB
content/locales/ar-SA/template-calendar-features.mdxmarkdown14.4 KB
content/locales/ar-SA/template-calendar-integrations.mdxmarkdown6.2 KB
content/locales/ar-SA/template-calendar.mdxmarkdown4.4 KB
content/locales/ar-SA/template-chat-agent.mdxmarkdown4.7 KB
content/locales/ar-SA/template-chat-developers.mdxmarkdown31.3 KB
content/locales/ar-SA/template-chat-features.mdxmarkdown9.9 KB

content/evals.mdx

View on GitHub
---
title: "CI Eval Gate"
description: "Write *.eval.ts test cases that run the real agent against fixed inputs, score the output with composable scorers, and gate CI/deploys on a threshold."
---

# CI Eval Gate

Evals are a first-class testing primitive: you declare a prompt plus the behavior you expect, the runner **actually runs the agent loop** against that input, scores the output with composable scorers, and exits non-zero if any case scores below its threshold. That non-zero exit makes `agent-native eval` a drop-in CI deploy gate.

This is complementary to the post-hoc scoring in [Observability](/docs/observability):

- **Observability evals** (`observability/evals.ts`) — _"how did this real run do?"_ Passive, sampled, lives next to traces.
- **`*.eval.ts` (this primitive)** — _"does the agent do the right thing on this fixed input?"_ Active, deterministic, a CI gate run via the CLI.
- **Promote** — turn a completed production trace into one of those `*.eval.ts` cases so a staging miss can fail CI the next time.

The runner resolves a provider-agnostic engine/model from the existing registry — no model is hardcoded — so the same suite runs against whatever engine the app is configured for.

<Diagram id="doc-block-eval-flow" title="From a production trace to a CI gate" summary="Promote a completed run into a *.eval.ts case, run the agent, and block deploy when a scorer misses its threshold.">

```html
<div class="eval-flow">
  <div class="diagram-node">
    Production trace<br /><small class="diagram-muted">completed run</small>
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-node">
    Promote<br /><small class="diagram-muted">eval promote --write</small>
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-node">
    *.eval.ts<br /><small class="diagram-muted"
      >prompt + expected behavior</small
    >
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-node">
    Run the agent loop<br /><small class="diagram-muted"
      >real engine/model</small
    >
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-node">
    Scorers<br /><small class="diagram-muted"
      >every one must pass threshold</small
    >
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-col">
    <div class="diagram-box ok">exit 0 &rarr; deploy</div>
    <div class="diagram-box warn">exit 1 &rarr; block</div>
  </div>
</div>
```

```css
.eval-flow {
  display: flex;
  align-items: center;
  gap: 12px;
  flex-wrap: wrap;
}
.eval-flow .diagram-node {
  display: flex;
  flex-direction: column;
  gap: 2px;
  padding: 10px 14px;
}
.eval-flow .diagram-col {
  display: flex;
  flex-direction: column;
  gap: 8px;
}
.eval-flow .diagram-arrow {
  font-size: 22px;
  line-height: 1;
}
```

</Diagram>

## Writing an eval {#writing}

Drop a `*.eval.ts` file anywhere in the app (or an `evals/*.ts` file). Each file `export default defineEval(...)` (or exports an array of them):

```ts filename="evals/greeting.eval.ts"
import { defineEval, contains, llmJudge } from "@agent-native/core/eval";

export default defineEval({
  name: "greets the user by name",
  input: { prompt: "Say hi to Ada." },
  threshold: 0.7, // per-scorer pass bar; default 0.5
  scorers: [
    contains("Ada"),
    llmJudge({ criteria: "friendliness", rubric: "1.0 = warm greeting" }),
  ],
});
```

An eval passes only when **every** scorer meets the threshold. Key `defineEval` fields:

| Field        | Type                  | Notes                                                                                                           |
| ------------ | --------------------- | --------------------------------------------------------------------------------------------------------------- |
| `name`       | string                | Required. Shown in the report.                                                                                  |
| `input`      | `{ prompt, history }` | Required `prompt`; optional prior `{ role, text }` turns.                                                       |
| `scorers`    | `Scorer[]`            | Required, at least one, unless `skipReason` is set.                                                             |
| `threshold`  | number `0..1`         | Per-scorer pass bar. Default `0.5`; overridable from the CLI.                                                   |
| `run`        | function              | Optional override for custom setup (seed data, multi-turn).                                                     |
| `skipReason` | string                | Optional reason to skip this eval without running the agent; useful for manual/nightly gates that need secrets. |

The agent run handed to scorers is small and transport-agnostic:

```ts
interface AgentRunOutput {
  text: string; // concatenated assistant text
  toolCalls: readonly string[]; // tool/action names, in call order
  ok: boolean; // completed without a terminal error
  error?: string;
  runId: string;
  durationMs: number;
}
```

## Built-in scorers {#built-in}

Imported from `@agent-native/core/eval`:

| Scorer                   | Score                                                             | Model? |
| ------------------------ | ----------------------------------------------------------------- | ------ |
| `exactMatch(expected)`   | `1.0` if text equals `expected` (trimmed, case-insensitive)       | No     |
| `contains(needles)`      | Fraction of required substrings present (so partial hits surface) | No     |
| `usesTool(toolName)`     | `1.0` if the agent invoked that tool/action at least once         | No     |
| `llmJudge({ criteria })` | LLM-as-judge scored against a natural-language rubric, → `0..1`   | Yes    |

`exactMatch` and `contains` take an optional `{ caseSensitive }`. `llmJudge` takes `{ criteria, rubric?, name?, scoreRange? }` — its output is normalized to `[0, 1]`, and the judge model is whatever the runner resolved (never a hardcoded provider).

## Custom scorers: the 4-step pipeline {#custom}

`createScorer` builds a scorer from a Mastra-style 4-step pipeline. Only `generateScore` is required:

<Diagram id="doc-block-x7o5bx" title="The 4-step scorer pipeline" summary={"preprocess and analyze default to identity; only generateScore is required. analyze can run plain JS or call an LLM judge via ctx."}>

```html
<div class="scorer">
  <div class="diagram-card">
    <span class="diagram-pill">preprocess(run)</span
    ><small class="diagram-muted"
      >transform the run/output &middot; optional</small
    >
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-card">
    <span class="diagram-pill">analyze(x, ctx)</span
    ><small class="diagram-muted"
      >plain JS or LLM judge &middot; optional</small
    >
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-card">
    <span class="diagram-pill accent">generateScore(a)</span
    ><small class="diagram-muted"
      >&rarr; 0..1 normalized &middot; <strong>required</strong></small
    >
  </div>
  <div class="diagram-arrow diagram-muted" aria-hidden="true">&rarr;</div>
  <div class="diagram-card">
    <span class="diagram-pill">generateReason</span
    ><small class="diagram-muted">human-readable why &middot; optional</small>
  </div>
</div>
```

```css
.scorer {
  display: flex;
  align-items: center;
  gap: 10px;
  flex-wrap: wrap;
}
.scorer .diagram-card {
  display: flex;
  flex-direction: column;
  gap: 2px;
  padding: 8px 12px;
}
.scorer .diagram-arrow {
  font-size: 20px;
  line-height: 1;
}
```

</Diagram>

```text
preprocess(run)     → x          transform the run/output (optional)
analyze(x, ctx)     → analysis   plain JS OR an LLM judge (optional)
generateScore(a)    → 0..1       REQUIRED, normalized
generateReason(...) → string     human-readable why (optional)
```

`preprocess` and `analyze` default to identity (the scorer sees the raw `AgentRunOutput`). The `analyze` step receives a `ctx` with a provider-agnostic `judge()` helper for LLM-backed scoring:

```ts
import { createScorer, clamp01 } from "@agent-native/core/eval";

// A scorer that rewards short, tool-using answers.
const concise = createScorer({
  name: "concise_with_tool",
  analyze(run) {
    return {
      words: run.text.trim().split(/\s+/).length,
      usedTool: run.toolCalls.length > 0,
    };
  },
  generateScore({ words, usedTool }) {
    if (!usedTool) return 0;
    return clamp01(1 - Math.max(0, words - 40) / 200);
  },
  generateReason({ analysis }) {
    return `${analysis.words} words, tool used: ${analysis.usedTool}`;
  },
});
```

## Running the gate {#cli}

```bash
agent-native eval                    # run every *.eval.ts; non-zero exit on failure
agent-native eval billing            # only files whose path contains "billing"
agent-native eval --json             # machine-readable report (for CI)
agent-native eval --threshold 0.8    # override every eval's pass threshold (0..1)
```

The command discovers `**/*.eval.ts` and `evals/*.ts` under the current app, runs the agent for each input, scores it, prints a readable table (or JSON), and **exits non-zero if any eval scores below its threshold**.

Exit codes:

| Code | Meaning                                                                          |
| ---- | -------------------------------------------------------------------------------- |
| `0`  | All evals passed, evals were skipped, or no eval files were found (CI-friendly). |
| `1`  | At least one eval scored below threshold, or the suite errored.                  |
| `2`  | Bad arguments (e.g. `--threshold` outside `[0, 1]`).                             |

### Skipping gated evals {#skipping}

Use `skipReason` when an eval suite is intentionally gated behind manual,
nightly, or secret-backed execution:

```ts
export default defineEval({
  name: "provider-backed source sync",
  input: { prompt: "Refresh the source and summarize the changes." },
  skipReason: process.env.SOURCE_SYNC_EVALS
    ? undefined
    : "Skipped because SOURCE_SYNC_EVALS is unset",
  scorers: process.env.SOURCE_SYNC_EVALS ? [contains("updated")] : [],
});
```

Skipped evals do not run the agent or scorers, do not fail the suite, and are
reported distinctly with `status: "skipped"` and `skipReason` in JSON output.
Use this for expensive or credential-dependent evals; keep cheap deterministic
coverage in normal PR CI.

### As a CI deploy gate {#ci}

Add it to the pipeline that runs before a deploy (excerpt):

```yaml filename=".github/workflows/deploy.yml"
- run: npx agent-native eval --json
```

A regression that drops any scorer below threshold fails the step and blocks the deploy. An app with no eval files exits `0`, so adopting evals is opt-in per app.

## From a production trace {#from-trace}

When a real run is surprising — it failed in production after working in a demo, or it called the wrong tool — promote that completed trace into a CI case instead of inventing a prompt.

```bash
agent-native eval promote <runId> --write evals/from-trace.eval.ts
```

That command persists an `EvalDataset` row (the same helper the dashboard and the `promote-trace-eval` action use) and writes a `defineEval` fixture the existing `agent-native eval` gate already discovers. Default scorers require each tool that succeeded on the trace to complete successfully again on replay — a call that only starts, or that finishes with an error, does not pass — plus an optional `--must-contain` substring. Truncated, aborted, or prompt-less runs fail closed: they return a typed error and insert nothing.

The hosted action and `POST /_agent-native/observability/traces/:runId/promote` never write the app tree. Use `--write` (or copy the returned JSON into a `*.eval.ts`) when you want CI to gate on the case. The CLI and the signed-in action share one dataset per run.

The Observability dashboard's trace detail has a **Promote to eval** control that calls the same endpoint. A run with no successful tool call needs the reply text that case must contain; the control accepts that text before promoting.

## What's next

- [**Observability**](/docs/observability) — post-hoc scoring of real production runs (the complementary layer)
- [**Actions**](/docs/actions-overview) — the tools/actions that show up in `toolCalls`
- [**Agent Teams**](/docs/agent-teams) — sub-agents an eval might exercise
rea-changelog-updatemorluto/reaPrepare or rewrite REA release changelogs and GitHub release notes from pinned Git history, with verified contributor thanks and Release Please synchronization.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
academy-guideanthropics/skillsStop and check this skill before finishing any reply to a question about how to use Claude or a Claude product — it recommends matching courses, tutorials, and use cases from Claude Academy (academy.claude.com), Anthropic's learning hub. Trigger on: "how do I", "how can I", "getting started with", "what can Claude do", "teach me", "learn to use"; questions about artifacts, projects, skills, plugins, connectors, MCP; requests about rolling Claude out to a team, class, or organization; and any ask for training materials, onboarding content, or learning resources. Use it when the user is learning how to use a feature or product — not when they are mid-task and just want the task done. This skill composes with other skills: after consulting product documentation to answer how a Claude feature works, also check here for a matching course or tutorial — a docs-grounded answer and an Academy recommendation belong together. Only recommend on a strong match; never invent Academy content.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
claude-apianthropics/skillsReference for the Claude API / Anthropic SDK — model ids, pricing, params, streaming, tool use, MCP, agents, caching, token counting, model migration. TRIGGER — read BEFORE opening the target file; don't skip because it "looks like a one-liner" — whenever: the prompt names Claude/Anthropic in any form (Claude, Anthropic, Fable, Opus, Sonnet, Haiku, `anthropic`, `@anthropic-ai`, `claude-*`, `us.anthropic.*`, `[1m]`); the user asks about an LLM (pricing/model choice/limits/caching) — never answer from memory; OR the task is LLM-shaped with provider unstated (agent/MCP/tool-definition/multi-agent/RAG/LLM-judge/computer-use; generate/summarize/extract/classify/rewrite/converse over NL; debugging refusals/cutoffs/streaming/tool-calls/tokens). SKIP only when another provider is being worked on (overrides all triggers): OpenAI/GPT/Gemini/Llama/Mistral/Cohere/Ollama named in the query; OR `grep -rE 'openai|langchain_openai|google.generativeai|genai|mistralai|cohere|ollama'` over the project hits (run this grep FIRST if no provider named — don't Read the file).Stars: 197kUpdated Claude CodeCodexUnverifiedInstall
doc-coauthoringanthropics/skillsGuide users through a structured workflow for co-authoring documentation. Use when user wants to write documentation, proposals, technical specs, decision docs, or similar structured content. This workflow helps users efficiently transfer context, refine content through iteration, and verify the doc works for readers. Trigger when user mentions writing docs, creating proposals, drafting specs, or similar documentation tasks.Stars: 197kUpdated Claude CodeCodexVerifiedInstall