monte-carlo-data/mc-agent-toolkit / skills/monitoring-advisor
monte-carlo-monitoring-advisor
VerifiedAnalyze data coverage, create monitors for warehouse tables and AI agents. Covers coverage gaps, use-case analysis, data monitor creation, and agent observability.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 3.9 KB |
| SKILL.md | skill md | 21.1 KB |
| references/agent-evaluation-monitor.md | markdown | 32.7 KB |
| references/agent-metric-monitor.md | markdown | 16.0 KB |
| references/agent-monitor-creation.md | markdown | 25.6 KB |
| references/agent-span-fields.md | markdown | 5.1 KB |
| references/agent-trajectory-monitor.md | markdown | 16.2 KB |
| references/agent-validation-monitor.md | markdown | 11.0 KB |
| references/data-comparison-monitor.md | markdown | 19.2 KB |
| references/data-custom-sql-monitor.md | markdown | 12.1 KB |
| references/data-metric-monitor.md | markdown | 17.5 KB |
| references/data-monitor-creation.md | markdown | 22.7 KB |
| references/data-table-monitor.md | markdown | 10.9 KB |
| references/data-validation-monitor.md | markdown | 19.3 KB |
references/agent-monitor-creation.md
View on GitHub# Agent Monitor Creation Procedure
This is the agent monitor creation procedure for AI agent observability. Follow
these steps in order when a user asks to monitor their AI agents — setting up
alerts on agent behavior or creating agent monitors.
All tools are available via the `monte-carlo-mcp` MCP server.
---
## Step 1: Discover agents
Call `get_agent_metadata` to list all AI agents in the account. Present agent
names to the user (never expose MCONs or internal IDs). Ask which agent(s)
they want to monitor.
Key fields in the response:
| Field | Description |
|-------|-------------|
| `agentName` | Human-readable agent name |
| `agentReference` | The value to pass as the `agent` arg when creating monitors — a platform `{database}:{schema}.{name}` reference (Snowflake Cortex / Databricks) or an OpenTelemetry `service_name`. May be null for agents that cannot be referenced. |
| `traceTableMcon` | Trace table MCON — used as the `trace_table_mcon` input for the read tools (`get_agent_conversations`, `get_agent_conversation`, `get_agent_traces`, `get_agent_segments`; the parameter is named `mcon` on `get_agent_trace`) |
| `sourceType` | `TRACE_TABLE` (custom) or `PLATFORM_AGENT` (Monte Carlo native) |
| `backend_class` | Which backend the agent's traces live in — `ao_clickhouse_otel`, `platform_agent`, `customer_otel_trace_table`, `databricks_genie`, `databricks_mlflow_sdk`, or `databricks_mlflow_ka`. Null when the server could not classify the agent (or predates the field). |
| `warehouse_uuid` | Warehouse holding the agent's trace data — the value to pass as the `warehouse` arg when creating monitors. Null when the warehouse was deleted or cannot be resolved; fall back to `get_warehouses` (see Warehouse below). |
| `warehouse_name` | Display name of that warehouse — what you show the user. Null alongside `warehouse_uuid`; fall back to `get_warehouses`. |
**What `backend_class` tells you about capabilities:** conversation-grain
evaluation monitors (`is_agent_conversation_aggregation=True`) are supported for
`ao_clickhouse_otel`, `platform_agent` (Snowflake Cortex), and `databricks_genie`;
the MLflow classes are span-only (the backend rejects conversation aggregation for
them). At conversation grain, set `include_tool_calls: true` on every eval transform
by default — it adds the agent's tool calls (name, inputs, outputs, errors) to the
judged conversation as clearly identifiable TOOL entries, in call order, so evals
score what the agent did, not just what it said. Omit it only for pure style/tone
judges; the field is rejected at span grain. `databricks_genie` agents emit no
token or model data — skip token-usage
metrics for them (see `agent-metric-monitor.md`). A null `backend_class` means the
server couldn't classify the agent — default to span-grain proposals.
**Duplicate agent names:** The same agent name may appear more than once (e.g.,
deployed in both prod and staging). Each entry is distinguished by its own
`agentReference`, `traceTableMcon`, and warehouse — ask the user which one they
want to monitor and pass that entry's `agentReference` verbatim. When you ask the
user to choose, present each entry's `warehouse_name`, never a UUID.
---
## Step 2: Investigate agent behavior
Use the read tools to understand the agent's behavior before suggesting monitors.
These read tools are your sampling surface — do **not** query the trace store with
SQL. Agents are tracked as agents, not tables: platform agents (Snowflake Cortex /
Databricks) have no queryable trace table, and for custom agents the raw table's
columns are not the fields monitors use.
### 2a. Review recent conversations
Call `get_agent_conversations` with the agent's `agent_name` and `trace_table_mcon`
to list recent conversations (newest first). Filter to surface interesting ones —
`has_errors`, `status`, or turn/token/duration bounds — and set
`include_transcript=True` to read the prompt/completion transcripts inline. Drill
into one conversation you already have the id for with `get_agent_conversation`.
Look for:
- **Error patterns** — spans with error status or failure indicators
- **Latency outliers** — unusually long durations
- **Token usage** — high token counts that may indicate inefficiency
- **Conversation quality** — check prompt/completion text for relevance
### 2b. Inspect execution shape and traces
Call `get_agent_traces` to list traces with per-trace `workflows`, `tasks`,
`models`, `count_llm_calls`, `total_tokens`, `duration_seconds`, and error counts —
sort by the field you plan to monitor to see typical values and outliers. Call
`get_agent_segments` to enumerate the distinct `workflow` / `task` / `model` values
so you can scope a monitor to a real segment. Then pick a trace id and call
`get_agent_trace` to see the full span tree. Look for:
- **Excessive tool calls** — an agent calling the same tool many times
- **Missing steps** — expected spans that don't appear
- **Error cascades** — a failed span causing downstream failures
- **Unusual paths** — the agent taking an unexpected execution route
You are identifying **what** to monitor — you don't need exact percentiles up
front; anomaly-detection operators (`AUTO`) learn the baseline themselves.
### 2c. Summarize the agent before proposing
Condense the investigation into a short agent understanding and show it to the
user — every monitor you propose should trace back to an item in it:
- **Purpose** — 1–2 sentences on what this agent does, grounded in the sampled
transcripts (e.g. "a revenue-analytics assistant that answers questions about
bookings").
- **Conversational?** — multi-turn user conversations (eval-worthy for
satisfaction / task completion) vs. a batch / single-shot pipeline where
structural and span checks fit better.
- **Tools and the dominant span** — which tool spans the agent runs, and which
one does its core work (the SQL execution tool for an analytics agent,
retrieval for a RAG agent).
- **Healthy trajectory shape** — how many times the dominant span runs per answer
in healthy traces (the per-trace distribution and its max), typical turn counts,
and latency/token magnitudes. This is the basis for every derived threshold.
- **Recurring intents** — what users repeatedly ask (from the transcripts) —
seeds for custom conversation evals.
- **Observed failure modes** — what actually went wrong in the sample — seeds for
evals and structural monitors.
- **Existing monitors** — from `get_monitors`, so proposals don't duplicate
coverage.
---
## Propose with the POBC framing (walk the user through all four pillars)
When the user is setting up monitoring for an agent (rather than asking for one
specific monitor), structure the proposal around the four pillars of agent
observability — **Performance, Output, Behavior, Context (POBC)** — and walk
through them one at a time. Do NOT dump every proposed monitor in one
monolithic list.
### 1. Open with the framing
Before presenting any monitors, briefly explain the framework. Use this copy,
adapting the agent's actual name into the prose where it reads naturally:
> A quick word on how we think about agent observability. Agents fail in four
> distinct ways, so we monitor four distinct things — **Performance, Output,
> Behavior, and Context (POBC)**: how efficiently the agent answers, what it
> says, how it gets there, and the data it stands on.
>
> **Performance** — is it fast and affordable? Latency, token cost, and error
> monitoring catch drift in both the typical experience and the worst one.
>
> **Output** — is the agent giving good answers? Evals score response quality
> (helpfulness, non-answers, user corrections) so quality regressions surface
> as alerts, not user complaints.
>
> **Behavior** — is it working sensibly under the hood? Trajectory monitoring
> flags runs that loop or take paths a healthy run never takes — including
> failures the agent recovers from and hides.
>
> **Context** — is the data it relies on healthy? The agent's answers are only
> as good as its upstream tables; we monitor those for freshness, schema
> changes, and anomalies.
>
> Everything below maps to one of these four. Here's the plan:
### Monitor conventions — every monitor in this playbook
Four conventions apply to EVERY monitor created in this playbook — the agent
monitors (metric, evaluation, trajectory, validation) AND any warehouse
data-quality monitor created for the Context pillar (table or field monitors
on the agent's upstream tables; see `data-monitor-creation.md`):
- **Agent tag on every create.** Pass
`tags=[{"name": "agent", "value": "<AGENT_NAME>"}]`, where `<AGENT_NAME>` is
the agent's display name exactly as returned by `get_agent_metadata`
(trimmed, case preserved — do not slugify or rename). This single tag is the
footprint contract: one filter retrieves everything the playbook created for
this agent.
- **Audit / teardown contract.**
`get_monitors(monitor_tags=["agent:<AGENT_NAME>"])` (or the UI monitors tag
filter) returns the agent's full monitoring footprint — use it to audit
what exists, tune, or tear down everything for an agent. This is why a
create call without the tag is a defect: it silently drops the monitor out
of the footprint.
- **Audience — ask once, apply everywhere.** Before the FIRST create call of
the playbook (not per monitor), ask the user once which audiences should be
notified when these monitors fire — call `get_audiences` to list the
options; the user can pick one, several, or none. Pass the chosen audience
**names** (labels, never UUIDs) as `audiences` on EVERY monitor created in
the playbook, and set `failure_audiences` to the same selection unless the
user asks for a different failure-notification audience. Do not re-ask per
pillar or per monitor; if the user declines, omit `audiences`.
- **Domain — same for every monitor.** If the account uses domains, resolve
one domain for the agent's footprint and pass the same `domain_uuids` on
every monitor — including the Context-pillar DQ monitors (resolve via the
domain-assignment steps in `data-monitor-creation.md`) — so the whole
footprint lives in one domain.
### 2. Walk through the plan pillar by pillar
Present the pillars in order — Performance, Output, Behavior, Context — one
short block each:
1. **Evidence** — one or two sentences of what you observed in Step 2 that
motivates this pillar's monitors ("p95 latency is 40s with outliers over
three minutes", "several conversations show repeated user corrections").
If you found nothing notable for a pillar, say so and propose baseline
coverage anyway — monitoring exists to catch what hasn't happened yet.
2. **Proposed monitors** — the specific monitors for this pillar, each with
its monitor type, field or judge, and alert condition.
3. **Confirm** — ask whether to keep, adjust, or drop this pillar's monitors,
and fold the answer in before moving to the next pillar.
A healthy agent usually warrants coverage in every pillar you can serve —
keep proposals broad across pillars, not deep in one.
**Global defaults for proposed monitors** (apply unless the user asks
otherwise):
- **Daily schedule** — pass `interval_minutes=1440` explicitly; the tools'
built-in default is hourly (see Schedule configuration below).
- **Eval sampling** — `sampling_config={"count": 100}` (a fixed 100-sample
budget per run), not a percentage, so evaluation cost stays predictable as
traffic grows.
- **Audience on every create** — apply the playbook-level audience selection
(see "Monitor conventions — every monitor in this playbook" above); never
create a monitor without that once-asked selection applied (Step 4).
What each pillar maps to:
| Pillar | Monitor types | Reference |
|---|---|---|
| **Performance** | Metric (validation for hard limits) — latency (`duration_sec`), token cost (`total_tokens`), error rate (`status_code`), volume (`ROW_COUNT_CHANGE`) | `agent-metric-monitor.md` |
| **Output** | Evaluation — lead with the Output-pillar starting packs (see Step 3): baseline pack for every agent, analytics pack for Cortex/Genie. Add predefined judges (`answer_relevance`, `task_completion`, `clarity`, `prompt_adherence`), rule checks (`output_length`, `json_validity`), and one custom eval per recurring user intent or failure mode you observed | `agent-evaluation-monitor.md` |
| **Behavior** | Trajectory (validation for aggregate assertions) — runaway loops (`SPAN_OCCURRENCE`), missing or mis-ordered steps (`SPAN_RELATION`), token budgets | `agent-trajectory-monitor.md`, `agent-validation-monitor.md` |
| **Context** | Table monitors (freshness / schema changes / volume) on the agent's upstream tables, plus an optional `Context for {AGENT_NAME}` data product wrapper — see below | `data-table-monitor.md` |
Mind the backend caveats from Step 1 (`backend_class`): no token or model
metrics for Genie / Knowledge Assistant agents, and conversation-grain evals
only on OTel/ClickHouse, Snowflake Cortex, and Genie — with
`include_tool_calls: true` on every conversation-grain eval transform by default
(rejected at span grain; see Step 1). Aggregate (per-trace)
validation assertions require `is_agent_trace_aggregation=True`, supported
only on `ao_clickhouse_otel` / `customer_otel_trace_table` agents — on other
backends, use per-span assertions instead.
**Context — monitor the tables the agent reads.** Unlike the other pillars,
Context coverage lives on warehouse tables, not spans. Automatic
lineage-derived table discovery is not available on this surface, so ask the
user which upstream tables the agent depends on (the tables its SQL tools
query, its knowledge bases are built from, or its features are loaded from) —
never guess table names, and never drop the pillar silently. On the named
tables:
1. **Create the table monitors** with `create_or_update_table_monitor`,
following `data-table-monitor.md` for warehouse resolution and asset
selection (scope as narrowly as that reference allows). The tool's
default alert conditions are exactly the Context coverage — freshness,
schema changes, and volume — so omit `alert_conditions` unless the user
asks for more. Apply the Monitor conventions above on every create: the
`agent` tag, the playbook `audiences`, and `domain_uuids`. Dry-run
preview first, deploy on explicit confirmation, like every other create
in this playbook.
2. **Optionally wrap the tables in a data product** named
`Context for {AGENT_NAME}` via `create_or_update_data_product`: pass the
tables' `mcons` (from `search` / `get_asset` — do not guess them) and a
`description` naming the agent. Keep the default `dry_run=True` to show
the asset-footprint preview; set `dry_run=False` only on explicit
confirmation. The preview's asset count can exceed the tables you
named — the tool's backend automatically expands the footprint to
include their upstream dependencies. When the count is notably larger
than the named set, explain to the user what is being added before
asking them to confirm the live create. Two caveats: data products
take `audience_ids` — UUIDs from `get_audiences` — unlike monitors,
which take audience names; and the data product itself carries no
`agent` tag (the footprint contract rides on the monitors). If the
live create is rejected because the account lacks the Data Mesh
module, that is terminal for the wrapper: say so in one line (their
Monte Carlo representative can enable it) and keep the table monitors
— the data product is packaging; the monitors are the pillar.
3. **Add field-level depth** where the user wants specific field checks
(null rates, distributions, custom rules) on an upstream table: use the
data-monitor workflow's field-level references (`data-metric-monitor.md`,
`data-validation-monitor.md`, `data-custom-sql-monitor.md`), carrying
the same tag, audiences, and domain on every create.
If the user cannot name any upstream tables, present the pillar as a
recommendation — name what you would monitor and why — rather than failing
or silently dropping it.
### 3. Create the confirmed monitors
Once the user has confirmed the pillars, continue to Step 3 (pick each
monitor's reference doc) and Step 4 (dry-run preview, applying the
already-collected audience selection, creation on explicit confirmation) for
each approved monitor.
---
## The `agent` reference
All four `create_or_update_agent_*_monitor` tools author the monitor's source from
a single top-level **`agent`** argument (there is no `dw_id` and no `data_source`
argument — the `agent` reference is the whole source). Two accepted forms:
- **Platform agent reference** — `{database}:{schema}.{name}` (Snowflake Cortex /
Databricks agents), e.g. `analytics:agents.support_bot`.
- **OpenTelemetry `service_name`** — for OTel-instrumented agents, e.g. `checkout-agent`.
Get the exact value from `get_agent_metadata`'s **`agentReference`** field and pass
it verbatim — never construct, modify, or truncate it, and never pass an MCON. Two
optional companions:
- `trace_table` — only for non-ClickHouse OTel agents whose trace storage cannot be
inferred from the agent reference.
- The per-type reference tells you whether `warehouse` is required (see below).
---
## Warehouse
`warehouse` names the warehouse the agent's trace data lives in — pass it as a name
or UUID. Use the agent entry's `warehouse_uuid` from `get_agent_metadata` (and its
`warehouse_name` when talking to the user); when both are null, use `get_warehouses`.
Whether `warehouse` is required or optional depends on the monitor type — see the
per-type reference.
---
## Agent span filters
The optional `agent_span_filters` parameter refines which spans are monitored. It
accepts **at most one** filter object. Each field of the object holds a
`{"value": "..."}` sub-object.
| Filter field | Description | Example |
|-------------|-------------|---------|
| `agent` | Filter by agent name | `{"agent": {"value": "My Agent"}}` |
| `workflow` | Filter by workflow name | `{"workflow": {"value": "Chat Agent"}}` |
| `task` | Filter by task name | `{"task": {"value": "call_model"}}` |
| `span_name` | Filter by span name | `{"span_name": {"value": "ChatBedrockConverse.chat"}}` |
Multiple fields can be combined in the single filter object:
```json
[{"workflow": {"value": "Chat Agent"}, "task": {"value": "call_model"}}]
```
`agent_span_filters` is a refinement and is optional — the `agent` reference already
scopes the monitor. Some monitor types restrict which fields are allowed here (e.g.
trajectory monitors, and trace-aggregated metric / validation monitors) — see the
per-type reference for the exact rule.
---
## Schedule configuration
**Propose daily schedules by default** — pass `interval_minutes=1440` explicitly.
Schedule is set via two top-level args, not a nested object:
- `schedule_type` — defaults to `fixed`. Valid values: `fixed`, `manual`.
- `interval_minutes` — defaults to `60` (hourly), so omitting it creates an hourly
monitor. The floor and alignment differ per monitor type — see each reference.
No agent monitor accepts a dynamic schedule — use `fixed` or `manual`. Daily is
the right cadence for most agents. If you judge an agent critical enough that a
same-hour alert would matter, suggest hourly to the user and let them decide —
the default stays daily unless they opt in.
---
## Time filter configuration
Used by trajectory and validation monitors. The `time_field` is an object with
a `field` property — always use `ingest_ts`:
```json
{"time_field": {"field": "ingest_ts"}, "lookback_in_hrs": 24}
```
---
## Step 3: Choose the right monitor type
Based on your investigation, recommend one or more monitor types. Read the
corresponding reference doc for the detailed creation guide.
| I want to... | Monitor type | Reference file |
|-------------|-------------|----------------|
| Track a numeric metric trend (latency, tokens) | Agent Metric | `agent-metric-monitor.md` |
| Set up performance coverage (latency, token cost, errors, SLO) | Agent Metric — Performance pillar | `agent-metric-monitor.md` |
| Score output quality with LLM evaluation | Agent Evaluation | `agent-evaluation-monitor.md` |
| Alert on execution patterns or span sequences | Agent Trajectory | `agent-trajectory-monitor.md` |
| Assert a logical rule on span data | Agent Validation | `agent-validation-monitor.md` |
| Monitor span volume over time | Agent Metric | `agent-metric-monitor.md` |
| Detect answer relevance drops | Agent Evaluation | `agent-evaluation-monitor.md` |
| Catch runaway tool call loops | Agent Trajectory | `agent-trajectory-monitor.md` |
| Ensure token count stays below threshold | Agent Validation | `agent-validation-monitor.md` |
**Output-pillar starting packs** — for a newly onboarded agent (or one with no eval coverage
yet), lead with the named packs from `agent-evaluation-monitor.md` rather than inventing a
one-off list:
- **Baseline pack — every agent:** the predefined `helpfulness_conversation` judge (plain
`helpfulness` on span-grain-only backends) plus the `frustration_free_score` template.
Defaults: daily schedule (`interval_minutes=1440`), `{"count": 100}` sampling, an `agent`
tag (`{"name": "agent", "value": "<AGENT_NAME>"}`) on every monitor, and
`include_tool_calls: true` on every conversation-grain transform (see the `backend_class`
capabilities in Step 1).
- **Analytics pack — only when `backend_class` is `platform_agent` (Snowflake Cortex) or
`databricks_genie`:** the `answer_attempt_score` and `user_correction` templates — the
dominant NL2SQL/analytics failure modes are deflected answers and user-corrected answers.
Do not propose this pack for other agents.
Render each template with the agent's actual name and observed intents (never boilerplate) and
show the full prompt text for approval — see "Custom-prompt template library" and
"Output-pillar eval packs" in `agent-evaluation-monitor.md`.
After selecting the monitor type, **read the reference doc** for that type to
get the detailed parameter guide, examples, constraints, and creation workflow.
For a blanket "performance monitoring" ask, follow the **Performance pillar**
baseline set in `agent-metric-monitor.md` rather than assembling one-offs.
### Behavior monitors — two trajectory proposals for (almost) every agent
Grounded in the Step 2c summary, propose these two patterns whenever they apply
(`agent-trajectory-monitor.md` has the full playbooks and payload shapes):
1. **Runaway loop — create live.** SPAN_OCCURRENCE on the agent's dominant tool
span, threshold derived from the observed per-trace occurrence distribution:
max observed + headroom, never a stock number. The proposal's evidence must
show the dominant span, the distribution, and the derived threshold with its
headroom rationale — plus a pre-create breach `preview` (dry run) proving zero
historical matches; if the preview breaches, the sample missed the heavy tail
(e.g. multi-turn accumulation in one trace) — re-derive from a wider window.
Zero historical matches is the point — it is a regression guardrail that stays
silent until the agent's behavior regresses.
2. **Ungrounded-in-data — create as a DRAFT** (only for agents that answer
questions from data). Negated `occurs_with` SPAN_RELATION: an answer was
produced without the agent's data-access tool span. Show a breach `preview`
(dry run) as evidence, then create with `is_draft=True` — generic questions
legitimately skip the data tool, and the LLM-judge filter needed to separate
them from real data questions cannot be combined with a trajectory condition
yet.
Tag both with the agent's name (`tags=[{"name": "agent", "value": "<AGENT_NAME>"}]`)
and schedule them daily (`interval_minutes=1440`).
Beyond these two, propose **agent-tailored behavioral custom prompts** for
behaviors a span pattern can't see — e.g. "did the agent claim it ran a query it
never executed?", "did the agent re-ask for information the user already gave?".
One boolean `custom_prompt` per behavior, alerting on `TRUE_RATE` / `FALSE_RATE`,
at conversation grain where the backend supports it (see `backend_class` in
Step 1 and `agent-evaluation-monitor.md`) — keep `include_tool_calls: true` on
these so the judge can see the tool calls it is judging.
---
## Step 4: Create the monitor
All four tools follow the same **two-call preview-then-confirm pattern** as the data
monitor tools: the first call (`dry_run=True`, the default) returns the rendered MaC
YAML for review; the second call (`dry_run=False`) deploys the monitor live and
returns its UUID. Pass `monitor_uuid` on either call to update an existing agent
monitor in place instead of creating a new one (PUT semantics — re-pass every field
you want to keep, since omitted fields revert to defaults).
1. **Always start with `dry_run=True`** (the default). Show the user the
configuration preview (the rendered YAML).
2. **Apply the playbook-level audience selection** (see "Monitor conventions —
every monitor in this playbook"): the audience question was already asked
once before the playbook's first create — pass that same selection of
audience **names** (not UUIDs) as the `audiences` list, and default
`failure_audiences` to the same selection. Fall back to asking here (one
question, `get_audiences` for options) only when the playbook-level ask has
not happened (e.g. the user jumped straight to a single monitor outside the
walkthrough).
3. After showing the preview, offer to create or adjust settings.
4. Only set `dry_run=False` when the user explicitly confirms creation.
---
## Field name reference
See `agent-span-fields.md` for the complete list of known span field names
available in agent monitors. Do not guess field names — use only the ones
documented there.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |