monte-carlo-data/mc-agent-toolkit / skills/monitoring-advisor
monte-carlo-monitoring-advisor
VerifiedAnalyze data coverage, create monitors for warehouse tables and AI agents. Covers coverage gaps, use-case analysis, data monitor creation, and agent observability.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 3.9 KB |
| SKILL.md | skill md | 21.1 KB |
| references/agent-evaluation-monitor.md | markdown | 32.7 KB |
| references/agent-metric-monitor.md | markdown | 16.0 KB |
| references/agent-monitor-creation.md | markdown | 25.6 KB |
| references/agent-span-fields.md | markdown | 5.1 KB |
| references/agent-trajectory-monitor.md | markdown | 16.2 KB |
| references/agent-validation-monitor.md | markdown | 11.0 KB |
| references/data-comparison-monitor.md | markdown | 19.2 KB |
| references/data-custom-sql-monitor.md | markdown | 12.1 KB |
| references/data-metric-monitor.md | markdown | 17.5 KB |
| references/data-monitor-creation.md | markdown | 22.7 KB |
| references/data-table-monitor.md | markdown | 10.9 KB |
| references/data-validation-monitor.md | markdown | 19.3 KB |
references/agent-validation-monitor.md
View on GitHub# Agent Validation Monitor
## When to use
Assert a logical condition on agent span data and alert on violations. Best for:
- **Business rule assertions** — "total_tokens must be below 10000"
- **Data quality checks on span attributes** — "completions must never be null"
- **Compliance checks** — "PII detection must run on every trace"
Do NOT use it to trend a numeric metric (use `create_or_update_agent_metric_monitor`)
or to alert on span sequences / call counts (use
`create_or_update_agent_trajectory_monitor`).
## Constraints
> **CRITICAL:** The monitor's source is the `agent` reference. Pass the
> `agentReference` value from `get_agent_metadata` verbatim — a platform
> `{database}:{schema}.{name}` reference or an OTel `service_name`. Never modify,
> truncate, or reconstruct it, and never pass an MCON.
> **CRITICAL:** `warehouse` is REQUIRED. Pass the agent's `warehouse_uuid` from
> `get_agent_metadata`; use `get_warehouses` when it is null or to resolve by name.
> **CRITICAL:** `alert_condition` matches the rows to ALERT on. A `null` predicate
> alerts on spans where the field IS null. Express negation with the `negated` flag —
> there is NO `not_equal` and NO `not_null` predicate.
> **IMPORTANT:** BINARY conditions use `left`/`right`; UNARY conditions use `value`
> (NOT `left`). Getting this wrong is the most common failure.
> **IMPORTANT:** `time_filter` is REQUIRED and `time_field` is always
> `{"field": "ingest_ts"}`. `time_filter` is `{"time_field": {"field": "ingest_ts"}, "lookback_in_hrs": <hours>}`.
> **IMPORTANT:** `schedule_type` is `fixed` (default) or `manual` — never dynamic.
> `interval_minutes` defaults to `60` and must be at least 5 (sub-hourly is allowed;
> no 60-minute alignment).
## Key characteristics
- Uses `alert_condition` as a `FilterGroup` — an `operator` (`AND`/`OR`) plus a
`conditions` array of `BINARY` / `UNARY` / `SQL` / `GROUP` entries
- Requires `time_filter` (time field is always `ingest_ts`)
- Optional `is_agent_trace_aggregation` for trace-level assertions (OTel agents only)
## Parameters
| Parameter | Type | Required | Description |
|-----------|------|----------|-------------|
| `description` | string | Yes | Human-readable monitor description (shown as display name) |
| `agent` | string | Yes | Agent reference — `agentReference` from `get_agent_metadata` (`{db}:{schema}.{name}` or OTel `service_name`) |
| `alert_condition` | object | Yes | FilterGroup — the condition that marks INVALID rows (see below) |
| `time_filter` | object | Yes | `{"time_field": {"field": "ingest_ts"}, "lookback_in_hrs": 24}` |
| `warehouse` | string | Yes | Warehouse name or UUID where the agent's traces live |
| `trace_table` | string | No | Explicit trace table — only for non-ClickHouse OTel agents |
| `agent_span_filters` | array | No | Optional span-scope refinement; at most ONE filter object |
| `is_agent_trace_aggregation` | boolean | No | Aggregate per trace for trace-level assertions (OTel only) |
| `schedule_type` | string | No | `fixed` (default) or `manual` |
| `interval_minutes` | int | No | Default `60`; at least 5 |
| `tags` | array | No | Key-value tags, e.g. `[{"name": "agent", "value": "Support Bot"}]`. Tag every agent monitor with its agent's name — `{"name": "agent", "value": "<AGENT_NAME>"}` — so all of one agent's monitors are filterable as a group |
| `domain_uuids` | array | No | Domain UUIDs to assign this monitor to — the agent-onboarding playbook passes the footprint's single resolved domain on every create (see agent-monitor-creation.md conventions) |
| `monitor_uuid` | string | No | UUID of an existing monitor to update in place (PUT semantics) |
| `dry_run` | boolean | No | Default `True` — preview YAML; set `False` to deploy |
## alert_condition structure (FilterGroup)
The top level is a group: an `operator` (`AND`/`OR`) and a `conditions` array. Each
condition is one of `BINARY`, `UNARY`, `SQL`, or `GROUP`.
### BINARY (compare two values)
```json
{
"type": "BINARY",
"predicate": {"name": "greater_than"},
"left": [{"type": "FIELD", "field": "total_tokens"}],
"right": [{"type": "LITERAL", "literal": "10000"}]
}
```
- `left`: exactly one `FIELD` value (the column being validated).
- `right`: exactly one value — usually a `LITERAL` (a string, even for numbers). The
`in_set` predicate is the exception: it takes several `LITERAL`s in `right`.
### UNARY (single-value check)
```json
{
"type": "UNARY",
"predicate": {"name": "null"},
"value": [{"type": "FIELD", "field": "completions"}]
}
```
- The field list is named **`value`** (NOT `left`), and holds exactly one `FIELD`.
- The example above matches spans where `completions` **is null**. To alert on
non-null instead, add `"negated": true` (→ IS NOT NULL).
### SQL (custom boolean expression)
```json
{"type": "SQL", "sql": "total_tokens > 1000 AND duration_sec < 60"}
```
Use only when the condition can't be expressed with a predicate.
### GROUP (nested conditions)
```json
{
"type": "GROUP",
"operator": "OR",
"conditions": [
{"type": "BINARY", "...": "..."},
{"type": "UNARY", "...": "..."}
]
}
```
### Predicates
Predicate names are matched by exact name — call `get_validation_predicates` to list
the full set. Key rules:
- **Express negation with the `negated` flag** — e.g. `{"name": "equal", "negated": true}`
or `{"name": "null", "negated": true}`. Do NOT prefix names with `not_`. There is
**no `not_equal` and no `not_null` predicate**.
- **BINARY predicates** include `equal`, `in_set`, `greater_than`,
`greater_than_or_equal`, `less_than`, `less_than_or_equal`, `contains`,
`starts_with`, `ends_with`, `matches_regex`. The four comparators (`greater_than` /
`less_than` / `*_or_equal`) cannot be negated — use the inverse comparator instead.
- **UNARY predicates** include `null`, `empty_string`, `is_zero`, `is_negative`,
`is_nan`, `is_between_0_and_1`, `is_between_0_and_100`, `is_uuid`, plus many locale /
PII / timestamp checks. Fetch the full list with `get_validation_predicates`.
### Value types
| Type | Format |
|------|--------|
| `FIELD` | `{"type": "FIELD", "field": "column_name"}` — references a span field |
| `LITERAL` | `{"type": "LITERAL", "literal": "value_string"}` — a static value, always a string even for numbers |
| `SQL` | `{"type": "SQL", "sql": "..."}` — a SQL expression (used inside a BINARY `right`) |
Condition/value/operator keywords are uppercase: `BINARY`, `UNARY`, `SQL`, `GROUP`;
`FIELD`, `LITERAL`; `AND`, `OR`.
## Field notes
Use the span field names from `agent-span-fields.md` in `FIELD` values — not raw
table columns. Note in particular:
- `status_code` is **numeric** — the OTel span status code (`2` = error). Compare
with a numeric literal (e.g. `equal` / `"2"`), not `"ERROR"`.
- The time field is always `ingest_ts`.
## Trace aggregation
`is_agent_trace_aggregation=True` is **OpenTelemetry-only.** A platform agent reference
(`{database}:{schema}.{name}`) is rejected — target an OTel `service_name`, or pass an
explicit `trace_table` to force the agent to be read as OpenTelemetry. At trace grain,
use trace-level fields (`span_count`, `llm_call_count`, `total_tokens`, …) and filter
only by `agent`.
## Examples
The `agent` value below comes from `get_agent_metadata`'s `agentReference` field —
a platform `{database}:{schema}.{name}` reference or an OTel `service_name`.
### Assert total_tokens stays below a threshold (platform agent reference)
```
create_or_update_agent_validation_monitor(
description="Alert when total_tokens exceeds 10000",
agent="analytics:agents.support_bot",
warehouse="Analytics WH",
tags=[{"name": "agent", "value": "Support Bot"}],
alert_condition={
"operator": "AND",
"conditions": [
{
"type": "BINARY",
"predicate": {"name": "greater_than"},
"left": [{"type": "FIELD", "field": "total_tokens"}],
"right": [{"type": "LITERAL", "literal": "10000"}]
}
]
},
time_filter={"time_field": {"field": "ingest_ts"}, "lookback_in_hrs": 24},
dry_run=True
)
```
### Alert when a required field is null (UNARY uses `value`)
Business rule "completions must always be populated" → alert on the violating rows,
i.e. spans where `completions` **is null**, so the condition is a plain `null`
predicate.
```
create_or_update_agent_validation_monitor(
description="Alert when the completions field is null",
agent="analytics:agents.support_bot",
warehouse="Analytics WH",
alert_condition={
"operator": "AND",
"conditions": [
{
"type": "UNARY",
"predicate": {"name": "null"},
"value": [{"type": "FIELD", "field": "completions"}]
}
]
},
time_filter={"time_field": {"field": "ingest_ts"}, "lookback_in_hrs": 24},
dry_run=True
)
```
### Span-level assertion scoped to a workflow
```
create_or_update_agent_validation_monitor(
description="Alert when Chat Agent spans exceed 120s",
agent="analytics:agents.support_bot",
warehouse="Analytics WH",
alert_condition={
"operator": "AND",
"conditions": [
{
"type": "BINARY",
"predicate": {"name": "greater_than"},
"left": [{"type": "FIELD", "field": "duration_sec"}],
"right": [{"type": "LITERAL", "literal": "120"}]
}
]
},
time_filter={"time_field": {"field": "ingest_ts"}, "lookback_in_hrs": 24},
agent_span_filters=[{"workflow": {"value": "Chat Agent"}}],
dry_run=True
)
```
### Trace-level assertion (OTel agent)
```
create_or_update_agent_validation_monitor(
description="Alert when a trace has more than 50 spans",
agent="checkout-agent",
warehouse="OTel WH",
alert_condition={
"operator": "AND",
"conditions": [
{
"type": "BINARY",
"predicate": {"name": "greater_than"},
"left": [{"type": "FIELD", "field": "span_count"}],
"right": [{"type": "LITERAL", "literal": "50"}]
}
]
},
time_filter={"time_field": {"field": "ingest_ts"}, "lookback_in_hrs": 24},
is_agent_trace_aggregation=True,
dry_run=True
)
```
## Common errors
| Error message | Cause | Fix |
|--------------|-------|-----|
| Warehouse not found | `warehouse` omitted or wrong | Pass the agent's `warehouse_uuid` from `get_agent_metadata`; if null, list warehouses via `get_warehouses` |
| invalid / unresolvable `agent` reference | The `agent` value wasn't taken from `get_agent_metadata` | Use the exact `agentReference` value — do not construct it by hand, and never pass an MCON |
| unknown predicate `not_equal` / `not_null` | Used a `not_`-prefixed predicate | Use the base predicate (`equal` / `null`) with `"negated": true` |
| UNARY condition rejected | Used `left` instead of `value` | UNARY conditions put the field in `value`; only BINARY uses `left`/`right` |
| "Field X doesn't exist" | Field name not in the PARSED_SPANS schema | Check `agent-span-fields.md`; `status_code` is numeric (compare with `"2"`, not `"ERROR"`) |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |