monte-carlo-data/mc-agent-toolkit / skills/monitoring-advisor
monte-carlo-monitoring-advisor
VerifiedAnalyze data coverage, create monitors for warehouse tables and AI agents. Covers coverage gaps, use-case analysis, data monitor creation, and agent observability.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 3.9 KB |
| SKILL.md | skill md | 21.1 KB |
| references/agent-evaluation-monitor.md | markdown | 32.7 KB |
| references/agent-metric-monitor.md | markdown | 16.0 KB |
| references/agent-monitor-creation.md | markdown | 25.6 KB |
| references/agent-span-fields.md | markdown | 5.1 KB |
| references/agent-trajectory-monitor.md | markdown | 16.2 KB |
| references/agent-validation-monitor.md | markdown | 11.0 KB |
| references/data-comparison-monitor.md | markdown | 19.2 KB |
| references/data-custom-sql-monitor.md | markdown | 12.1 KB |
| references/data-metric-monitor.md | markdown | 17.5 KB |
| references/data-monitor-creation.md | markdown | 22.7 KB |
| references/data-table-monitor.md | markdown | 10.9 KB |
| references/data-validation-monitor.md | markdown | 19.3 KB |
references/data-validation-monitor.md
View on GitHub# Validation Monitor Reference
Detailed reference for building `create_or_update_validation_monitor` tool calls. The tool follows the **two-call preview-then-confirm pattern** — see `data-monitor-creation.md` for the full flow.
## Critical Constraints
- **NEVER guess column names.** Always get them from `get_asset`. Every field referenced in a validation condition must exist in the table schema exactly as spelled.
- **IMPORTANT: Conditions match INVALID data, not valid data.** The monitor alerts when it finds rows matching the condition, so the condition must describe the BAD rows. Getting this backwards is the number one mistake with validation monitors.
- **NEVER put a SELECT statement in a condition-level `SQL` node.** `{"type": "SQL", "sql": "..."}` as a top-level condition must be a boolean predicate expression (e.g. `amount < 0 OR amount > 1e9`), not a full query. Backend error: `Invalid SQL expression. Please provide a direct expression; it shouldn't begin with SELECT.`
- **NEVER use an aggregate or SQL expression in a `FIELD` value.** `{"type": "FIELD", "field": "COUNT(*)"}` is rejected as `Field "COUNT(*)" doesn't exist`. Fields are column names only — use `get_asset` to list valid ones. For counts/aggregates, fall back to a custom SQL monitor.
- **NEVER put a `SQL` value on the LEFT side of a BINARY condition.** Only `FIELD` references are allowed on the left. A `SQL` value is valid only on the right side (typically as a scalar subquery). Backend error: `Filter left side value must be a field or map key: FilterValueSql(...)`.
- **`alert_condition` is a dict (JSON object), NEVER a JSON-encoded string.** Pass the condition tree as a structured object — `{"type": "GROUP", "operator": "AND", "conditions": [...]}`. Serializing it to a string first is rejected with `Input should be a valid dictionary [type=dict_type, input_value='{"conditions":[...]'...]`.
---
## When to Use
Use a validation monitor when the user wants to:
- Check that specific fields are never null
- Validate that values are within an allowed set (e.g., status in 'active', 'pending', 'inactive')
- Enforce referential integrity (field values exist in another table)
- Apply row-level business rules (e.g., "amount must be positive")
- Combine multiple conditions with AND/OR logic
---
## Getting the Logic Right: Conditions Match INVALID Data
This is the single most confusing aspect of validation monitors and the number one source of mistakes. **Conditions describe what INVALID data looks like -- the data you want to be alerted about.** They do NOT describe what valid data looks like.
Think of it this way: the monitor scans rows and fires an alert when it finds rows matching the condition. So the condition must match the BAD rows.
| User wants | Condition should match | Common mistake |
|------------|----------------------|----------------|
| "id should never be null" | id IS NULL (alert when null found) | id IS NOT NULL (would alert on every valid row) |
| "status must be in [active, pending]" | status NOT IN [active, pending] (alert on unexpected values) | status IN [active, pending] (would alert on valid rows) |
| "amount must be positive" | amount IS NEGATIVE (alert on bad values) | amount > 0 (would alert on valid rows) |
| "email must not be empty" | email IS NULL **OR** email = '' (alert on missing) | email IS NOT NULL (would alert on valid rows) |
**Before building any condition, ask yourself: "If a row matches this condition, is the row INVALID?" If the answer is no, the logic is backwards.**
---
## Pre-Step: Verify Field Existence
Before constructing the `alert_condition`, verify that every field name you plan to reference exists in the table's column list. This is the number two source of validation monitor failures -- referencing columns that do not exist or are misspelled.
1. You should already have the column list from `get_asset` with `include: ["fields"]` (done in Step 2 of the main skill).
2. For every field name in your planned conditions, confirm it appears in the column list exactly as spelled (field names are case-sensitive on most warehouses).
3. If a field does not exist, stop and ask the user to clarify the correct column name. Do not guess.
---
## Required Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `name` | string | Unique identifier for the monitor. Use a descriptive slug (e.g., `orders_not_null_check`). |
| `description` | string | Human-readable description of what the monitor checks. |
| `table` | string | Table MCON (preferred) or `database:schema.table` format. If not MCON, also pass `warehouse`. |
| `alert_condition` | object | Condition tree defining when to alert (see Alert Condition Structure below). |
## Optional Parameters
| Parameter | Type | Description |
|-----------|------|-------------|
| `warehouse` | string | Warehouse name or UUID. Required if `table` is not an MCON. |
| `domain_uuids` | array of string (uuid) | Domain UUIDs (use `get_domains` to list). Data monitors accept exactly one UUID in the list. |
| `schedule_type` | string | Schedule type: `"fixed"` (default), `"dynamic"`, `"manual"`. |
| `interval_minutes` | int | Schedule interval in minutes (only for `schedule_type="fixed"`). |
| `audiences` | array of string | Notification audience **names** (not UUIDs) to alert when the monitor triggers. |
| `failure_audiences` | array of string | Notification audience names to alert on query execution failures. |
| `notes` | string | Free-text notes shown in the UI (separate from `description`). |
| `priority` | string | Monitor priority (e.g. `"P1"`, `"P2"`). |
| `tags` | array of `{name, value}` | Key-value tags to attach. |
| `is_draft` | bool | When `True`, saves the monitor as a draft (not active). Default `False`. |
| `monitor_uuid` | string (uuid) | UUID of an existing monitor to update in place. Omit to create a new monitor. **PUT semantics:** the call fully replaces the monitor's configuration — fields you omit revert to tool defaults, they are NOT left untouched. Before editing, read the current config with `get_monitors(monitor_ids=[<uuid>], include_fields=["config"])` and re-pass every field you want to keep. See `data-monitor-creation.md` (Step 7) for the safe-edit workflow. |
| `dry_run` | bool | Default `True`. Preview mode. When omitted or `True`, returns YAML preview in `result.yaml`. When `False`, actually creates/updates the monitor and returns `result.monitor_uuid` + a deep link in `result.instructions`. See `data-monitor-creation.md`. |
---
## Alert Condition Structure
The top level of `alert_condition` must always be a GROUP node. This GROUP contains one or more conditions combined with AND or OR logic.
```json
{
"type": "GROUP",
"operator": "AND",
"conditions": [...]
}
```
### Condition Types
There are four condition types: UNARY, BINARY, SQL, and GROUP.
#### UNARY (single-value checks)
Used for predicates that operate on a single field with no comparison value.
```json
{
"type": "UNARY",
"predicate": {"name": "null", "negated": false},
"value": [{"type": "FIELD", "field": "column_name"}]
}
```
- `predicate.name` -- the predicate to apply (see Predicates Reference below).
- `predicate.negated` -- set to `true` to invert the predicate (e.g., `null` with `negated: true` means "is NOT null").
- `value` -- an array with a single value descriptor (usually a FIELD reference).
#### BINARY (comparison checks)
Used for predicates that compare a field against a value.
```json
{
"type": "BINARY",
"predicate": {"name": "greater_than", "negated": false},
"left": [{"type": "FIELD", "field": "column_name"}],
"right": [{"type": "LITERAL", "literal": "0"}]
}
```
- `left` -- the left-hand side of the comparison (typically a FIELD reference).
- `right` -- the right-hand side (typically a LITERAL value, SQL expression, or FIELD reference).
- Both `left` and `right` are arrays of value descriptors.
#### SQL (custom SQL expression)
Used for complex conditions that are difficult to express with UNARY/BINARY nodes. The SQL expression should evaluate to true for INVALID rows.
```json
{
"type": "SQL",
"sql": "amount > 0 AND amount < 1000000"
}
```
#### GROUP (nested conditions)
Used to combine multiple conditions with AND or OR logic. Groups can be nested.
```json
{
"type": "GROUP",
"operator": "OR",
"conditions": [
{"type": "UNARY", "...": "..."},
{"type": "BINARY", "...": "..."}
]
}
```
---
## Value Types
Value descriptors appear in the `value`, `left`, and `right` arrays of UNARY and BINARY conditions.
| Type | Field | Description | Example |
|------|-------|-------------|---------|
| `FIELD` | `"field": "column_name"` | References a column in the table. Must be a plain column name — never an aggregate like `COUNT(*)` or a SQL snippet. | `{"type": "FIELD", "field": "user_id"}` |
| `LITERAL` | `"literal": "value"` | A static value (always a string, even for numbers). | `{"type": "LITERAL", "literal": "100"}` |
| `SQL` | `"sql": "..."` | A scalar SQL expression or subquery. **Right-side only** — cannot appear on the `left` of a BINARY. Valid forms: a scalar subquery (`SELECT MAX(id) FROM ref_table`) or a scalar expression. | `{"type": "SQL", "sql": "SELECT MAX(id) FROM ref_table"}` |
---
## Predicates Reference
Before building conditions, call `get_validation_predicates` to get the full list of supported predicates for the connected warehouse. The list below covers common predicates but may not be exhaustive.
### Unary Predicates
These predicates take no comparison value -- they check a property of the field itself.
| Predicate | Description | Example use |
|-----------|-------------|-------------|
| `null` | Field value is null. | Alert on null ids. |
| `is_negative` | Field value is negative. | Alert on negative amounts. |
| `is_between_0_and_1` | Field value is between 0 and 1 (inclusive). | Alert on rates that should be percentages (0-100). |
| `is_future_date` | Field value is a date/timestamp in the future. | Alert on future-dated records. |
| `is_uuid` | Field value matches UUID format. | Alert on non-UUID values in a UUID field (use with `negated: true`). |
### Binary Predicates
These predicates compare a field against a value.
| Predicate | Right-hand side | Description | Example use |
|-----------|----------------|-------------|-------------|
| `equal` | Single LITERAL | Field equals the given value. | Alert when `status` equals `'deleted'`. |
| `greater_than` | Single LITERAL | Field is greater than the given value. | Alert when `discount_pct` exceeds 100. |
| `less_than` | Single LITERAL | Field is less than the given value. | Alert when `quantity` is below 0. |
| `in_set` | Multiple LITERALs | Field value is in the given set. | Alert when `status` is in an invalid set (see example below). |
| `contains` | Single LITERAL | Field value contains the given substring. | Alert when `email` contains `'test@'`. |
| `starts_with` | Single LITERAL | Field value starts with the given prefix. | Alert when `phone` starts with `'000'`. |
| `between` | Two LITERALs | Field value is between the two given values (inclusive). | Alert when `score` is between 0 and 10 (if that range is invalid). |
### Using `negated` to Invert Predicates
Any predicate can be inverted by setting `"negated": true` in the predicate object. This is essential for "must be in set" validations:
- **"status must be in [active, pending]"** becomes `in_set` with values `["active", "pending"]` and `negated: true` -- meaning "alert when status is NOT in [active, pending]".
- **"id must not be null"** becomes `null` with `negated: false` -- meaning "alert when id IS null" (no inversion needed since the condition already matches invalid data).
### Semantic gotchas
Two constraints the backend enforces that don't show up in the predicate list:
- **Predicate/field-type compatibility.** Predicates have a target data type. `is_not_a_number` (NaN detection), `is_negative`, `is_between_0_and_1`, `numeric_*` — these only work on numeric columns; the backend rejects them on string/text fields with messages like `'not a number (NaN)' does not support fields of type 'string'`. Same pattern for `is_future_date` on non-timestamp columns. Use `get_validation_predicates` to confirm a predicate is supported for the target column type, and check the column's type from `get_asset` before picking one.
- **Field references on the RIGHT side of a BINARY.** The right-hand side is usually a `LITERAL` or `SQL` (subquery). You can put a `FIELD` reference on the right **only** when the left field is a date/timestamp column, OR when the operator is `in_set` (`in`). Other combinations are rejected with `Fields are only allowed on the right side for date and timestamp fields, or when using the 'in' operator`.
---
## Examples
### Alert when id is null
Verify that `id` exists in the table schema from `get_asset` before proceeding.
```json
{
"name": "orders_id_not_null",
"description": "Alert when order id is null",
"table": "MCON++a1b2c3d4-e5f6-7890-abcd-ef1234567890++1++1++analytics:core.orders",
"alert_condition": {
"type": "GROUP",
"operator": "AND",
"conditions": [
{
"type": "UNARY",
"predicate": {"name": "null", "negated": false},
"value": [{"type": "FIELD", "field": "id"}]
}
]
}
}
```
The condition matches rows where `id` IS NULL -- these are the invalid rows we want to be alerted about.
### Alert when status is not in allowed set
Verify that `status` exists in the table schema from `get_asset` before proceeding.
```json
{
"name": "orders_status_allowed_values",
"description": "Alert when order status is outside the allowed set",
"table": "MCON++a1b2c3d4-e5f6-7890-abcd-ef1234567890++1++1++analytics:core.orders",
"alert_condition": {
"type": "GROUP",
"operator": "AND",
"conditions": [
{
"type": "BINARY",
"predicate": {"name": "in_set", "negated": true},
"left": [{"type": "FIELD", "field": "status"}],
"right": [
{"type": "LITERAL", "literal": "active"},
{"type": "LITERAL", "literal": "pending"},
{"type": "LITERAL", "literal": "inactive"}
]
}
]
}
}
```
Note `negated: true` -- the predicate is `in_set`, but we want to alert when the value is NOT in the set. This catches any unexpected status values.
### Alert when amount is negative
Verify that `amount` exists in the table schema from `get_asset` before proceeding.
```json
{
"name": "orders_positive_amount",
"description": "Alert when order amount is negative",
"table": "MCON++a1b2c3d4-e5f6-7890-abcd-ef1234567890++1++1++analytics:core.orders",
"alert_condition": {
"type": "GROUP",
"operator": "AND",
"conditions": [
{
"type": "UNARY",
"predicate": {"name": "is_negative", "negated": false},
"value": [{"type": "FIELD", "field": "amount"}]
}
]
}
}
```
The condition matches rows where `amount` is negative -- these are the invalid rows.
### Combined conditions: null OR negative
Verify that both `amount` and `quantity` exist in the table schema from `get_asset` before proceeding.
```json
{
"name": "orders_amount_quality",
"description": "Alert when amount is null or quantity is negative",
"table": "MCON++a1b2c3d4-e5f6-7890-abcd-ef1234567890++1++1++analytics:core.orders",
"alert_condition": {
"type": "GROUP",
"operator": "OR",
"conditions": [
{
"type": "UNARY",
"predicate": {"name": "null", "negated": false},
"value": [{"type": "FIELD", "field": "amount"}]
},
{
"type": "UNARY",
"predicate": {"name": "is_negative", "negated": false},
"value": [{"type": "FIELD", "field": "quantity"}]
}
]
}
}
```
The OR operator means an alert fires if either condition matches -- the row has a null amount OR a negative quantity.
### Between check with nested AND/OR
Verify that `score` and `status` exist in the table schema from `get_asset` before proceeding.
```json
{
"name": "records_score_validation",
"description": "Alert when score is outside 0-100 range for active records",
"table": "MCON++a1b2c3d4-e5f6-7890-abcd-ef1234567890++1++1++warehouse:metrics.records",
"alert_condition": {
"type": "GROUP",
"operator": "AND",
"conditions": [
{
"type": "BINARY",
"predicate": {"name": "equal", "negated": false},
"left": [{"type": "FIELD", "field": "status"}],
"right": [{"type": "LITERAL", "literal": "active"}]
},
{
"type": "BINARY",
"predicate": {"name": "between", "negated": true},
"left": [{"type": "FIELD", "field": "score"}],
"right": [
{"type": "LITERAL", "literal": "0"},
{"type": "LITERAL", "literal": "100"}
]
}
]
}
}
```
This uses `between` with `negated: true` to alert when score is outside the 0-100 range, but only for active records (the AND operator requires both conditions to match).
### Referential integrity with SQL subquery
Verify that `customer_id` exists in the table schema from `get_asset` before proceeding.
```json
{
"name": "orders_valid_customer",
"description": "Alert when customer_id does not exist in customers table",
"table": "MCON++a1b2c3d4-e5f6-7890-abcd-ef1234567890++1++1++analytics:core.orders",
"alert_condition": {
"type": "GROUP",
"operator": "AND",
"conditions": [
{
"type": "SQL",
"sql": "customer_id IS NOT NULL AND customer_id NOT IN (SELECT id FROM analytics.core.customers)"
}
]
}
}
```
The SQL condition type is useful for referential integrity checks that require subqueries. The `customer_id IS NOT NULL` guard avoids alerting on null values (which should be caught by a separate null check if needed).
### Contains and starts_with checks
Verify that `email` and `phone` exist in the table schema from `get_asset` before proceeding.
```json
{
"name": "contacts_format_validation",
"description": "Alert when email contains test data or phone has invalid prefix",
"table": "MCON++a1b2c3d4-e5f6-7890-abcd-ef1234567890++1++1++warehouse:crm.contacts",
"alert_condition": {
"type": "GROUP",
"operator": "OR",
"conditions": [
{
"type": "BINARY",
"predicate": {"name": "contains", "negated": false},
"left": [{"type": "FIELD", "field": "email"}],
"right": [{"type": "LITERAL", "literal": "@test.example.com"}]
},
{
"type": "BINARY",
"predicate": {"name": "starts_with", "negated": false},
"left": [{"type": "FIELD", "field": "phone"}],
"right": [{"type": "LITERAL", "literal": "000"}]
}
]
}
}
```
---
## Fallback: Custom SQL Monitor
If `create_or_update_validation_monitor` fails -- for example because a referenced column does not exist yet in the live table, or the warehouse does not support a particular predicate -- fall back to `create_or_update_sql_monitor` with an explicit SQL query instead.
A custom SQL monitor lets you express any validation logic as a SQL query that returns rows or a count. This is always available as a backup when the structured validation condition tree cannot express what you need or encounters an API error.
When falling back:
1. Translate the intended validation logic into a SQL query.
2. The SQL should select rows that violate the rule (matching the same "conditions match INVALID data" principle).
3. Use `create_or_update_sql_monitor` with the translated query.
4. Inform the user that you used a custom SQL monitor as a fallback and explain why.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |