Skip to content

monte-carlo-data/mc-agent-toolkit / skills/tune-monitor

tune-monitor

Verified

Tune noisy Monte Carlo monitors, or find which ones to tune. Starts from Monte Carlo's tuning agent (stored suggestions or a fresh preview), applies only what the user approves. Metric, SQL, validation, table and agent monitors.

references/metric-monitor.md

View on GitHub
# Tuning Metric Monitors

This reference covers type-specific tuning guidance for metric monitors. Read this file after
determining the monitor type in Phase 1.5.

## Config fields to extract

Extract these from the `get_monitors` config response for your Phase 2 analysis:

- Monitor metric (e.g., `RELATIVE_ROW_COUNT`, `NULL_RATE`, `NUMERIC_MEAN`)
- Segment field(s) (`segment_fields`)
- WHERE condition (`where_condition`)
- Aggregation bucket (`aggregate_by`: `hour`, `day`, `week`, `month`)
- Aggregation time field (`aggregate_time_field`)
- Collection lag (`collection_lag_minutes`)

---

## Threshold adjustment (explicit thresholds)

For metric monitors using explicit thresholds (`GT`, `LT`, `GTE`, `LTE`, `EQ`, `NE`) instead of
`AUTO`:

- If anomalies are consistently marginal — the observed value just barely crosses the threshold —
  recommend loosening the threshold based on observed values from the report.
- **CRITICAL:** Always explain what the threshold value represents in the context of the metric
  before recommending a change. For example, "NULL_RATE GT 0.05 means alert when more than 5% of
  values are null."
- **NEVER** recommend a threshold change without citing observed anomaly values from the report.

---

## WHERE condition / segment exclusion

**When to recommend a WHERE condition:**
- One or more specific segment values fire repeatedly and are assessed as expected behavior
  (e.g., a sparse/bursty event type, a scheduled batch event)
- The noisy segments are identifiable from the incident history

**Syntax examples:**
```yaml
where_condition: "event_type NOT IN ('inactive_monitor', 'agent_evaluation_anom')"
```
```yaml
where_condition: "status != 'test'"
```

**IMPORTANT:** Always verify the column name and values exist in the table before recommending a
WHERE condition. Reference specific segment values from the monitor report.

**High-cardinality segments:**
- If the segment field has very high cardinality with many sparse values → recommend
  `"high_segment_count": true` or consider removing segmentation entirely
- **NEVER** recommend removing segmentation without explaining what signal would be lost

---

## Aggregation bucket changes

If the monitor aggregates by `hour` and anomalies are caused by sparse or bursty segments
(e.g., event types that fire only at certain hours), switching to `"aggregate_by": "day"` can
dramatically reduce false positives. The daily bucket smooths out intra-day spikes that are
normal over a 24-hour window.

**When to recommend:**
- Anomaly values are marginal at the hour level but would be within range at the daily level
- The segment naturally has low and variable hourly counts

**Trade-off:** You lose hourly granularity and may detect issues later. Always state this.

**CRITICAL:** Do not recommend changing `aggregate_by` without also checking whether the
`interval_minutes` needs to align. Hourly aggregation requires a schedule ≥60 min; daily
requires ≥1440 min.

---

## Monitor restructure

Recommend splitting into separate monitors when:
- Different segment values have fundamentally different expected behaviors (e.g., one segment
  is bursty by design, another should be steady)
- No single `where_condition` can cleanly separate noisy segments from signal-carrying ones

Recommend reviewing whether the metric and field combination is the right approach when:
- The monitor consistently fires on patterns that are inherent to the data shape
- A different metric would better capture the actual data quality concern

---

## Applying changes

Use `create_or_update_metric_monitor` to update the monitor in place.

1. **Always pass `monitor_uuid=<uuid>`** so the tool updates the existing monitor rather than
   creating a new one. Use the monitor UUID from Phase 1.
2. **Always dry-run first** (`dry_run=True`, the default) — show the user the YAML preview
   returned in `result.yaml` and ask for confirmation before applying.
3. **On confirmation**, call again with `dry_run=False` (and the same `monitor_uuid` plus the
   same other parameters). The response carries the monitor's UUID in `result.monitor_uuid` and
   a deep link in `result.instructions` — surface that to the user. `result.yaml` is `None` on
   the live call by design.
4. **Stale-uuid handling.** If the monitor was deleted between read and write, the tool raises a
   clear error instructing you to retry without `monitor_uuid` (turning the intent from "update"
   into "create"). Confirm with the user before recreating.

### Common mistakes

- **NEVER** omit `monitor_uuid` — this creates a duplicate monitor instead of updating.
- **NEVER** apply changes without showing the dry-run preview first.
- **CRITICAL: PUT semantics.** `create_or_update_metric_monitor` with `monitor_uuid` fully
  replaces the monitor's configuration — fields you omit revert to tool defaults, they are NOT
  left untouched. The full config from Phase 1's `get_monitors(monitor_ids=[<uuid>],
  include_fields=["config"])` call is your source of truth: re-pass every field you want to
  keep (schedule, audiences, segment_fields, where_condition, sensitivity, collection_lag_hours,
  notes, priority, tags, etc.) alongside the ones you're changing.
- **Diff the preview against the original.** Before running `dry_run=False`, compare the
  rendered YAML returned in `result.yaml` against the original config — if anything you meant to
  preserve is missing or changed, fix the call before committing.
deprecation-and-migrationaddyosmani/agent-skillsManages deprecation and migration. Use when removing old systems, APIs, or features. Use when migrating users from one implementation to another. Use when migrating a database schema in production, such as renaming or dropping a column without downtime (expand/contract). Use when deciding whether to maintain or sunset existing code.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
performance-optimizationaddyosmani/agent-skillsOptimizes application performance across frontend, backend, queries, and databases. Use when performance requirements exist, when you suspect performance regressions, when Core Web Vitals or load times need improvement, when N+1 query patterns need fixing, or when profiling reveals bottlenecks.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
agent-cost-reportthedotmack/claude-memBelievable agent cost report for any period, default the last 7 full days PT, not counting today. Measured tokens from Claude Code transcripts priced at OpenRouter list prices (ESTIMATED), measured provider spend when a sanctioned source exists, note-taker cost separate, Timing-style HTML/PDF plus report.json, line-items.csv, evidence.json.Stars: 118kUpdated Claude CodeCodexVerifiedInstall