monte-carlo-data/mc-agent-toolkit / skills/troubleshoot-agent-traces
monte-carlo-troubleshoot-agent-traces
VerifiedTroubleshoots Monte Carlo AI agent alerts and traces — eval score drops, latency/token spikes, trajectory and validation breaches. Not for data incidents (monte-carlo-analyze-root-cause) or monitor creation (monte-carlo-monitoring-advisor).
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 5.8 KB |
| SKILL.md | skill md | 17.4 KB |
| references/agent-alert-evaluation.md | markdown | 9.0 KB |
| references/agent-alert-metric.md | markdown | 6.1 KB |
| references/agent-alert-trajectory.md | markdown | 6.1 KB |
| references/agent-alert-validation.md | markdown | 6.0 KB |
| references/agent-backend-clickhouse.md | markdown | 6.6 KB |
| references/agent-backend-cortex.md | markdown | 6.0 KB |
| references/agent-backend-customer-otel.md | markdown | 5.2 KB |
| references/agent-backend-genie.md | markdown | 6.3 KB |
| references/agent-backend-mlflow-ka.md | markdown | 5.3 KB |
| references/agent-backend-mlflow-sdk.md | markdown | 5.0 KB |
| references/agent-direct-trace.md | markdown | 5.6 KB |
references/agent-backend-cortex.md
View on GitHub# Backend: Snowflake Cortex / Snowflake Intelligence (`platform_agent`)
## What this backend is
A **declarative platform agent** running inside the customer's Snowflake account —
Cortex Agents / Snowflake Intelligence. There is no agent source code: the agent is
defined by its configuration (instructions, tools, semantic models/views), and Snowflake
runs it. Monte Carlo reads its AI-observability events, normalized to the standard span
vocabulary.
**CRITICAL: there is no code repo and no PRs here — the fix surface is the agent's
configuration. A "which PR broke it" question has no answer on this backend; a "which
config change broke it" question usually does.**
## Signal available here
- **Real span structure with token counts** — token, duration, and status questions all
have answers; latency and error trends are readable through `get_agent_traces`.
- **A config surface with change timestamps** — the agent's instructions, tools, and
semantic views carry created/modified times. This is the platform analog of code
history and the primary change-correlation signal.
- **Conversations** — `get_agent_conversations` (rank by tokens/duration/errors to find
the expensive or failing ones) and `get_agent_conversation` for a full thread.
Transcript content is consent-gated (see Gotchas). **Conversation-grain evaluation
monitors** run on this backend, so a breached eval alert may name whole conversations.
- **Conversation clustering** — when the account has clustering enabled for this agent,
Monte Carlo groups its conversations into an intent-cluster taxonomy, shown alongside
the agent's conversations in the Monte Carlo UI. No toolkit tool reads clusters
directly: point the user at the cluster view to see which *kind* of conversations a
regression concentrates in, or hand off to `run_troubleshooting_agent`, which uses
cluster-share shifts as evidence.
- **Data lineage bridge** — the agent's generated SQL queries real source tables; a
freshness/volume/schema incident on a source table (check `get_alerts`) is a data
root cause only Monte Carlo can surface.
- **Segments** — `get_agent_segments` for the workflow/task values in play.
## Absent by design — do not chase
- **No source code, no GitHub, no PRs.** Do not look for repositories or recommend code
changes.
- **No OTel exception attributes** — error semantics come from span status and the
tool-span structure, not from exception type/message fields.
- **Latency and infrastructure are Snowflake-managed** and not user-fixable — focus on
answer quality and cost via the config surface, not on infra tuning.
- **Raw content is consent-gated** — prompts, completions, generated SQL, and
transcripts require the account's data-sampling consent. Structure, status, tokens,
and config are always available.
## Investigation approach
1. **Confirm the backend** — `get_alert_agent_classification` / `get_agent_metadata`.
2. **Config-space investigation FIRST.** Compare the agent's configuration modification
times (instructions, semantic views, tools) against the regression window. An
instruction edit, a semantic-model change, or a tool change landing right before the
anomaly is a top root cause on this backend.
3. **Compare BEFORE vs AFTER.** The incident window vs the immediately-prior baseline —
traces, tokens, duration, errors — plus the longer daily trend via `get_agent_traces`
aggregation. Did the breached metric *step* on a specific day (points at a config
change that day) or drift? Never describe only the bad window.
4. **Look at WHAT breached — and rule out a false positive.** Pull the flagged
conversations (`get_agent_conversations`, sampled from the breaching side of the score
per the metric + breach direction — see `agent-alert-evaluation.md` step 3) and read
them (`get_agent_conversation`). Decide: genuine problem (runaway tool loop,
bloated answer, error spike) vs false positive (a legitimately long-but-correct
conversation, an expected seasonal spike, a too-tight threshold). A breach the sample
shows to be benign **is a finding** — say so and recommend adjusting the monitor.
When clustering is enabled for the agent, localize first: a cluster whose share moved
in the breach window tells you which kind of conversations to sample (cluster view in
the UI, or the automated run's cluster evidence).
5. **Bridge to the data.** Identify the source tables the agent queried and check
`get_alerts` for incidents on them — a data incident upstream explains a quality drop
better than anything in the agent itself.
6. For a full automated root-cause run, hand off to `run_troubleshooting_agent` and
collect results with `get_troubleshooting_agent_results`.
## Gotchas
- **Consent gating is a fact to report, not a failure.** If conversation content comes
back blocked or empty because of the account's data-sampling settings, say that content
is unavailable on this account and continue from structure, tokens, and config — do
not retry or treat it as an error.
- **Prefer summaries and rollups over raw span pulls.** Wide raw reads get truncated on
this backend; aggregate views (`get_agent_traces` grouping, `get_agent_segments`,
conversation lists ordered by impact) are the reliable way to see the picture.
- **Eval scores are computed in-warehouse.** A quality-score movement correlates with a
config-surface change or a source-data shift — never with a Monte Carlo–side scoring
change.
- **Schema questions are answered from the normalized vocabulary.** Do not run
exploratory queries against the trace store to discover fields — the field list is in
the span-field reference below.
- **Fix language is config-surface only:** edit instructions, adjust semantic views,
add/remove tools, fix the upstream data incident. Never recommend code changes or
infrastructure tuning.
## Cross-links
- Span-field vocabulary: `../../monitoring-advisor/references/agent-span-fields.md`
- Alert-type playbooks: `agent-alert-*.md`
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |