monte-carlo-data/mc-agent-toolkit / skills/troubleshoot-agent-traces
monte-carlo-troubleshoot-agent-traces
VerifiedTroubleshoots Monte Carlo AI agent alerts and traces — eval score drops, latency/token spikes, trajectory and validation breaches. Not for data incidents (monte-carlo-analyze-root-cause) or monitor creation (monte-carlo-monitoring-advisor).
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 5.8 KB |
| SKILL.md | skill md | 17.4 KB |
| references/agent-alert-evaluation.md | markdown | 9.0 KB |
| references/agent-alert-metric.md | markdown | 6.1 KB |
| references/agent-alert-trajectory.md | markdown | 6.1 KB |
| references/agent-alert-validation.md | markdown | 6.0 KB |
| references/agent-backend-clickhouse.md | markdown | 6.6 KB |
| references/agent-backend-cortex.md | markdown | 6.0 KB |
| references/agent-backend-customer-otel.md | markdown | 5.2 KB |
| references/agent-backend-genie.md | markdown | 6.3 KB |
| references/agent-backend-mlflow-ka.md | markdown | 5.3 KB |
| references/agent-backend-mlflow-sdk.md | markdown | 5.0 KB |
| references/agent-direct-trace.md | markdown | 5.6 KB |
references/agent-backend-mlflow-ka.md
View on GitHub# Backend: Databricks Knowledge Assistant / Agent Bricks (`databricks_mlflow_ka`)
## What this backend is
A **no-code** Agent Bricks Knowledge Assistant — a RAG assistant over document sets,
configured entirely through the Databricks UI. Each trace is one Q&A turn: the root span
carries the user's question and the final answer, and retriever spans carry the document
chunks the answer was grounded on. Its behavior is driven by its configuration — the
assistant's instructions and its **knowledge sources** — so the investigation centers on
retrieval grounding and document quality, not on code or models.
**CRITICAL: model and token fields are empty by design here — a token-usage, cost, or
model-swap question has no answer on this backend. Never chase token anomalies.**
## Signal available here
- **Per-turn traces** — each trace is one turn: a root span with the question and final
answer, chain steps, and retriever tool-call spans. (`get_agent_trace` does not read
this backend — it errors; span-grain detail comes from `run_troubleshooting_agent`,
and manual reads work at turn grain via `get_agent_traces`.)
- **Retrieval grounding (the differentiator)** — the retriever spans' tool-call output
holds the retrieved document chunks. What was asked, what was retrieved, and whether
the answer was grounded in it is THE signal on this backend. Chunk content is
consent-gated; retrieval counts and structure are always readable.
- **Failure status per span** — failed turns carry a Knowledge-Assistant-specific error
marker; error rates and failed-turn trends are readable via `get_agent_traces`.
- **Volume and latency trends** — turns, retrievals, failed turns, duration, via
`get_agent_traces` aggregation.
- **A config surface** — instructions plus the knowledge sources (the document sets it
retrieves from), the platform analog of code history.
## Absent by design — do not chase
- **No model, no tokens** — always null. No cost or model-swap framing applies.
- **No agent source code and no PRs** — the assistant is configured, not coded.
- **Conversation grouping is usually absent** — `conversation_id` is frequently null;
each trace is then a standalone turn. Do not rely on conversation reads
(`get_agent_conversations` / `get_agent_conversation` may come back empty — that is
expected, not a data problem; work at turn grain with `get_agent_traces` instead).
- **Eval scores are not in the spans** — quality scores are Monte Carlo–computed and
live on the monitor/alert.
- **No SQL and no lineage** — a Knowledge Assistant generates no SQL, so there are no
source tables to extract from queries; retrieval grounding is the analogous
root-cause bridge.
## Investigation approach
1. **Confirm the backend** — `get_alert_agent_classification` / `get_agent_metadata`.
2. **Inspect the retrieval grounding FIRST (the primary root-cause move).** For the
failing or low-scored turns, look at what the retriever spans fetched. Missing,
stale, or off-topic chunks point at the knowledge source — a document set that
changed, went stale, or lost coverage — not at the assistant itself. This is the KA
analogue of tracing a wrong answer back to its data source.
3. **Compare BEFORE vs AFTER.** Incident window vs the prior baseline: turns, traces,
retrievals, failed turns, duration — **not tokens**. A step on a specific day points
at a knowledge-source or config change that day.
4. **Look at WHAT happened — and rule out a false positive.** Read the failing turns
(and, when consent allows, the actual question/answer pairs). Genuine problem (bad
grounding, real error spike) vs false positive (legitimately hard questions, an
expected spike, a too-tight threshold). A benign breach is a finding — say so and
recommend adjusting the monitor.
5. **Diff the config surface against the onset** — the instructions and the list of
knowledge sources. Check `get_alerts` for data incidents on the tables/documents
behind the knowledge sources.
6. For a full automated root-cause run, hand off to `run_troubleshooting_agent` and
collect results with `get_troubleshooting_agent_results`.
## Gotchas
- **Never produce PR, token, cost, or model findings.** Evidence must be about the
retrieval grounding, the failure/volume trend, or the assistant's configuration.
- **Consent gating is a fact to report, not a failure.** Question/answer text and
retrieved chunks may be blocked by the account's data-sampling settings — say so and
continue from retrieval counts and failure structure.
- **Don't lean on conversation ids** — treat each trace as a standalone turn unless the
data proves otherwise.
- **Schema questions are answered from the normalized vocabulary** — see the span-field
reference below; never run exploratory queries against the trace store to discover
fields.
- **Fix language:** adjust the assistant's instructions, refresh or fix a knowledge
source, or resolve the data incident behind it. Verification steps are grounding
checks ("inspect the retrieved chunks for the failing turns", "diff the knowledge
sources against the onset date") — never "review the PR" or model/token tuning.
## Cross-links
- Span-field vocabulary: `../../monitoring-advisor/references/agent-span-fields.md`
- Alert-type playbooks: `agent-alert-*.md`
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |