datarobot-oss/datarobot-agent-skills / skills/datarobot-external-agent-monitoring
datarobot-external-agent-monitoring
VerifiedInstrument any external or existing AI agent with OpenTelemetry to send traces, logs, and metrics to DataRobot for monitoring, observability, and governance. Use when the user says "add tracing/observability/monitoring to my agent", wants to instrument an existing agent project in their IDE, wants to send agent traces, logs, or metrics to DataRobot, or wants to record and replay agent trajectories (Tensile) from DataRobot traces.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 19.1 KB |
| frameworks/crewai.md | markdown | 2.9 KB |
| frameworks/generic-python.md | markdown | 6.7 KB |
| frameworks/google-adk.md | markdown | 14.7 KB |
| frameworks/langchain-langgraph.md | markdown | 5.0 KB |
| frameworks/llamaindex.md | markdown | 3.3 KB |
| frameworks/pydantic-ai.md | markdown | 8.0 KB |
| reference/dr_otel_config.md | markdown | 8.5 KB |
| scripts/create_shell_deployment.pyexecutable | script | 6.3 KB |
| scripts/create_use_case.pyexecutable | script | 4.0 KB |
| scripts/verify_otel_connection.pyexecutable | script | 5.9 KB |
frameworks/pydantic-ai.md
View on GitHub# PydanticAI — DataRobot OTel Integration
## Overview
PydanticAI's OpenTelemetry instrumentation is **opt-in**: call `Agent.instrument_all()`
at startup to emit spans. `configure_otel()` sets up the provider, and PydanticAI reuses
the global `TracerProvider` it installs, so the generic `dr_otel_config.py` works
unchanged. Skip the `Agent.instrument_all()` call and PydanticAI emits **no spans** —
silently, with no error, so telemetry never appears in DataRobot.
PydanticAI also has built-in integration with Pydantic Logfire, which exports
OTel-compatible telemetry. The preferred approach for DataRobot is direct OTel setup
(no Logfire dependency needed) plus the required `instrument_all()` opt-in.
## OTel Strategy
| Signal | Strategy |
|-----------|--------------------------------------------------------------------|
| **Traces** | Standard `configure_otel()` **+ required `Agent.instrument_all()`** (opt-in) |
| **Metrics** | Standard setup (optional custom metrics) |
| **Logs** | Standard setup |
## Required Setup
### 1. Use the generic `dr_otel_config.py` as-is
Call `configure_otel()` at startup, before any PydanticAI import or agent creation.
### 2. Opt in to instrumentation and wire the entrypoint
Call `Agent.instrument_all()` after `configure_otel()`. This is the step that makes
PydanticAI emit spans; skipping it is the most common cause of "no telemetry."
```python
import os
from dr_otel_config import configure_otel
configure_otel() # sets the global TracerProvider (additive) BEFORE PydanticAI opt-in
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider
# Opt in to PydanticAI's OTel instrumentation — without this, no spans are emitted.
Agent.instrument_all()
# Example: routing through the DataRobot LLM Gateway (any provider works).
provider = OpenAIProvider(
base_url=f"{os.environ['DATAROBOT_ENDPOINT'].rstrip('/')}/genai/llmgw",
api_key=os.environ["DATAROBOT_API_TOKEN"],
)
model = OpenAIChatModel("azure/gpt-5-mini-2025-08-07", provider=provider)
agent = Agent(model, system_prompt="You are a helpful assistant.")
```
`Agent.instrument_all()` uses the global `TracerProvider` that `configure_otel()` set,
so every agent in the process exports to DataRobot without editing each `Agent(...)`
constructor. PydanticAI creates spans for agent runs, LLM API calls, tool/function
calls, and retries/error handling.
## Version robustness
PydanticAI's instrumentation API has changed across versions. Prefer the global call;
fall back to per-agent only if you cannot call `instrument_all()` at startup.
- **Recommended — global, version-robust:**
```python
Agent.instrument_all()
```
- **Per-agent alternative — PydanticAI v2.x:**
```python
from pydantic_ai import Agent
from pydantic_ai.capabilities import Instrumentation
from pydantic_ai.models.instrumented import InstrumentationSettings
agent = Agent(
model,
capabilities=[Instrumentation(settings=InstrumentationSettings())],
)
```
- **Per-agent alternative — older PydanticAI v1.x:**
```python
agent = Agent(model, instrument=True)
```
- **`InstrumentationSettings(version=…)`** controls the telemetry data format
(`Literal[2, 3, 4, 5]`, default `5`, following OTel GenAI semantic conventions
1.37.0). Versions 2–4 are deprecated and emit `PydanticAIDeprecationWarning`. Only
pin a version if you must match a specific downstream schema (see Verify below).
- **Privacy — exclude prompt/completion content:**
```python
from pydantic_ai import Agent
from pydantic_ai.models.instrumented import InstrumentationSettings
Agent.instrument_all(InstrumentationSettings(include_content=False))
```
By default PydanticAI captures prompt and completion text in spans. Set
`include_content=False` to suppress it (this also empties DataRobot's
Prompt/Completion columns).
## Verify
After wiring, confirm telemetry lands in DataRobot: run the agent once, then check the
Use Case Tracing view (or the `dr xp` panel — see SKILL.md Step 5).
- **Spans present, Prompt/Completion columns populated** → done.
- **No spans at all** → `Agent.instrument_all()` was not called, or `configure_otel()`
ran after agent creation. Fix ordering.
- **Spans present but Prompt/Completion columns empty** → attribute-name mismatch.
PydanticAI's default `version=5` emits `gen_ai.input.messages` /
`gen_ai.output.messages` (OTel GenAI semconv), while DataRobot's tracing table reads
`gen_ai.prompt` / `gen_ai.completion`. To fix, add the `gen_ai.prompt` / `gen_ai.completion` span attributes yourself — note that **no** `InstrumentationSettings(version=…)` value produces them (every selectable format, 2–5, emits `gen_ai.input.messages` / `gen_ai.output.messages`), so pinning a version does not help. Also confirm `include_content` is not `False`, and check whether your DataRobot instance already normalizes the newer semconv attributes.
## Optional: Logfire instrumentation
If the user already uses Logfire or wants richer PydanticAI-specific spans:
```python
import logfire
logfire.configure(
send_to_logfire=False
) # don't send to Logfire cloud; keep global provider
logfire.instrument_pydantic_ai()
```
This layers detailed Pydantic validation spans on top of the standard OTel traces. Use
either direct OTel (`instrument_all()`) or Logfire, not both, to avoid duplicate traces.
## Extra Dependencies
Recommended, for intelligent trace analysis (see **Intelligent Trace Analysis via Tensile** below):
```
tensile>=0.12.0
```
None beyond the generic OTel packages.
Optional (if using Logfire):
```
logfire[pydantic-ai]
```
## Custom Metrics (Optional)
For custom metrics, wrap the agent run:
```python
import time
from opentelemetry import metrics
meter = metrics.get_meter("my-pydantic-agent")
request_counter = meter.create_counter("agent.requests", unit="1")
request_duration = meter.create_histogram("agent.request.duration_ms", unit="ms")
async def run_with_metrics(agent, prompt):
start = time.time()
try:
result = await agent.run(prompt)
request_counter.add(1, {"status": "success"})
return result
except Exception:
request_counter.add(1, {"status": "error"})
raise
finally:
request_duration.record((time.time() - start) * 1000)
```
## Intelligent Trace Analysis via Tensile
DataRobot provides intelligent trace analysis and remediation. This is made possible by the `tensile` package, which records agentic trajectories and allows them to be replayed. To enable this feature, the agent needs `tensile>=0.12.0` installed as a dependency and the OTel capture hook wired in directly after `configure_otel()`:
```python
from dr_otel_config import configure_otel
configure_otel()
import tensile.otel
tensile.otel.install(tracer=False)
from pydantic_ai import Agent # PydanticAI imports and Agent.instrument_all() follow, as above
```
## Known Pitfalls
| Issue | Cause | Fix |
|-------|-------|-----|
| No spans from PydanticAI | `Agent.instrument_all()` never called (instrumentation is opt-in) | Call `Agent.instrument_all()` after `configure_otel()`, before running agents |
| No spans from PydanticAI | `configure_otel()` called after agent creation | Ensure `configure_otel()` runs before any PydanticAI import or agent instantiation |
| Spans appear but Prompt/Completion columns empty | v5 semconv attributes (`gen_ai.input.messages`) differ from DataRobot's `gen_ai.prompt`/`gen_ai.completion`; or `include_content=False` | Add the `gen_ai.prompt`/`gen_ai.completion` span attributes yourself (no `version` value emits them); ensure `include_content` is not `False` (see Verify) |
| Logfire overrides TracerProvider | `logfire.configure()` called with default settings | Use `send_to_logfire=False` to keep the global provider |
| Duplicate traces | Both direct OTel and Logfire active | Choose one approach — prefer direct OTel |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |