Skip to content

datarobot-oss/datarobot-agent-skills / skills/datarobot-external-agent-monitoring

datarobot-external-agent-monitoring

Verified

Instrument any external or existing AI agent with OpenTelemetry to send traces, logs, and metrics to DataRobot for monitoring, observability, and governance. Use when the user says "add tracing/observability/monitoring to my agent", wants to instrument an existing agent project in their IDE, wants to send agent traces, logs, or metrics to DataRobot, or wants to record and replay agent trajectories (Tensile) from DataRobot traces.

FileKindSize
SKILL.mdskill md19.1 KB
frameworks/crewai.mdmarkdown2.9 KB
frameworks/generic-python.mdmarkdown6.7 KB
frameworks/google-adk.mdmarkdown14.7 KB
frameworks/langchain-langgraph.mdmarkdown5.0 KB
frameworks/llamaindex.mdmarkdown3.3 KB
frameworks/pydantic-ai.mdmarkdown8.0 KB
reference/dr_otel_config.mdmarkdown8.5 KB
scripts/create_shell_deployment.pyexecutablescript6.3 KB
scripts/create_use_case.pyexecutablescript4.0 KB
scripts/verify_otel_connection.pyexecutablescript5.9 KB

frameworks/pydantic-ai.md

View on GitHub
# PydanticAI — DataRobot OTel Integration

## Overview

PydanticAI's OpenTelemetry instrumentation is **opt-in**: call `Agent.instrument_all()`
at startup to emit spans. `configure_otel()` sets up the provider, and PydanticAI reuses
the global `TracerProvider` it installs, so the generic `dr_otel_config.py` works
unchanged. Skip the `Agent.instrument_all()` call and PydanticAI emits **no spans** —
silently, with no error, so telemetry never appears in DataRobot.

PydanticAI also has built-in integration with Pydantic Logfire, which exports
OTel-compatible telemetry. The preferred approach for DataRobot is direct OTel setup
(no Logfire dependency needed) plus the required `instrument_all()` opt-in.

## OTel Strategy

| Signal    | Strategy                                                            |
|-----------|--------------------------------------------------------------------|
| **Traces**  | Standard `configure_otel()` **+ required `Agent.instrument_all()`** (opt-in) |
| **Metrics** | Standard setup (optional custom metrics)                           |
| **Logs**    | Standard setup                                                     |

## Required Setup

### 1. Use the generic `dr_otel_config.py` as-is

Call `configure_otel()` at startup, before any PydanticAI import or agent creation.

### 2. Opt in to instrumentation and wire the entrypoint

Call `Agent.instrument_all()` after `configure_otel()`. This is the step that makes
PydanticAI emit spans; skipping it is the most common cause of "no telemetry."

```python
import os

from dr_otel_config import configure_otel

configure_otel()  # sets the global TracerProvider (additive) BEFORE PydanticAI opt-in

from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

# Opt in to PydanticAI's OTel instrumentation — without this, no spans are emitted.
Agent.instrument_all()

# Example: routing through the DataRobot LLM Gateway (any provider works).
provider = OpenAIProvider(
    base_url=f"{os.environ['DATAROBOT_ENDPOINT'].rstrip('/')}/genai/llmgw",
    api_key=os.environ["DATAROBOT_API_TOKEN"],
)
model = OpenAIChatModel("azure/gpt-5-mini-2025-08-07", provider=provider)

agent = Agent(model, system_prompt="You are a helpful assistant.")
```

`Agent.instrument_all()` uses the global `TracerProvider` that `configure_otel()` set,
so every agent in the process exports to DataRobot without editing each `Agent(...)`
constructor. PydanticAI creates spans for agent runs, LLM API calls, tool/function
calls, and retries/error handling.

## Version robustness

PydanticAI's instrumentation API has changed across versions. Prefer the global call;
fall back to per-agent only if you cannot call `instrument_all()` at startup.

- **Recommended — global, version-robust:**
  ```python
  Agent.instrument_all()
  ```

- **Per-agent alternative — PydanticAI v2.x:**
  ```python
  from pydantic_ai import Agent
  from pydantic_ai.capabilities import Instrumentation
  from pydantic_ai.models.instrumented import InstrumentationSettings

  agent = Agent(
      model,
      capabilities=[Instrumentation(settings=InstrumentationSettings())],
  )
  ```

- **Per-agent alternative — older PydanticAI v1.x:**
  ```python
  agent = Agent(model, instrument=True)
  ```

- **`InstrumentationSettings(version=…)`** controls the telemetry data format
  (`Literal[2, 3, 4, 5]`, default `5`, following OTel GenAI semantic conventions
  1.37.0). Versions 2–4 are deprecated and emit `PydanticAIDeprecationWarning`. Only
  pin a version if you must match a specific downstream schema (see Verify below).

- **Privacy — exclude prompt/completion content:**
  ```python
  from pydantic_ai import Agent
  from pydantic_ai.models.instrumented import InstrumentationSettings

  Agent.instrument_all(InstrumentationSettings(include_content=False))
  ```
  By default PydanticAI captures prompt and completion text in spans. Set
  `include_content=False` to suppress it (this also empties DataRobot's
  Prompt/Completion columns).

## Verify

After wiring, confirm telemetry lands in DataRobot: run the agent once, then check the
Use Case Tracing view (or the `dr xp` panel — see SKILL.md Step 5).

- **Spans present, Prompt/Completion columns populated** → done.
- **No spans at all** → `Agent.instrument_all()` was not called, or `configure_otel()`
  ran after agent creation. Fix ordering.
- **Spans present but Prompt/Completion columns empty** → attribute-name mismatch.
  PydanticAI's default `version=5` emits `gen_ai.input.messages` /
  `gen_ai.output.messages` (OTel GenAI semconv), while DataRobot's tracing table reads
  `gen_ai.prompt` / `gen_ai.completion`. To fix, add the `gen_ai.prompt` / `gen_ai.completion` span attributes yourself — note that **no** `InstrumentationSettings(version=…)` value produces them (every selectable format, 2–5, emits `gen_ai.input.messages` / `gen_ai.output.messages`), so pinning a version does not help. Also confirm `include_content` is not `False`, and check whether your DataRobot instance already normalizes the newer semconv attributes.

## Optional: Logfire instrumentation

If the user already uses Logfire or wants richer PydanticAI-specific spans:

```python
import logfire

logfire.configure(
    send_to_logfire=False
)  # don't send to Logfire cloud; keep global provider
logfire.instrument_pydantic_ai()
```

This layers detailed Pydantic validation spans on top of the standard OTel traces. Use
either direct OTel (`instrument_all()`) or Logfire, not both, to avoid duplicate traces.

## Extra Dependencies

Recommended, for intelligent trace analysis (see **Intelligent Trace Analysis via Tensile** below):
```
tensile>=0.12.0
```

None beyond the generic OTel packages.

Optional (if using Logfire):
```
logfire[pydantic-ai]
```

## Custom Metrics (Optional)

For custom metrics, wrap the agent run:

```python
import time
from opentelemetry import metrics

meter = metrics.get_meter("my-pydantic-agent")
request_counter = meter.create_counter("agent.requests", unit="1")
request_duration = meter.create_histogram("agent.request.duration_ms", unit="ms")


async def run_with_metrics(agent, prompt):
    start = time.time()
    try:
        result = await agent.run(prompt)
        request_counter.add(1, {"status": "success"})
        return result
    except Exception:
        request_counter.add(1, {"status": "error"})
        raise
    finally:
        request_duration.record((time.time() - start) * 1000)
```

## Intelligent Trace Analysis via Tensile

DataRobot provides intelligent trace analysis and remediation. This is made possible by the `tensile` package, which records agentic trajectories and allows them to be replayed. To enable this feature, the agent needs `tensile>=0.12.0` installed as a dependency and the OTel capture hook wired in directly after `configure_otel()`:

```python
from dr_otel_config import configure_otel

configure_otel()

import tensile.otel

tensile.otel.install(tracer=False)

from pydantic_ai import Agent  # PydanticAI imports and Agent.instrument_all() follow, as above
```

## Known Pitfalls

| Issue | Cause | Fix |
|-------|-------|-----|
| No spans from PydanticAI | `Agent.instrument_all()` never called (instrumentation is opt-in) | Call `Agent.instrument_all()` after `configure_otel()`, before running agents |
| No spans from PydanticAI | `configure_otel()` called after agent creation | Ensure `configure_otel()` runs before any PydanticAI import or agent instantiation |
| Spans appear but Prompt/Completion columns empty | v5 semconv attributes (`gen_ai.input.messages`) differ from DataRobot's `gen_ai.prompt`/`gen_ai.completion`; or `include_content=False` | Add the `gen_ai.prompt`/`gen_ai.completion` span attributes yourself (no `version` value emits them); ensure `include_content` is not `False` (see Verify) |
| Logfire overrides TracerProvider | `logfire.configure()` called with default settings | Use `send_to_logfire=False` to keep the global provider |
| Duplicate traces | Both direct OTel and Logfire active | Choose one approach — prefer direct OTel |
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall