Skip to content

datarobot-oss/datarobot-agent-skills / skills/datarobot-external-agent-monitoring

datarobot-external-agent-monitoring

Verified

Instrument any external or existing AI agent with OpenTelemetry to send traces, logs, and metrics to DataRobot for monitoring, observability, and governance. Use when the user says "add tracing/observability/monitoring to my agent", wants to instrument an existing agent project in their IDE, wants to send agent traces, logs, or metrics to DataRobot, or wants to record and replay agent trajectories (Tensile) from DataRobot traces.

FileKindSize
SKILL.mdskill md19.1 KB
frameworks/crewai.mdmarkdown2.9 KB
frameworks/generic-python.mdmarkdown6.7 KB
frameworks/google-adk.mdmarkdown14.7 KB
frameworks/langchain-langgraph.mdmarkdown5.0 KB
frameworks/llamaindex.mdmarkdown3.3 KB
frameworks/pydantic-ai.mdmarkdown8.0 KB
reference/dr_otel_config.mdmarkdown8.5 KB
scripts/create_shell_deployment.pyexecutablescript6.3 KB
scripts/create_use_case.pyexecutablescript4.0 KB
scripts/verify_otel_connection.pyexecutablescript5.9 KB

frameworks/langchain-langgraph.md

View on GitHub
# LangChain / LangGraph — DataRobot OTel Integration

## Overview

LangChain and LangGraph work with the standard `configure_otel()` pattern — they do NOT override the global TracerProvider. Use the generic `dr_otel_config.py` without modification.

Auto-instrumentation is available via `opentelemetry-instrumentation-langchain`, which automatically creates spans for chains, LLM calls, tool invocations, and retriever operations.

## OTel Strategy

| Signal    | Strategy                                    |
|-----------|---------------------------------------------|
| **Traces**  | Standard setup + auto-instrumentor         |
| **Metrics** | Standard setup (optional custom callbacks) |
| **Logs**    | Standard setup                             |

## Setup

### 1. Use the generic `dr_otel_config.py` as-is

No modifications needed. Call `configure_otel()` at startup.

### 2. Add auto-instrumentor after `configure_otel()`

In the agent's entrypoint:

```python
from dr_otel_config import configure_otel

configure_otel()

# Auto-instrument LangChain — must be called AFTER configure_otel()
from opentelemetry.instrumentation.langchain import LangchainInstrumentor

LangchainInstrumentor().instrument()

# Your agent code below...
```

The auto-instrumentor captures:
- Chain executions (with input/output)
- LLM calls (model, prompt, completion, token usage)
- Tool/function calls
- Retriever operations (for RAG)
- Agent reasoning steps (for LangGraph)

**Important: `tool_name` attribute for DataRobot.** LangGraph does NOT set the `tool_name` span attribute by default. DataRobot's tracing table requires `tool_name` (underscore) to populate the Tools column. Add it manually inside your tools:

```python
from opentelemetry import trace


@tool
def search_database(query: str) -> str:
    """Search the database."""
    span = trace.get_current_span()
    span.set_attribute("tool_name", "search_database")
    # ... tool logic ...
```

Without this, tool spans will appear in the trace hierarchy but the Tools column in DataRobot will be empty.

### 3. Optional: Add OpenAI/Anthropic SDK instrumentors

If the agent uses OpenAI or Anthropic SDKs directly (in addition to LangChain), add their instrumentors too:

```python
# Optional — for direct SDK calls outside LangChain
from opentelemetry.instrumentation.openai import OpenAIInstrumentor

OpenAIInstrumentor().instrument()
```

## Extra Dependencies

Recommended, for intelligent trace analysis (see **Intelligent Trace Analysis via Tensile** below):
```
tensile>=0.12.0
```

```
opentelemetry-instrumentation-langchain
```

Optional:
```
opentelemetry-instrumentation-openai      # If using OpenAI SDK directly
opentelemetry-instrumentation-anthropic    # If using Anthropic SDK directly
```

## Custom Metrics (Optional)

LangChain/LangGraph agents don't require a custom metrics callback module — the auto-instrumentor handles trace spans. For custom metrics (request counts, latency histograms), you can optionally add a LangChain callback handler:

```python
from opentelemetry import metrics

meter = metrics.get_meter("my-langchain-agent")
request_counter = meter.create_counter("agent.requests", unit="1")
request_duration = meter.create_histogram("agent.request.duration_ms", unit="ms")

# Use LangChain's callback system to record metrics
from langchain_core.callbacks import BaseCallbackHandler


class DataRobotMetricsHandler(BaseCallbackHandler):
    def on_chain_start(self, serialized, inputs, **kwargs):
        import time

        kwargs.setdefault("metadata", {})["_dr_start"] = time.time()

    def on_chain_end(self, outputs, **kwargs):
        import time

        start = kwargs.get("metadata", {}).get("_dr_start", time.time())
        elapsed_ms = (time.time() - start) * 1000
        request_counter.add(1, {"status": "success"})
        request_duration.record(elapsed_ms)

    def on_chain_error(self, error, **kwargs):
        request_counter.add(1, {"status": "error"})
```

This is optional — the auto-instrumentor already provides comprehensive trace spans.

## Intelligent Trace Analysis via Tensile

DataRobot provides intelligent trace analysis and remediation. This is made possible by the `tensile` package, which records agentic trajectories and allows them to be replayed. To enable this feature, the agent needs `tensile>=0.12.0` installed as a dependency and the OTel capture hook wired in directly after `configure_otel()`:

```python
from dr_otel_config import configure_otel

configure_otel()

from opentelemetry.instrumentation.langchain import LangchainInstrumentor

LangchainInstrumentor().instrument()

import tensile.otel

tensile.otel.install(tracer=False)

# Your agent code below...
```

## Known Pitfalls

| Issue | Cause | Fix |
|-------|-------|-----|
| No spans from LangChain | Instrumentor not called | Ensure `LangchainInstrumentor().instrument()` is called AFTER `configure_otel()` |
| Duplicate spans | Multiple instrumentors active | Only instrument once; check for existing instrumentation |
| Missing retriever spans | Old instrumentor version | Update `opentelemetry-instrumentation-langchain` to latest |
reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness.Stars: 323kUpdated Claude CodeCodexVerifiedInstall
dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependenciesStars: 323kUpdated Claude CodeCodexVerifiedInstall
executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is availableStars: 323kUpdated Claude CodeCodexVerifiedInstall