datarobot-oss/datarobot-agent-skills / skills/datarobot-external-agent-monitoring
datarobot-external-agent-monitoring
VerifiedInstrument any external or existing AI agent with OpenTelemetry to send traces, logs, and metrics to DataRobot for monitoring, observability, and governance. Use when the user says "add tracing/observability/monitoring to my agent", wants to instrument an existing agent project in their IDE, wants to send agent traces, logs, or metrics to DataRobot, or wants to record and replay agent trajectories (Tensile) from DataRobot traces.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 19.1 KB |
| frameworks/crewai.md | markdown | 2.9 KB |
| frameworks/generic-python.md | markdown | 6.7 KB |
| frameworks/google-adk.md | markdown | 14.7 KB |
| frameworks/langchain-langgraph.md | markdown | 5.0 KB |
| frameworks/llamaindex.md | markdown | 3.3 KB |
| frameworks/pydantic-ai.md | markdown | 8.0 KB |
| reference/dr_otel_config.md | markdown | 8.5 KB |
| scripts/create_shell_deployment.pyexecutable | script | 6.3 KB |
| scripts/create_use_case.pyexecutable | script | 4.0 KB |
| scripts/verify_otel_connection.pyexecutable | script | 5.9 KB |
frameworks/llamaindex.md
View on GitHub# LlamaIndex — DataRobot OTel Integration
## Overview
LlamaIndex works with the standard `configure_otel()` pattern — it does NOT override the global TracerProvider. Use the generic `dr_otel_config.py` without modification.
Auto-instrumentation is available via `opentelemetry-instrumentation-llamaindex`, which creates spans for query engines, retrievers, LLM calls, and embedding operations.
## OTel Strategy
| Signal | Strategy |
|-----------|---------------------------------------------|
| **Traces** | Standard setup + auto-instrumentor |
| **Metrics** | Standard setup (optional custom callbacks) |
| **Logs** | Standard setup |
## Setup
### 1. Use the generic `dr_otel_config.py` as-is
No modifications needed. Call `configure_otel()` at startup.
### 2. Add auto-instrumentor after `configure_otel()`
In the agent's entrypoint:
```python
from dr_otel_config import configure_otel
configure_otel()
# Auto-instrument LlamaIndex — must be called AFTER configure_otel()
from opentelemetry.instrumentation.llamaindex import LlamaIndexInstrumentor
LlamaIndexInstrumentor().instrument()
# Your LlamaIndex code below...
```
The auto-instrumentor captures:
- Query engine executions
- Retriever operations (vector search, keyword search)
- LLM calls (prompts, completions, token usage)
- Embedding generation
- Node postprocessing
## Extra Dependencies
Recommended, for intelligent trace analysis (see **Intelligent Trace Analysis via Tensile** below):
```
tensile>=0.12.0
```
```
opentelemetry-instrumentation-llamaindex
```
## Alternative: LlamaIndex Built-in Callback
LlamaIndex also has a built-in OpenTelemetry callback handler. If the auto-instrumentor package is unavailable or incompatible:
```python
from llama_index.core.callbacks import CallbackManager
from llama_index.core.callbacks.open_inference_callback import (
OpenInferenceCallbackHandler,
)
callback_manager = CallbackManager([OpenInferenceCallbackHandler()])
# Pass callback_manager to your index/query engine
```
Prefer the auto-instrumentor when available — it's more comprehensive and doesn't require modifying query engine construction.
## Intelligent Trace Analysis via Tensile
DataRobot provides intelligent trace analysis and remediation. This is made possible by the `tensile` package, which records agentic trajectories and allows them to be replayed. To enable this feature, the agent needs `tensile>=0.12.0` installed as a dependency and the OTel capture hook wired in directly after `configure_otel()`:
```python
from dr_otel_config import configure_otel
configure_otel()
from opentelemetry.instrumentation.llamaindex import LlamaIndexInstrumentor
LlamaIndexInstrumentor().instrument()
import tensile.otel
tensile.otel.install(tracer=False)
# Your LlamaIndex code below...
```
## Known Pitfalls
| Issue | Cause | Fix |
|-------|-------|-----|
| No spans from LlamaIndex | Instrumentor not called | Ensure `LlamaIndexInstrumentor().instrument()` is called AFTER `configure_otel()` |
| Import path changed | LlamaIndex v0.10+ restructured packages | Check if instrumentation package matches your LlamaIndex version |
| Missing embedding spans | Old instrumentor version | Update to latest `opentelemetry-instrumentation-llamaindex` |
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 323k | Updated | Claude CodeCodex | Verified | Install |