sb-dev/openquant / .agents/skills/otel-instrumentation
otel-instrumentation
VerifiedConfigures trace spans, defines custom metrics, sets up log exporters, and optimizes sampling strategies for OpenTelemetry instrumentation. Use when instrumenting applications with traces, metrics, or logs. Triggers on requests for observability, telemetry, tracing, metrics collection, logging integration, or OTel setup.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 4.8 KB |
| SKILL.md | skill md | 6.0 KB |
| rules/capture-database-query-parameters.md | markdown | 4.9 KB |
| rules/logs.md | markdown | 10.1 KB |
| rules/metrics.md | markdown | 22.7 KB |
| rules/platforms/k8s.md | markdown | 7.7 KB |
| rules/resolve-values.md | markdown | 11.3 KB |
| rules/resources.md | markdown | 10.9 KB |
| rules/sdks/browser.md | markdown | 5.2 KB |
| rules/sdks/dotnet.md | markdown | 20.6 KB |
| rules/sdks/go.md | markdown | 29.9 KB |
| rules/sdks/java.md | markdown | 17.2 KB |
| rules/sdks/nextjs.md | markdown | 16.2 KB |
| rules/sdks/nodejs.md | markdown | 24.7 KB |
| rules/sdks/php.md | markdown | 16.2 KB |
| rules/sdks/python.md | markdown | 18.4 KB |
| rules/sdks/ruby.md | markdown | 15.1 KB |
| rules/sdks/scala.md | markdown | 23.6 KB |
| rules/sensitive-data.md | markdown | 16.2 KB |
| rules/spans.md | markdown | 32.0 KB |
| rules/telemetry.md | markdown | 2.1 KB |
| rules/validation.md | markdown | 8.9 KB |
| rules/verify-dependencies.md | markdown | 6.6 KB |
rules/sdks/python.md
View on GitHub---
title: "Python Instrumentation"
impact: HIGH
tags:
- python
- backend
- server
---
# Python Instrumentation
Instrument Python applications to generate traces, logs, and metrics for deep insights into behavior and performance.
## Use cases
- **HTTP Request Monitoring**: Understand outgoing and incoming HTTP requests through traces and metrics, with drill-downs to database level
- **Database Performance**: Observe which database statements execute and measure their duration for optimization
- **Error Detection**: Reveal uncaught errors and the context in which they happened
## Installation
```bash
pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install
```
The `opentelemetry-distro` package includes dependencies for auto-instrumentation, the OTLP exporter, and the SDK.
The `opentelemetry-bootstrap` command detects installed libraries and installs the corresponding instrumentation packages automatically.
**Note**: Installing the packages alone is insufficient—you must activate the SDK AND configure exporters.
### Verifying dependencies
Never write an `opentelemetry-instrumentation-*` package or version into `requirements.txt` (or `pyproject.toml`) from memory; verify it first, per [verify-dependencies](../verify-dependencies.md).
`opentelemetry-bootstrap` is the authoritative check: it lists only the instrumentation packages the installed contrib release ships, matched against the libraries present in the environment.
```bash
opentelemetry-bootstrap # prints package==version for every detected library
pip index versions opentelemetry-instrumentation-flask # existence and published versions
```
A library missing from the bootstrap output has no current instrumentation — retired instrumentations (such as elasticsearch) still install from PyPI at their last release but are no longer maintained or shipped.
Do not hand-pin such a package; instrument the library manually instead.
pip enforces `Requires-Python` natively, resolving to the newest release compatible with the interpreter — but only the interpreter pip runs under, so run the resolution (or `pip index versions`) under the same Python version the project builds and deploys with (base image, CI), not whatever is on your PATH.
Plain `requirements.txt` carries no lockfile, so there is nothing to regenerate; projects locked with Poetry, uv, or Pipenv follow the manifest-and-lockfile rule in [verify-dependencies](../verify-dependencies.md#keeping-the-lockfile-in-step) — regenerate the lockfile (`poetry lock`, `uv lock`, `pipenv lock`) with every manifest edit.
## Environment variables
All environment variables that control the SDK behavior:
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `OTEL_SERVICE_NAME` | Yes | `unknown_service` | Identifies your service in telemetry data |
| `OTEL_TRACES_EXPORTER` | No | `otlp` | Trace exporter (defaults to `otlp`, unlike Node.js) |
| `OTEL_METRICS_EXPORTER` | No | `none` | Set to `otlp` to export metrics |
| `OTEL_LOGS_EXPORTER` | No | `otlp` | Log exporter (defaults to `otlp`, unlike Node.js) |
| `OTEL_TRACES_SAMPLER` | Yes | `parentbased_always_on` | **Set to `always_on`** — the default lets callers switch off tracing, see [sampling](../spans.md#sampling) |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Yes | `http://localhost:4317` | OTLP collector endpoint |
| `OTEL_EXPORTER_OTLP_HEADERS` | No | - | Headers for authentication (e.g., `Authorization=Bearer TOKEN`) |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | No | `grpc` | Protocol: `grpc`, `http/protobuf`, or `http/json` |
| `OTEL_RESOURCE_ATTRIBUTES` | No | - | Additional resource attributes (e.g., `deployment.environment.name=production`) |
**Note**: Unlike Node.js, the Python SDK defaults `OTEL_TRACES_EXPORTER` and `OTEL_LOGS_EXPORTER` to `otlp`, so traces and logs are exported without explicitly setting these variables.
### Where to get configuration values
1. **OTLP Endpoint**: Your observability platform's OTLP endpoint
- In Dash0: [Settings → Organization → Endpoints](https://app.dash0.com/settings/endpoints?s=eJwtyzEOgCAQRNG7TG1Cb29h5REMcVclIUDYsSLcXUxsZ95vcJgbxNObEjNET_9Eok9wY2FIlzlNUnJItM_GYAM2WK7cqmgdlbcDE0yjHlRZfr7KuDJj2W-yoPf-AmNVJ2I%3D)
- Format: `https://<region>.your-platform.com`
2. **Auth Token**: API token for telemetry ingestion
- In Dash0: [Settings → Auth Tokens → Create Token](https://app.dash0.com/settings/auth-tokens)
3. **Service Name**: Choose a descriptive name (e.g., `order-api`, `checkout-service`)
## Configuration
### 1. Activate the SDK
The SDK is activated by running your application with the `opentelemetry-instrument` command:
```bash
opentelemetry-instrument python main.py
```
**Flask:**
```bash
opentelemetry-instrument flask --app myapp run
```
**Django:**
```bash
opentelemetry-instrument python manage.py runserver
```
### 2. Set service name
```bash
export OTEL_SERVICE_NAME="my-service"
```
### 3. Enable exporters
Traces and logs default to `otlp`, so you only need to enable metrics explicitly:
```bash
# Optional: also export metrics
export OTEL_METRICS_EXPORTER="otlp"
```
### 4. Configure endpoint
```bash
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<OTLP_ENDPOINT>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_AUTH_TOKEN"
```
### 5. Optional: target specific dataset
```bash
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_AUTH_TOKEN,Dash0-Dataset=my-dataset"
```
## Complete setup
### Using environment variables
```bash
# Service identification
export OTEL_SERVICE_NAME="my-service"
# Enable metrics exporter (traces and logs default to otlp)
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_TRACES_SAMPLER="always_on"
# Configure endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<OTLP_ENDPOINT>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_AUTH_TOKEN"
# Run with auto-instrumentation
opentelemetry-instrument python main.py
```
### Using .env file
Python does not automatically load `.env` files.
Use a library like [python-dotenv](https://pypi.org/project/python-dotenv/) or export variables in your shell before running:
**.env:**
```bash
OTEL_SERVICE_NAME=my-service
OTEL_METRICS_EXPORTER=otlp
OTEL_TRACES_SAMPLER=always_on
OTEL_EXPORTER_OTLP_ENDPOINT=https://<OTLP_ENDPOINT>
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer YOUR_AUTH_TOKEN
```
**Run with:**
```bash
# Using shell export
set -a && source .env && set +a
opentelemetry-instrument python main.py
```
### Using a Makefile
Add instrumented targets to your `Makefile`:
```makefile
run:
python main.py
run-otel:
set -a && source .env && set +a && opentelemetry-instrument python main.py
run-otel-console:
OTEL_SERVICE_NAME=my-service OTEL_TRACES_EXPORTER=console OTEL_LOGS_EXPORTER=console opentelemetry-instrument python main.py
```
**Usage:**
```bash
make run-otel # Run with OTLP export to backend
make run-otel-console # Run with console output (no collector needed)
```
## Local development
### Console exporter
For development without a collector, use the console exporter to see telemetry in your terminal:
```bash
export OTEL_SERVICE_NAME="my-service"
export OTEL_TRACES_EXPORTER="console"
export OTEL_METRICS_EXPORTER="console"
export OTEL_LOGS_EXPORTER="console"
opentelemetry-instrument python main.py
```
This prints spans, metrics, and logs directly to stdout—useful for verifying instrumentation works before configuring a remote backend.
### Without a collector
If you use the default `otlp` exporter but have no collector running, you'll see connection errors. This is expected behavior:
<!-- eval:skip -->
```
Failed to export batch. UNAVAILABLE: failed to connect to all addresses
```
**Options:**
1. Use `console` exporter during development (recommended for quick testing)
2. Run a local OpenTelemetry Collector
3. Point directly to your observability backend
## Resource configuration
Set `service.name`, `service.version`, and `deployment.environment.name` for every deployment.
See [resource attributes](../resources.md) for the full list of required and recommended attributes.
## Kubernetes setup
See [Kubernetes deployment](../platforms/k8s.md) for pod metadata injection, resource attributes, and Dash0 Kubernetes Operator guidance.
## Supported libraries
The auto-instrumentation packages automatically instrument:
| Category | Libraries |
|----------|-----------|
| HTTP | Flask, Django, FastAPI, requests, urllib3, aiohttp |
| Database | psycopg2, mysql-connector, pymongo, redis |
| ORM | SQLAlchemy |
| Messaging | Celery, pika (RabbitMQ), confluent-kafka |
| AWS | boto3, botocore |
| Logging | logging (stdlib) |
| gRPC | grpcio |
| AI/LLM | [OpenLLMetry](https://github.com/traceloop/openllmetry), [OpenLit](https://github.com/openlit/openlit) (generative AI, Langchain observability) |
Refer to [OpenTelemetry documentation](https://opentelemetry.io/ecosystem/registry/?language=python) for the complete list.
## Database query parameters
Prepared-statement parameter values are off by default.
Read [capturing database query parameters](../capture-database-query-parameters.md) first — it covers the cross-language risks and the Collector-side defence-in-depth that must be in place before enabling capture.
Python does **not** expose an environment variable.
Enable capture per instrumentor at SDK initialization:
```python
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor
Psycopg2Instrumentor().instrument(capture_parameters=True)
```
The same kwarg exists on `PsycopgInstrumentor`, `AsyncPGInstrumentor`, `TortoiseORMInstrumentor`, and the dbapi-base `trace_integration(..., capture_parameters=True)`.
| | |
|---|---|
| Default | `False` |
| Attribute key | **`db.statement.parameters`** — singular attribute with plural name. Does **not** use the standard `db.query.parameter.<key>` schema. |
| Value shape | The entire DBAPI parameters tuple stringified via `str(args[1])` — one attribute carries everything, with no per-parameter keying. |
| Coverage | `dbapi` base, `psycopg`, `psycopg2`, `asyncpg`, `tortoiseorm`. |
| Silently dropped | `mysql`, `mysqlclient`, `pymysql`, `pymssql`, `sqlite3`, `aiopg` — the dbapi base supports the kwarg, but these wrappers do not forward it. |
| No-op | `sqlalchemy` has no parameter capture at all. |
| Type whitelist | None. |
| Length cap | None. |
**The attribute key is non-standard.**
A consumer or backend that keys off `db.query.parameter.*` will not pick up Python values, and `db.statement.parameters` carries the entire DBAPI parameters tuple as a single language-specific `str()` representation — per-parameter values cannot be reconstructed downstream without parsing Python repr output.
Treat the Python attribute as opaque debugging context, not as a structured replacement for `db.query.parameter.<key>`.
## Custom spans
Add business context to auto-instrumented traces:
```python
from opentelemetry import trace
from opentelemetry.trace import StatusCode
tracer = trace.get_tracer("my-service")
def process_order(order):
with tracer.start_as_current_span("order.process") as span:
try:
span.set_attribute("order.id", order["id"])
span.set_attribute("order.total", order["total"])
result = save_order(order)
return result
except Exception as error:
span.set_status(StatusCode.ERROR, str(error))
ctx = span.get_span_context()
logger.error("order.process.failed", extra={
"trace_id": format(ctx.trace_id, "032x"),
"span_id": format(ctx.span_id, "016x"),
"exception.type": type(error).__name__,
"exception.message": str(error),
"exception.stacktrace": traceback.format_exc(),
})
raise
```
### Retrieving the active span
Auto-instrumentation creates spans you do not control directly (e.g., the `SERVER` span for an HTTP request).
To enrich these spans with business context or set their status, retrieve the active span from the current context.
See [adding attributes to auto-instrumented spans](../spans.md#adding-attributes-to-auto-instrumented-spans) for when to use this pattern.
```python
from opentelemetry import trace
@app.route("/api/orders", methods=["POST"])
def create_order():
span = trace.get_current_span()
span.set_attribute("order.id", request.json["order_id"])
span.set_attribute("tenant.id", request.headers.get("X-Tenant-Id"))
# ... handler logic
```
`trace.get_current_span()` returns a non-recording span if no span is active.
Calling `set_attribute` or `set_status` on a non-recording span is a no-op, so no guard is needed.
### Span status rules
See [span status code](../spans.md#span-status-code) for the full rules.
This section shows how to apply them in Python.
#### Always include a status message with `ERROR`
The second argument to `set_status` is the status message.
It must contain the error type and a short explanation — enough to understand the failure without opening the full trace.
```python
from opentelemetry.trace import StatusCode
# BAD: no status message
span.set_status(StatusCode.ERROR)
# BAD: generic message with no diagnostic value
span.set_status(StatusCode.ERROR, "something went wrong")
# GOOD: specific message with error type and context
span.set_status(StatusCode.ERROR, f"TimeoutError: upstream payment service did not respond within 5s")
```
Do not include tracebacks in the status message.
Record those in a log record with `exception.stacktrace` instead.
```python
import traceback
# BAD: traceback in the status message
span.set_status(StatusCode.ERROR, traceback.format_exc())
# GOOD: short message only
span.set_status(StatusCode.ERROR, str(error))
```
#### Use `OK` only for confirmed success
Set status to `OK` when application logic has explicitly verified the operation succeeded.
Leave status `UNSET` if the code simply did not encounter an error.
```python
# GOOD: explicit confirmation from downstream
response = requests.get(url)
if response.ok:
span.set_status(StatusCode.OK)
# BAD: setting OK speculatively
span.set_status(StatusCode.OK)
return some_function() # might still fail after this point
```
## Structured logging
Configure your logging framework to serialize exceptions into a single structured field so that stack traces do not break the one-line-per-record contract.
See [logs](../logs.md) for general guidance on structured logging and exception stack traces.
### python-json-logger
The standard `logging` module prints multi-line stack traces by default.
Use [python-json-logger](https://pypi.org/project/python-json-logger/) to output single-line JSON with the stack trace captured in a structured field.
```python
import logging
from pythonjsonlogger.json import JsonFormatter
handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter(
fmt="%(asctime)s %(levelname)s %(name)s %(message)s",
rename_fields={"levelname": "level", "asctime": "timestamp"},
))
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.INFO)
try:
process_order(order)
except Exception:
logger.exception("order.failed", extra={"order_id": order_id})
```
`logger.exception()` automatically captures the stack trace.
The JSON formatter serializes it into an `exc_info` field as a single escaped string, keeping the log record on one line.
### structlog
[structlog](https://www.structlog.org/) produces single-line JSON output by default when configured with its JSON renderer.
```python
import structlog
structlog.configure(
processors=[
structlog.processors.add_log_level,
structlog.processors.TimeStamper(fmt="iso"),
structlog.processors.format_exc_info,
structlog.processors.JSONRenderer(),
],
)
logger = structlog.get_logger()
try:
process_order(order)
except Exception:
logger.exception("order.failed", order_id=order_id)
```
The `format_exc_info` processor converts the stack trace into a single string field before JSON serialization.
## Graceful shutdown
The `opentelemetry-instrument` command registers an `atexit` hook automatically.
When the process exits normally (including unhandled exceptions in most WSGI/ASGI servers), the hook flushes all pending spans, metrics, and log records before termination.
No additional code is needed for the auto-instrumented setup.
If you use a programmatic SDK setup (without `opentelemetry-instrument`), register a shutdown hook manually:
```python
import atexit
atexit.register(tracer_provider.shutdown)
atexit.register(meter_provider.shutdown)
atexit.register(logger_provider.shutdown)
```
`shutdown()` flushes pending batches and releases resources.
The call blocks until export completes or the timeout expires (default 30 seconds).
## Troubleshooting
### No telemetry appearing
**Check the exporter configuration:**
```bash
echo $OTEL_TRACES_EXPORTER # Defaults to "otlp"; check it is not set to "none"
```
**Verify bootstrap installed instrumentations:**
```bash
opentelemetry-bootstrap -a requirements
```
This lists the instrumentation packages that should be installed based on your project's dependencies.
If packages are missing, re-run `opentelemetry-bootstrap -a install`.
**Verify SDK is active:**
Ensure you are running your application with the `opentelemetry-instrument` command, not just `python`.
### Connection errors
<!-- eval:skip -->
```
Failed to export batch. UNAVAILABLE: failed to connect to all addresses
```
This means the SDK is working but cannot reach the collector:
- **No collector running**: Start a local collector or use `OTEL_TRACES_EXPORTER=console`
- **Wrong endpoint**: Check `OTEL_EXPORTER_OTLP_ENDPOINT` is correct
- **Port mismatch**: gRPC uses 4317, HTTP uses 4318
### Debug logging
Enable debug-level logging to see detailed SDK output:
```bash
export OTEL_LOG_LEVEL="debug"
opentelemetry-instrument python main.py
```
This reveals exporter activity, span creation, and configuration issues.
### Missing instrumentations
**Symptom**: SDK loads but specific libraries are not instrumented
**Fix**: Run the bootstrap command to detect and install missing instrumentation packages:
```bash
opentelemetry-bootstrap -a install
```
Ensure the bootstrap command runs in the same virtual environment as your application.
## Resources
- [OpenTelemetry Python Documentation](https://opentelemetry.io/docs/languages/python/getting-started/)
- [Auto-Instrumentation Package](https://pypi.org/project/opentelemetry-distro/)
- [Environment Variable Specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/)
- [Dash0 Kubernetes Operator](https://github.com/dash0hq/dash0-operator)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |