sb-dev/openquant / .agents/skills/otel-instrumentation
otel-instrumentation
VerifiedConfigures trace spans, defines custom metrics, sets up log exporters, and optimizes sampling strategies for OpenTelemetry instrumentation. Use when instrumenting applications with traces, metrics, or logs. Triggers on requests for observability, telemetry, tracing, metrics collection, logging integration, or OTel setup.
| File | Kind | Size |
|---|---|---|
| README.md | markdown | 4.8 KB |
| SKILL.md | skill md | 6.0 KB |
| rules/capture-database-query-parameters.md | markdown | 4.9 KB |
| rules/logs.md | markdown | 10.1 KB |
| rules/metrics.md | markdown | 22.7 KB |
| rules/platforms/k8s.md | markdown | 7.7 KB |
| rules/resolve-values.md | markdown | 11.3 KB |
| rules/resources.md | markdown | 10.9 KB |
| rules/sdks/browser.md | markdown | 5.2 KB |
| rules/sdks/dotnet.md | markdown | 20.6 KB |
| rules/sdks/go.md | markdown | 29.9 KB |
| rules/sdks/java.md | markdown | 17.2 KB |
| rules/sdks/nextjs.md | markdown | 16.2 KB |
| rules/sdks/nodejs.md | markdown | 24.7 KB |
| rules/sdks/php.md | markdown | 16.2 KB |
| rules/sdks/python.md | markdown | 18.4 KB |
| rules/sdks/ruby.md | markdown | 15.1 KB |
| rules/sdks/scala.md | markdown | 23.6 KB |
| rules/sensitive-data.md | markdown | 16.2 KB |
| rules/spans.md | markdown | 32.0 KB |
| rules/telemetry.md | markdown | 2.1 KB |
| rules/validation.md | markdown | 8.9 KB |
| rules/verify-dependencies.md | markdown | 6.6 KB |
rules/sdks/java.md
View on GitHub---
title: "Java Instrumentation"
impact: HIGH
tags:
- java
- jvm
- backend
- server
---
# Java Instrumentation
Instrument Java applications to generate traces, logs, and metrics for deep insights into behavior and performance.
## Use cases
- **HTTP Request Monitoring**: Understand outgoing and incoming HTTP requests through traces and metrics, with drill-downs to database level
- **Database Performance**: Observe which database statements execute and measure their duration for optimization
- **Error Detection**: Reveal uncaught errors and the context in which they happened
## Installation
Download the agent JAR:
```sh
OTEL_JAVA_VERSION=$(curl -sf "https://api.github.com/repos/open-telemetry/opentelemetry-java-instrumentation/releases/latest" \
| grep '"tag_name"' | cut -d'"' -f4)
[ -z "$OTEL_JAVA_VERSION" ] && { echo "Failed to resolve OTel Java version" >&2; exit 1; }
wget "https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/download/${OTEL_JAVA_VERSION}/opentelemetry-javaagent.jar"
```
**Note**: The javaagent.jar contains both the agent and instrumentation libraries, enabling automatic instrumentation without modifying source code.
### Verifying dependencies
The javaagent needs no per-library coordinates; prefer it over adding instrumentation libraries to the build.
When you do add library coordinates, never write a version from memory; verify the coordinate resolves from Maven Central first, per [verify-dependencies](../verify-dependencies.md), and prefer importing the OpenTelemetry BOM at a verified version so per-artifact versions can be omitted:
```bash
mvn dependency:get -Dartifact=io.opentelemetry:opentelemetry-bom:1.51.0 -Dpackaging=pom
```
## Environment variables
All environment variables that control the SDK behavior:
| Variable | Required | Default | Description |
|----------|----------|---------|-------------|
| `OTEL_SERVICE_NAME` | Yes | `unknown_service:java` | Identifies your service in telemetry data |
| `OTEL_TRACES_EXPORTER` | No | `otlp` | Traces exporter (default is already `otlp`) |
| `OTEL_METRICS_EXPORTER` | No | `otlp` | Metrics exporter (default is already `otlp`) |
| `OTEL_LOGS_EXPORTER` | No | `otlp` | Logs exporter (default is already `otlp`) |
| `OTEL_TRACES_SAMPLER` | Yes | `parentbased_always_on` | **Set to `always_on`** — the default lets callers switch off tracing, see [sampling](../spans.md#sampling) |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Yes | `http://localhost:4318` | OTLP collector endpoint |
| `OTEL_EXPORTER_OTLP_HEADERS` | No | - | Headers for authentication (e.g., `Authorization=Bearer TOKEN`) |
| `OTEL_EXPORTER_OTLP_PROTOCOL` | No | `http/protobuf` | Protocol: `grpc`, `http/protobuf`, or `http/json` |
| `OTEL_RESOURCE_ATTRIBUTES` | No | - | Additional resource attributes (e.g., `deployment.environment.name=production`) |
**Note**: The Java agent defaults all exporters to `otlp` and the protocol to `http/protobuf`.
### Where to get configuration values
1. **OTLP Endpoint**: Your observability platform's OTLP endpoint
- In Dash0: [Settings → Organization → Endpoints](https://app.dash0.com/settings/endpoints?s=eJwtyzEOgCAQRNG7TG1Cb29h5REMcVclIUDYsSLcXUxsZ95vcJgbxNObEjNET_9Eok9wY2FIlzlNUnJItM_GYAM2WK7cqmgdlbcDE0yjHlRZfr7KuDJj2W-yoPf-AmNVJ2I%3D)
- Format: `https://<region>.your-platform.com`
2. **Auth Token**: API token for telemetry ingestion
- In Dash0: [Settings → Auth Tokens → Create Token](https://app.dash0.com/settings/auth-tokens)
3. **Service Name**: Choose a descriptive name (e.g., `order-api`, `checkout-service`)
## Configuration
### 1. Activate the SDK
The SDK is activated by attaching the Java agent to the JVM.
**Via `-javaagent` JVM parameter:**
```sh
java -javaagent:path/to/opentelemetry-javaagent.jar -jar myapp.jar
```
**Via `JAVA_TOOL_OPTIONS` environment variable:**
```sh
export JAVA_TOOL_OPTIONS="-javaagent:path/to/opentelemetry-javaagent.jar"
```
**Note**: The `JAVA_TOOL_OPTIONS` approach applies the agent to every JVM process started in that shell session, including build tools like Maven and Gradle.
### 2. Set service name
```sh
export OTEL_SERVICE_NAME="my-service"
```
### 3. Enable exporters
The Java agent defaults all exporters to `otlp`, so no additional configuration is needed to start exporting telemetry.
To explicitly set exporters:
```sh
# Already the default, but can be set explicitly
export OTEL_TRACES_EXPORTER="otlp"
export OTEL_METRICS_EXPORTER="otlp"
export OTEL_LOGS_EXPORTER="otlp"
```
### 4. Configure endpoint
```sh
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<OTLP_ENDPOINT>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_AUTH_TOKEN"
```
### 5. Optional: target specific dataset
```sh
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_AUTH_TOKEN,Dash0-Dataset=my-dataset"
```
## Complete setup
### Using environment variables
```sh
# Service identification
export OTEL_SERVICE_NAME="my-service"
# Configure endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<OTLP_ENDPOINT>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_AUTH_TOKEN"
# Export every span and sample in the Collector
export OTEL_TRACES_SAMPLER="always_on"
# Run with the agent
java -javaagent:path/to/opentelemetry-javaagent.jar -jar myapp.jar
```
### Using JAVA_TOOL_OPTIONS
```sh
# Service identification
export OTEL_SERVICE_NAME="my-service"
# Configure endpoint
export OTEL_EXPORTER_OTLP_ENDPOINT="https://<OTLP_ENDPOINT>"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_AUTH_TOKEN"
# Export every span and sample in the Collector
export OTEL_TRACES_SAMPLER="always_on"
# Activate SDK for all JVM processes
export JAVA_TOOL_OPTIONS="-javaagent:path/to/opentelemetry-javaagent.jar"
java -jar myapp.jar
```
### Using JVM system properties
```sh
java \
-javaagent:path/to/opentelemetry-javaagent.jar \
-Dotel.service.name=my-service \
-Dotel.exporter.otlp.endpoint=https://<OTLP_ENDPOINT> \
-Dotel.exporter.otlp.headers="Authorization=Bearer YOUR_AUTH_TOKEN" \
-Dotel.traces.sampler=always_on \
-jar myapp.jar
```
## Local development
### Console exporter
For development without a collector, use the console exporter to see telemetry in your terminal:
```sh
export OTEL_SERVICE_NAME="my-service"
export OTEL_TRACES_EXPORTER="console"
export OTEL_METRICS_EXPORTER="console"
export OTEL_LOGS_EXPORTER="console"
java -javaagent:path/to/opentelemetry-javaagent.jar -jar myapp.jar
```
This prints spans, metrics, and logs directly to stdout—useful for verifying instrumentation works before configuring a remote backend.
### Without a collector
If you set `OTEL_EXPORTER_OTLP_ENDPOINT` but have no collector running, you will see connection errors.
This is expected behavior.
**Options:**
1. Use `console` exporter during development (recommended for quick testing)
2. Run a local OpenTelemetry Collector
3. Point directly to your observability backend
## Resource configuration
Set `service.name`, `service.version`, and `deployment.environment.name` for every deployment.
See [resource attributes](../resources.md) for the full list of required and recommended attributes.
## Kubernetes setup
See [Kubernetes deployment](../platforms/k8s.md) for pod metadata injection, resource attributes, and Dash0 Kubernetes Operator guidance.
## Supported libraries
The auto-instrumentation agent automatically instruments:
| Category | Libraries |
|----------|-----------|
| Web frameworks | Spring Boot, Spring MVC, Spring WebFlux, Servlet, JAX-RS |
| HTTP clients | OkHttp, Apache HttpClient, java.net.HttpURLConnection |
| Database | JDBC, Hibernate, R2DBC |
| Redis | Lettuce, Jedis |
| Messaging | Kafka, RabbitMQ, JMS |
| gRPC | gRPC |
| Logging | Log4j, Logback, java.util.logging |
| AWS | AWS SDK v1, AWS SDK v2 |
Refer to the [OpenTelemetry registry](https://opentelemetry.io/ecosystem/registry/?language=java) for the complete list.
## Database query parameters
Prepared-statement parameter values (`db.query.parameter.<key>`) are off by default.
Read [capturing database query parameters](../capture-database-query-parameters.md) first — it covers the cross-language risks (PII, credentials) and the Collector-side defence-in-depth that must be in place before enabling capture.
Enable via environment variable on the Java agent:
```sh
export OTEL_INSTRUMENTATION_JDBC_EXPERIMENTAL_CAPTURE_QUERY_PARAMETERS=true
```
Or the equivalent JVM system property:
```sh
java -Dotel.instrumentation.jdbc.experimental.capture-query-parameters=true \
-javaagent:path/to/opentelemetry-javaagent.jar \
-jar myapp.jar
```
| | |
|---|---|
| Default | `false` |
| Attribute key | `db.query.parameter.<N>` — 0-based positional index, stringified |
| Coverage | **JDBC only.** Not R2DBC, Hibernate, jOOQ, MongoDB, Cassandra, Vertx-SQL, Couchbase, Jedis/Lettuce/Redisson, ClickHouse, Elasticsearch/OpenSearch, or InfluxDB. Connection-pool wrappers (HikariCP, c3p0, DBCP, Druid, Oracle UCP) emit pool metrics only. |
| Type whitelist | `Boolean`, `Number`, `String`, `Date`/`Time`/`Timestamp`, `URL`, `RowId`. Binary streams, blobs, clobs, refs, and arrays are excluded. |
| Length cap | None — long string values are captured in full. |
| Batches | Parameters are **not** captured for batch statements. |
| Sanitizer side-effect | Enabling capture forces `db.query.text` sanitization **off**. Literals in the SQL text are emitted verbatim. |
| Stability | Incubator semantic convention. |
## Custom spans
Add business context to auto-instrumented traces:
```java
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
public class OrderService {
private static final Tracer tracer =
GlobalOpenTelemetry.getTracer("my-service");
private static String stackTraceAsString(Throwable t) {
var sw = new java.io.StringWriter();
t.printStackTrace(new java.io.PrintWriter(sw));
return sw.toString();
}
public Order processOrder(Order order) {
Span span = tracer.spanBuilder("order.process").startSpan();
try (var scope = span.makeCurrent()) {
span.setAttribute("order.id", order.getId());
span.setAttribute("order.total", order.getTotal());
Order result = saveOrder(order);
return result;
} catch (Exception e) {
span.setStatus(StatusCode.ERROR, e.getMessage());
var ctx = span.getSpanContext();
logger.atError()
.addKeyValue("trace_id", ctx.getTraceId())
.addKeyValue("span_id", ctx.getSpanId())
.addKeyValue("exception.type", e.getClass().getName())
.addKeyValue("exception.message", e.getMessage())
.addKeyValue("exception.stacktrace", stackTraceAsString(e))
.log("order.process.failed");
throw e;
} finally {
span.end();
}
}
}
```
### Retrieving the active span
Auto-instrumentation creates spans you do not control directly (e.g., the `SERVER` span for an HTTP request).
To enrich these spans with business context or set their status, retrieve the active span from the current context.
See [adding attributes to auto-instrumented spans](../spans.md#adding-attributes-to-auto-instrumented-spans) for when to use this pattern.
```java
import io.opentelemetry.api.trace.Span;
@PostMapping("/api/orders")
public ResponseEntity<Order> createOrder(@RequestBody OrderRequest request) {
Span span = Span.current();
span.setAttribute("order.id", request.getOrderId());
span.setAttribute("tenant.id", request.getTenantId());
// ... handler logic
}
```
`Span.current()` returns a non-recording span if no span is active.
Calling `setAttribute` or `setStatus` on a non-recording span is a no-op, so no null check is needed.
### Span status rules
See [span status code](../spans.md#span-status-code) for the full rules.
This section shows how to apply them in Java.
#### Always include a status message with `ERROR`
The second argument to `setStatus` is the status message.
It must contain the error type and a short explanation — enough to understand the failure without opening the full trace.
```java
// BAD: no status message
span.setStatus(StatusCode.ERROR);
// BAD: generic message with no diagnostic value
span.setStatus(StatusCode.ERROR, "something went wrong");
// GOOD: specific message with error type and context
span.setStatus(StatusCode.ERROR, e.getClass().getSimpleName() + ": " + e.getMessage());
```
Do not include stack traces in the status message.
Record those in a log record with `exception.stacktrace` instead.
```java
import java.io.StringWriter;
import java.io.PrintWriter;
// BAD: stack trace in the status message
StringWriter sw = new StringWriter();
e.printStackTrace(new PrintWriter(sw));
span.setStatus(StatusCode.ERROR, sw.toString());
// GOOD: short message only
span.setStatus(StatusCode.ERROR, e.getMessage());
```
#### Use `OK` only for confirmed success
Set status to `OK` when application logic has explicitly verified the operation succeeded.
Leave status `UNSET` if the code simply did not encounter an error.
```java
// GOOD: explicit confirmation from downstream
HttpResponse<String> response = httpClient.send(request, HttpResponse.BodyHandlers.ofString());
if (response.statusCode() == 200) {
span.setStatus(StatusCode.OK);
}
// BAD: setting OK speculatively
span.setStatus(StatusCode.OK);
return someMethod(); // might still fail after this point
```
## Structured logging
Configure your logging framework to serialize exceptions into a single structured field so that stack traces do not break the one-line-per-record contract.
See [logs](../logs.md) for general guidance on structured logging and exception stack traces.
### Logback with logstash-logback-encoder
The [logstash-logback-encoder](https://github.com/logfellow/logstash-logback-encoder) produces single-line JSON with stack traces serialized into a `stack_trace` field.
```xml
<!-- logback.xml -->
<configuration>
<appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender">
<encoder class="net.logstash.logback.encoder.LogstashEncoder" />
</appender>
<root level="INFO">
<appender-ref ref="STDOUT" />
</root>
</configuration>
```
The encoder automatically captures exceptions passed to `logger.error("message", exception)` and serializes the full stack trace as an escaped string in the `stack_trace` JSON field.
### Log4j2 with JSON layout
Log4j2's `JsonTemplateLayout` produces single-line JSON output with stack traces in a structured field.
```xml
<!-- log4j2.xml -->
<Configuration>
<Appenders>
<Console name="STDOUT" target="SYSTEM_OUT">
<JsonTemplateLayout eventTemplateUri="classpath:EcsLayout.json" />
</Console>
</Appenders>
<Loggers>
<Root level="INFO">
<AppenderRef ref="STDOUT" />
</Root>
</Loggers>
</Configuration>
```
Avoid using `PatternLayout` for production logging — it produces multi-line stack traces that break log collectors.
## Graceful shutdown
The Java agent registers a JVM shutdown hook automatically.
When the JVM receives `SIGTERM` or `Runtime.getRuntime().exit()` is called, the hook flushes all pending spans, metrics, and log records before the process terminates.
No additional code is needed.
If you use a programmatic SDK setup (without the agent), register a shutdown hook manually:
```java
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
tracerProvider.close();
meterProvider.close();
loggerProvider.close();
}));
```
`close()` calls `shutdown()` internally, which flushes pending batches and releases resources.
## Troubleshooting
### No telemetry appearing
**Verify the agent is loaded:**
```sh
ps aux | grep opentelemetry-javaagent
```
Look for `opentelemetry-javaagent` in the JVM arguments.
If it is missing, the agent is not attached to the JVM process.
**Enable debug logging:**
```sh
export OTEL_LOG_LEVEL="debug"
```
Or via JVM system property:
```sh
java -Dotel.javaagent.debug=true -javaagent:path/to/opentelemetry-javaagent.jar -jar myapp.jar
```
### Connection errors
<!-- eval:skip -->
```
WARN io.opentelemetry.exporter.internal.http.HttpExporter - Failed to export spans.
```
This means the SDK is working but cannot reach the collector:
- **No collector running**: Start a local collector or use `OTEL_TRACES_EXPORTER=console`
- **Wrong endpoint**: Check `OTEL_EXPORTER_OTLP_ENDPOINT` is correct
- **Port mismatch**: gRPC uses 4317, HTTP uses 4318
### Agent not instrumenting libraries
**Symptom**: Agent loads but specific libraries are not instrumented
**Fix**: Verify the library version is supported by checking the [OpenTelemetry registry](https://opentelemetry.io/ecosystem/registry/?language=java).
Some very old library versions may not be covered by the auto-instrumentation.
### Conflicts with other Java agents
Running multiple Java agents (e.g., APM agents) alongside the OpenTelemetry agent can cause conflicts.
Remove other Java agents before attaching the OpenTelemetry agent.
## Resources
- [OpenTelemetry Java Documentation](https://opentelemetry.io/docs/languages/java/)
- [Java Agent Releases](https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases)
- [Environment Variable Specification](https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/)
- [Dash0 Kubernetes Operator](https://github.com/dash0hq/dash0-operator)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack." | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |