Skip to content

sb-dev/openquant / .agents/skills/otel-instrumentation

otel-instrumentation

Verified

Configures trace spans, defines custom metrics, sets up log exporters, and optimizes sampling strategies for OpenTelemetry instrumentation. Use when instrumenting applications with traces, metrics, or logs. Triggers on requests for observability, telemetry, tracing, metrics collection, logging integration, or OTel setup.

FileKindSize
README.mdmarkdown4.8 KB
SKILL.mdskill md6.0 KB
rules/capture-database-query-parameters.mdmarkdown4.9 KB
rules/logs.mdmarkdown10.1 KB
rules/metrics.mdmarkdown22.7 KB
rules/platforms/k8s.mdmarkdown7.7 KB
rules/resolve-values.mdmarkdown11.3 KB
rules/resources.mdmarkdown10.9 KB
rules/sdks/browser.mdmarkdown5.2 KB
rules/sdks/dotnet.mdmarkdown20.6 KB
rules/sdks/go.mdmarkdown29.9 KB
rules/sdks/java.mdmarkdown17.2 KB
rules/sdks/nextjs.mdmarkdown16.2 KB
rules/sdks/nodejs.mdmarkdown24.7 KB
rules/sdks/php.mdmarkdown16.2 KB
rules/sdks/python.mdmarkdown18.4 KB
rules/sdks/ruby.mdmarkdown15.1 KB
rules/sdks/scala.mdmarkdown23.6 KB
rules/sensitive-data.mdmarkdown16.2 KB
rules/spans.mdmarkdown32.0 KB
rules/telemetry.mdmarkdown2.1 KB
rules/validation.mdmarkdown8.9 KB
rules/verify-dependencies.mdmarkdown6.6 KB

rules/platforms/k8s.md

View on GitHub
---
title: 'Kubernetes deployment'
impact: HIGH
tags:
  - kubernetes
  - deployment
  - resource-attributes
  - downward-api
---

# Kubernetes deployment

OpenTelemetry-instrumented applications running in Kubernetes need pod metadata injected as resource attributes for proper contextualization of traces, metrics, and logs.
This file covers the pod-spec changes needed for application containers and the Dash0 Kubernetes Operator approach.

For the full list of required and recommended resource attributes (including `service.name`, `service.namespace`, `service.version`, `service.instance.id`, and `deployment.environment.name`), see [resource attributes](../resources.md).
In Kubernetes, every application pod has more than one instance in aggregate across replicas — always set `service.instance.id` and the other [service identity](../resources.md#service-identity) attributes so telemetry can be attributed to a specific pod.

## Pod metadata via downward API

Use the Kubernetes downward API to expose pod metadata as environment variables.
The SDK reads `OTEL_RESOURCE_ATTRIBUTES` at startup and attaches these values to every signal.

### `k8s.pod.uid` (critical)

The most important Kubernetes resource attribute.
The `k8sattributes` processor uses it to resolve most other Kubernetes metadata (namespace, deployment, node) automatically.

Always set `k8s.pod.uid` explicitly rather than relying on IP-based pod detection.
IP-based association is unreliable with service meshes (Istio, Linkerd) or non-standard network configurations where multiple pods share the same IP.

### `k8s.container.name` (critical)

Identifies the container within a pod.
Set it for every multi-container pod (e.g., pods with sidecar proxies or log collectors).
The `k8sattributes` processor cannot distinguish between containers that share the same pod UID and IP.

There is no downward API field for the container name.
Set it as a literal value that matches the container name in the pod spec.

### `k8s.pod.name`

Human-readable pod identifier.
Pod names are easier to search than UIDs but are unique only within a namespace on a cluster.

### `k8s.node.name`

Identifies the node on which the pod runs.
Required when investigating performance issues caused by resource contention or node-pressure evictions.

Not needed for AWS EKS on Fargate, where each pod runs on a dedicated virtual node.

### Complete pod spec example

<!-- eval:k8s -->
```yaml
spec:
  containers:
    - name: <container-name>
      env:
        - name: K8S_POD_UID
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.uid
        - name: K8S_POD_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: K8S_NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: OTEL_SERVICE_NAME
          value: <service-name>
        - name: OTEL_EXPORTER_OTLP_ENDPOINT
          value: <otlp-endpoint>
        - name: OTEL_EXPORTER_OTLP_HEADERS
          valueFrom:
            secretKeyRef:
              name: otel-auth
              key: token
        - name: OTEL_TRACES_EXPORTER
          value: otlp
        - name: OTEL_METRICS_EXPORTER
          value: otlp
        - name: OTEL_LOGS_EXPORTER
          value: otlp
        - name: OTEL_RESOURCE_ATTRIBUTES
          value: "service.namespace=<service-namespace>,service.version=<service-version or commit sha>,service.instance.id=$(K8S_POD_UID),deployment.environment.name=<dev-env>,k8s.pod.uid=$(K8S_POD_UID),k8s.pod.name=$(K8S_POD_NAME),k8s.node.name=$(K8S_NODE_NAME),k8s.container.name=<container-name>"
```

The value of `service.instance.id` is set to the pod UID via the downward API so it is unique across pods and stable for the lifetime of the process.
This is the deterministic strategy referenced in [resolving `service.instance.id`](../resolve-values.md#serviceinstanceid); when pods run multiple worker processes in one container, generate a UUID v5 seeded on the pod UID and the worker index instead so each worker has its own identifier.
Setting `service.instance.id` does not remove the need for `k8s.pod.uid` — both must be set (see [`service.instance.id`](../resources.md#serviceinstanceid)).

Set `OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, and `OTEL_LOGS_EXPORTER` to `otlp` explicitly.
Supported values are `otlp`, `console`, and `none`.
The OTel specification designates `otlp` as the recommended default, but older SDK versions and manual SDK setups (such as Node.js without an auto-instrumentation package) may default to `none`, silently dropping all telemetry.

The `$(K8S_POD_UID)` syntax is a Kubernetes dependent environment variable reference — Kubernetes substitutes it with the value of the `K8S_POD_UID` variable defined earlier in the same `env` block.

The placeholders (`<service-name>`, `<service-version or commit sha>`, `<dev-env>`, `<container-name>`) require project-specific values.
See [resolving configuration values](../resolve-values.md) for ordered lookup strategies to derive each value from the codebase.

## Collector-level enrichment

Attributes that cannot be set from inside the pod — workload names and UIDs (`k8s.deployment.name`, `k8s.deployment.uid`, `k8s.namespace.name`, etc.) and cluster identity (`k8s.cluster.name`, `k8s.cluster.uid`) — are handled by OpenTelemetry Collector processors.

For `k8sattributes` processor configuration (metadata extraction, pod association, passthrough mode, RBAC), see [processors](../../../otel-collector/rules/processors.md).
For `resource` processor configuration (`k8s.cluster.name`, `k8s.cluster.uid`), see [processors](../../../otel-collector/rules/processors.md).
For Collector deployment manifests (DaemonSet, Deployment, RBAC), see [raw manifests](../../../otel-collector/rules/deployment/raw-manifests.md).

## Dash0 Kubernetes Operator

Use the [Dash0 Kubernetes Operator](https://github.com/dash0hq/dash0-operator) to automate instrumentation and resource attribute injection for all workloads in a cluster.
The operator handles SDK injection, collector configuration, and Kubernetes metadata enrichment without manual pod spec changes.

When the Dash0 Kubernetes Operator manages the workload, skip the manual downward API and Collector processor setup above.

## Anti-patterns

- **Using `k8s.pod.ip` instead of `k8s.pod.uid`.**
  Pod IPs are reused and can be shared across pods in service mesh configurations.
  Always set `k8s.pod.uid` via the downward API.
- **Omitting `k8s.container.name` in multi-container pods.**
  The `k8sattributes` processor cannot distinguish between containers sharing the same pod UID and IP.
  There is no downward API field for the container name — set it as a literal value matching the container name in the pod spec.
- **Missing downward API environment variables.**
  Without `k8s.pod.uid` in `OTEL_RESOURCE_ATTRIBUTES`, the Collector's `k8sattributes` processor falls back to unreliable connection-IP matching.
  Always expose pod metadata via the downward API and pass it to the SDK.
- **Omitting `service.instance.id` in Kubernetes pods.**
  Multiple pod replicas share the same `service.name`, `service.namespace`, and `service.version`, so instance-level analysis is impossible without `service.instance.id`.
  Derive it from the pod UID via the downward API — see the [service identity](../resources.md#service-identity) rule.

## References

- [Kubernetes attributes best practices](https://www.dash0.com/guides/opentelemetry-kubernetes-attributes-best-practices)
- [Kubernetes semantic conventions](https://opentelemetry.io/docs/specs/semconv/resource/k8s/)
- [Kubernetes downward API](https://kubernetes.io/docs/concepts/workloads/pods/downward-api/)
- [Resource attributes](../resources.md)
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 198kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 194kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 194kUpdated Claude CodeCodexVerifiedInstall