Skip to content

oliverv/caro / .agents/skills/oma-observability

oma-observability

Unverified

Design or troubleshoot telemetry, SLOs, and incident diagnostics. Route to the relevant signal, system boundary, or vendor guide.

FileKindSize
SKILL.mdskill md14.6 KB
resources/anti-patterns.mdmarkdown45.7 KB
resources/boundaries/cross-application.mdmarkdown19.5 KB
resources/boundaries/multi-tenant.mdmarkdown17.9 KB
resources/boundaries/release.mdmarkdown12.4 KB
resources/boundaries/slo.mdmarkdown6.5 KB
resources/checklist.mdmarkdown15.0 KB
resources/examples.mdmarkdown15.7 KB
resources/execution-protocol.mdmarkdown9.5 KB
resources/incident-forensics.mdmarkdown23.3 KB
resources/intent-rules.mdmarkdown11.8 KB
resources/layers/L3-network.mdmarkdown17.6 KB
resources/layers/L4-transport.mdmarkdown14.6 KB
resources/layers/L7-application/crash-analytics.mdmarkdown17.6 KB
resources/layers/L7-application/mobile-rum.mdmarkdown15.8 KB
resources/layers/L7-application/waf.mdmarkdown16.6 KB
resources/layers/L7-application/web-rum.mdmarkdown17.3 KB
resources/layers/mesh.mdmarkdown20.0 KB
resources/matrix.mdmarkdown21.1 KB
resources/meta-observability.mdmarkdown23.5 KB
resources/observability-as-code.mdmarkdown21.8 KB
resources/signals/audit.mdmarkdown13.9 KB
resources/signals/cost.mdmarkdown17.2 KB
resources/signals/logs.mdmarkdown10.2 KB
resources/signals/metrics.mdmarkdown19.6 KB
resources/signals/privacy.mdmarkdown17.5 KB
resources/signals/profiles.mdmarkdown15.2 KB
resources/signals/traces.mdmarkdown19.1 KB
resources/standards.mdmarkdown12.7 KB
resources/transport/collector-topology.mdmarkdown11.9 KB
resources/transport/otlp-grpc-vs-http.mdmarkdown7.6 KB
resources/transport/sampling-recipes.mdmarkdown11.0 KB
resources/transport/udp-statsd-mtu.mdmarkdown6.3 KB
resources/vendor-categories.mdmarkdown17.7 KB

resources/boundaries/cross-application.md

View on GitHub
---
otel_spec: "1.x (stable API/SDK)"
otel_semconv: "1.43.0 (2026-07)"
specs:
  - "W3C Trace Context: Level 1 Recommendation 2020-02-06; Level 2 Candidate Recommendation"
  - "W3C Baggage: Recommendation 2022-12-22"
---

# Cross-Application Observability Boundary

## 1. Scope

This file governs trace context, request correlation, causation chains, and baggage propagation across application boundaries; including service-to-service, cross-cloud, and cross-mesh calls.

Design decision D3 absorbs two formerly separate documents into this file:

- **Propagators** (formerly `propagators.md`): per-ecosystem header formats and normalization rules for W3C, B3, AWS X-Ray, GCP, Azure, Datadog, Cloudflare, Istio/Envoy, and Linkerd.
- **DDD bounded context** (formerly `multi-domain.md`): `service.namespace` as the domain grouping attribute for multi-service architectures.

Cross-references to other files in this skill use the `../` prefix convention.

---

## 2. 4-Layer Correlation Model

Distributed systems require four distinct correlation identifiers, each serving a different audience and cardinality need. Using only `trace_id` conflates concerns; using only `request_id` loses async chain visibility.

| Layer | Identifier | Propagation mechanism | Cardinality | Primary audience |
|-------|-----------|----------------------|-------------|-----------------|
| Infra / Trace | `trace_id` / `span_id` | W3C `traceparent` header | 1 per trace | Platform engineers, trace backends |
| Correlation | `request_id` (`x-request-id`) | HTTP header + log field `http.request.id` | 1 per request | Customer support, error banners, frontend users |
| Causation | `causation_id` | Message headers (Kafka, SQS, RabbitMQ) | 1 per event chain | Async workflow lineage, CQRS / event sourcing |
| Idempotency | `idempotency_key` | HTTP header (`Idempotency-Key`) or message header | 1 per business operation | Duplicate-processing guard; distinct from causation |
| Business | `user.id`, `tenant.id`, `order.id` | W3C Baggage (`baggage` header) | Per business entity | Product analytics, cross-cutting dashboards |

> **Causation vs. idempotency are distinct**; `causation_id` expresses event ancestry (what triggered this event) and is often unique per edge in an event chain; `idempotency_key` is a business-level deduplication token shared by all retries of the same operation. Using a single header for both breaks dedup logic during retry if the causation chain diverges. Implement as two separate attributes / headers.

### 2.1 trace_id

Auto-generated by the OTel SDK at the trace root. 128-bit, encoded as 32 lowercase hex characters. Carried in `traceparent`. This is an **internal platform identifier**; do not expose it directly to end users because it reveals internal infrastructure topology. Its role is to be the carrier that links all spans in one distributed trace.

### 2.2 request_id

Human-facing correlation ID. Must appear in:

- Every outbound HTTP response header (`x-request-id`)
- Every log record as a structured field (`http.request.id` span attribute)
- User-visible error banners on the frontend (support ticket anchor)

The frontend generates a UUIDv4 `request_id` and sends it on every outbound request. Middleware sets the span attribute and logs the field. Customer support agents hand users a `request_id` from the error banner; platform engineers query logs to retrieve `trace_id` and pivot to the distributed trace waterfall.

Cross-ref `../incident-forensics.md §MRA` for the support-to-trace lookup flow.

### 2.3 causation_id / idempotency_key

Event-driven and async workflows break the synchronous call chain. A Kafka consumer processing a payment event may run minutes after the HTTP request that produced it. `causation_id` links the new trace back to the originating event chain without forcing a parent-child span relationship.

- HTTP POST: set `Idempotency-Key` request header → record as span attribute
- Kafka producer: write `causation_id` into Kafka message header
- Kafka consumer: extract `causation_id` → create OTel span link (not parent-child) to the originating span

Span links are the correct OTel primitive for async causation. Cross-ref `../signals/traces.md §Messaging patterns`.

### 2.4 Business context via Baggage

Business identifiers (`user.id`, `tenant.id`, `order.id`, `feature.variant`) propagate as W3C Baggage key-value pairs. They flow through every service on the call path and appear as span attributes and log fields, enabling cross-cutting dashboard filters (e.g., "all traces for tenant ACME in the last hour").

Baggage carries PII risk. See Section 4 for trust-boundary filtering rules.

---

## 3. Propagator Matrix

The recommended default for all new deployments is **W3C Trace Context**. At ingress or multi-cloud boundaries, extract all known propagator formats and emit W3C downstream; normalization at the edge prevents split-brain traces inside the cluster.

| Ecosystem | Header(s) | OTel propagator | Normalization notes |
|-----------|-----------|----------------|---------------------|
| **W3C Trace Context** (default) | `traceparent`, `tracestate`, `baggage` | `tracecontext`, `baggage` | L1 Rec 2020-02-06; L2 CR adds random-id flag. Mandatory baseline for all new services. |
| **B3** (Zipkin legacy) | `x-b3-traceid`, `x-b3-spanid`, `x-b3-parentspanid`, `x-b3-sampled` | `b3multi` / `b3` (single-header) | Use multi-header format. Keep as secondary propagator for Zipkin-legacy services; migrate to W3C for new work. |
| **AWS X-Ray** | `X-Amzn-Trace-Id` (`Root=1-<ts>-<hex>;Parent=<id>;Sampled=<0\|1>`) | `awsxray` | AWS X-Ray SDK supports W3C output since 2022. Prefer W3C; use X-Ray propagator only for Lambda or services that require X-Ray console trace linking. |
| **GCP Cloud Trace** | `traceparent` (W3C preferred), legacy `X-Cloud-Trace-Context` | `gcp` propagator | GCP supports both formats. Emit W3C; accept legacy for backward compat. |
| **Azure Monitor** | `traceparent` (W3C preferred), legacy `Request-Id` | W3C preferred | Application Insights SDK v3+ defaults to W3C. Treat `Request-Id` as read-only legacy input only. |
| **Datadog APM** | `x-datadog-trace-id`, `x-datadog-parent-id`, `x-datadog-sampling-priority` | `datadog` propagator | Configure Datadog Agent to accept W3C input; use OTel Datadog exporter for span forwarding. |
| **Cloudflare Workers** | `cf-ray` + W3C `traceparent` | Map `cf-ray` → baggage | `cf-ray` is not a trace ID. Map it to `baggage: cf.ray=<value>` for correlation. Emit W3C `traceparent` on Worker egress calls. |
| **Istio / Envoy** | W3C `traceparent` + Envoy internal `x-request-id`, `x-ot-span-context` (legacy) | W3C (OTLP tracer, Envoy 1.29+) | `x-ot-span-context` is deprecated as of Envoy 1.29. Cross-ref `../layers/mesh.md §Propagator headers`. |
| **Linkerd** | `l5d-ctx-trace`, `l5d-ctx-span`, `l5d-ctx-parent`, `l5d-ctx-deadline` | `linkerd` propagator | Linkerd headers are not W3C-compatible. Translate to W3C at the mesh boundary gateway. Cross-ref `../layers/mesh.md §Cross-mesh compatibility rule`. |

### 3.1 Multi-propagator strategy at ingress

Configure the ingress gateway or API gateway collector with a composite propagator that extracts all known formats. After extraction, emit only W3C downstream:

```
Composite extract order (ingress):
  1. tracecontext  (W3C — primary)
  2. b3multi       (Zipkin legacy)
  3. awsxray       (AWS Lambda / API Gateway)
  4. datadog       (Datadog-instrumented upstream)

Inject order (egress, internal):
  1. tracecontext  (W3C — only format emitted internally)
  2. baggage       (W3C Baggage — always)
```

This pattern ensures that a request arriving from an AWS Lambda (X-Ray header) or a Datadog-instrumented partner API is absorbed into the W3C trace context and propagated uniformly to all internal services.

---

## 4. Baggage Rules and PII Guidance

W3C Baggage propagates to every service on the call path. The W3C Baggage specification §Security states:

> "Application owners should either ensure that no proprietary or confidential information is stored in baggage, or ensure baggage is not present in requests that cross trust boundaries."

The OTel Baggage API spec does not independently repeat this warning; the normative source is the W3C parent specification. Implementers MUST read both.

Cross-ref `../signals/privacy.md §Baggage rules` for the enforcement implementation.

### 4.1 Trust-boundary rule

At every egress trust boundary (API gateway, external webhook, third-party vendor API), apply a baggage filter:

- **Default**: strip all baggage unless the receiving service is on the explicit allowlist.
- **Allowlist**: maintained per-environment; requires security team sign-off to add entries.
- **Internal-to-internal**: baggage passes through without filtering (intra-cluster calls within the same trust zone).

### 4.2 Allowed and prohibited baggage values

| Allowed (safe at any trust level) | Prohibited (must never appear in baggage) |
|-----------------------------------|------------------------------------------|
| `tenant.id` (opaque identifier) | `user.email` |
| `user.tier` (tier name, not PII) | Authentication tokens or session IDs |
| `feature.variant` | Credit card numbers or PAN |
| `deployment.sha` | Passwords or API keys |
| `region.hint` | Any field classified as PII under GDPR / PIPA |

### 4.3 Enforcement point

The mesh ingress gateway is the primary enforcement point. Configure Envoy's Lua filter or an OTel Collector `attributes` processor at the gateway to drop prohibited keys. Cross-ref `../layers/mesh.md §Baggage scrubbing`.

---

## 5. DDD Bounded Context via service.namespace

OpenTelemetry resource attributes `service.name` and `service.namespace` map directly to DDD concepts:

| OTel attribute | DDD concept | Example |
|----------------|-------------|---------|
| `service.namespace` | Bounded context (domain group) | `payments`, `inventory`, `identity` |
| `service.name` | Individual microservice | `checkout-api`, `stock-api`, `auth-service` |

### 5.1 Why service.namespace matters

Without `service.namespace`, a system with 80+ microservices presents a flat list in trace backends and dashboards. Operators cannot distinguish domain-level KPIs from service-level noise.

With `service.namespace`, dashboards can be scoped:

- `service.namespace = payments` → filter all payment-domain spans, compute payment-domain SLI
- `service.namespace = inventory` → separate inventory burn-rate alert from payments alert

### 5.2 Configuration

Set `service.namespace` as an OTel resource attribute at deployment time, not at instrumentation time:

```yaml
# OTel Collector resourcedetection / resource processor
processors:
  resource:
    attributes:
      - key: service.namespace
        value: payments
        action: upsert
      - key: service.name
        value: checkout-api
        action: upsert
```

Alternatively, set via environment variable that the SDK's Environment Resource Detector picks up:

```
OTEL_RESOURCE_ATTRIBUTES=service.namespace=payments,service.name=checkout-api
```

### 5.3 Anti-pattern: flat service registry

One hundred services without namespace grouping causes cognitive overload in the trace backend service map. Namespace assignment is a required step in any multi-service deployment. Assign namespaces aligned with DDD bounded contexts, not with team names or infrastructure regions.

---

## 6. Cross-Cloud Trace Continuity

When a request crosses cloud provider boundaries (e.g., App A on AWS calls App B on GCP calls App C on-premises), trace continuity requires W3C `traceparent` on every hop.

| Provider | W3C support status |
|----------|--------------------|
| AWS X-Ray SDK | W3C output supported since 2022 |
| GCP Cloud Trace | W3C `traceparent` preferred; legacy `X-Cloud-Trace-Context` still accepted |
| Azure Monitor | W3C `traceparent` preferred; Application Insights SDK v3+ default |
| Jaeger (self-managed) | W3C Trace Context since Jaeger v1.35 |

Collector bridge strategy: deploy an OTel Collector at each cloud egress point. The Collector normalizes incoming vendor headers to W3C before forwarding to the next cloud's ingress. This ensures that a trace rooted in AWS survives the GCP boundary and terminates in an on-premises Jaeger backend with an unbroken span chain.

Cross-ref `../transport/collector-topology.md §Multi-cluster/regional` for Collector deployment patterns.

---

## 7. request_id to trace_id Integration

The support escalation flow depends on reliable linkage between the user-visible `request_id` and the internal `trace_id`.

**Flow:**

1. Frontend generates UUIDv4 `request_id` and sends it on every outbound request.
2. Backend middleware extracts `request_id`, sets span attribute `http.request.id`, and writes it as a structured log field.
3. On error, the frontend displays the `request_id` to the user (never `trace_id`; the latter leaks internal topology).
4. Customer support receives the `request_id` from the user's error screenshot.
5. Platform engineer queries the log system: `http.request.id = "<request_id>"` → retrieves `trace_id` from the matching log record.
6. Platform engineer opens the trace waterfall in the trace backend using `trace_id`.

Cross-ref `../incident-forensics.md §MRA` for the full minimum reproducible artifact workflow.

---

## 8. Idempotency and Event-Driven Trace Lineage

Idempotency keys and causation IDs are first-class observability primitives in event-driven architectures.

**HTTP pattern (Stripe-style):**

- Client sends `Idempotency-Key: <uuid>` header on POST.
- Server records the key as span attribute `http.idempotency_key`.
- On duplicate detection, the existing response is returned; the `trace_id` of the original attempt is logged alongside the duplicate attempt's `trace_id`.

**Kafka / SQS pattern:**

- Producer writes `causation_id` (UUIDv4) and original `traceparent` into message headers.
- Consumer extracts both headers before creating the `CONSUMER` span.
- Consumer span links to the producer span via `span.addLink(producerSpanContext)`.
- The link: not parent-child; preserves trace lineage without forcing synchronous ordering assumptions.

**DLQ replay:**

- When replaying from a dead letter queue, re-inject the original `traceparent` and `causation_id` from the failed message headers. Generating a new `trace_id` at replay severs the forensic chain.

Cross-ref `../signals/traces.md §Messaging patterns` for span link implementation details.

---

## 9. Trust Boundary Patterns

| Boundary type | Trace behavior | Baggage behavior |
|---------------|---------------|-----------------|
| Internal service to internal service (same cluster) | Forward `traceparent` unchanged | Forward baggage unchanged |
| Internal service to external API gateway (outbound) | Create new CLIENT span; include `traceparent` only if vendor is on allowlist | Strip all baggage unless vendor is on explicit allowlist |
| B2B webhook (outbound to partner) | Start a new trace; log original `trace_id` as annotation in the outbound span only | Do not propagate baggage |
| Third-party vendor API call | Do not propagate `traceparent` unless vendor explicitly supports it | Do not propagate baggage |
| External partner calling inbound | Extract `traceparent` if present; validate and sanitize | Strip baggage at ingress gateway; reattach only allowlisted keys |

---

## 10. Context Inheritance in Non-Interactive Sessions

CI/CD pipelines, agent workflows, and background jobs need trace context for end-to-end visibility across automated steps.

Use the `TRACEPARENT` environment variable for child process context inheritance. The OTel SDK's Environment Resource Detector reads `TRACEPARENT` and applies it as the parent context for the first span created in that process.

```bash
# CI/CD pipeline: propagate trace context to child build steps
export TRACEPARENT="00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01"
# Child process (e.g., test runner) inherits the parent span context
```

This pattern enables a single trace to span from the CI trigger event through build, test, deploy, and smoke-test verification steps.

---

## 11. Matrix Coverage

These cells from `../matrix.md` are primarily driven by this file:

| Layer | Boundary | Signal | Symbol | Primary artifact |
|-------|----------|--------|--------|-----------------|
| mesh | cross-application | traces | PASS | W3C `traceparent` in mesh proxy; zero-code injection |
| L7-application | cross-application | traces | PASS | OTel SDK W3C propagation; DDD `service.namespace` grouping |
| L7-application | cross-application | privacy | PASS | Baggage trust-boundary filtering at API gateway |
| L7-application | cross-application | audit | PASS | Correlation across audit events via `trace_id` + `request_id` |
| L3-network | cross-application | traces | PARTIAL | L3 packets carry no trace context; tagging at egress only |
| L4-transport | cross-application | traces | PARTIAL | TCP is not trace-native; trace context begins at mesh or L7 |

---

## 12. Anti-Patterns

The following extend `../anti-patterns.md §Cross-application`.

| Anti-pattern | Consequence | Remediation |
|---|---|---|
| Mixed propagators without normalization at ingress | Broken traces at mesh or cloud boundary; spans appear as disconnected root spans | Apply composite propagator at ingress gateway; normalize to W3C downstream |
| Baggage carrying PII crossing a trust boundary | W3C Baggage spec §Security violation; PII exposed to untrusted downstream services | Strip baggage at egress gateway; use allowlist; never put PII in baggage |
| `request_id` not exposed to frontend | Customer support cannot correlate user-reported errors to backend traces | Return `x-request-id` in every response; display in error banners |
| `service.namespace` not set | 100+ services appear as a flat unordered list; domain-level KPIs are impossible | Set `service.namespace` per DDD bounded context in OTel resource attributes |
| B3-only propagation without W3C fallback | Traces break when crossing any non-Zipkin boundary (GCP, AWS, Azure) | Add `tracecontext` as primary propagator; keep `b3multi` as secondary for legacy |
| New `trace_id` generated on DLQ replay | Original failure span is orphaned; forensic chain from root cause to replay is severed | Re-inject original `traceparent` and `causation_id` from failed message headers |

---

## References



Internal cross-references:

- `../standards.md`: normative semconv stability tiers and W3C spec versions
- `../matrix.md`: full 112-cell coverage map (cross-application boundary row)
- `../layers/mesh.md`: propagator headers per mesh; baggage scrubbing at ingress gateway
- `../signals/traces.md`: OTel span data model, messaging span links, baggage security
- `../signals/privacy.md`: PII classification, baggage PII redaction rules
- `../signals/audit.md`: audit event correlation across application boundaries
- `../transport/collector-topology.md`: multi-cluster and multi-region Collector bridge patterns
- `../incident-forensics.md`: request_id to trace_id lookup flow (MRA section)

### Primary sources

- W3C Trace Context Level 1 Recommendation: <https://www.w3.org/TR/trace-context/>
- W3C Trace Context Level 2 Candidate Recommendation: <https://www.w3.org/TR/trace-context-2/>
- W3C Baggage Recommendation: <https://www.w3.org/TR/baggage/>
- OTel Baggage API specification: <https://opentelemetry.io/docs/specs/otel/baggage/api/>
- AWS X-Ray trace header: <https://docs.aws.amazon.com/xray/latest/devguide/xray-concepts.html#xray-concepts-tracingheader>
- GCP Cloud Trace context propagation: <https://cloud.google.com/trace/docs/trace-context>
- Datadog trace context propagation: <https://docs.datadoghq.com/tracing/trace_collection/trace_context_propagation/>
- Linkerd distributed tracing: <https://linkerd.io/2.15/features/distributed-tracing/>
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 322kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall