opensite-ai/opensite-skills / aws-observability
aws-observability
UnverifiedBuilds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.
references/cloudwatch/tracing.md
View on GitHub# Distributed Tracing: X-Ray and ADOT
X-Ray SDK is in maintenance mode. Use ADOT (OpenTelemetry) for all new projects.
## Contents
- [ADOT vs X-Ray SDK](#adot-vs-x-ray-sdk)
- [Trace structure](#trace-structure)
- [Annotations vs metadata](#annotations-vs-metadata)
- [Sampling rules](#sampling-rules)
- [ADOT collector configuration](#adot-collector-configuration)
- [Instrumentation patterns](#instrumentation-patterns)
- [Migration constraints](#migration-constraints-x-ray-sdk--otel)
- [Common mistakes](#common-mistakes)
---
## ADOT vs X-Ray SDK
| Criteria | X-Ray SDK | ADOT (OpenTelemetry) |
|----------|----------|---------------------|
| Status | **Maintenance mode** | Actively developed |
| Multi-backend | X-Ray only | CloudWatch, X-Ray, Prometheus, OpenSearch |
| Auto-instrumentation | Limited | Java, Python (compute); Node.js (Lambda layer only) |
| Vendor lock-in | AWS-specific | Vendor-neutral (OTel standard) |
| Lambda support | Built-in daemon | Lambda layer (auto-instrumentation) |
| **Recommendation** | **Legacy apps only** | **All new projects** |
**Migration path**: AWS provides migration guides from X-Ray SDK to OpenTelemetry SDK. The CloudWatch agent now also supports sending traces to X-Ray — no separate daemon needed.
---
## Trace structure
- **Trace** — collection of all segments from a single request, identified by trace ID
- **Segment** — JSON document with a **64 KB** documented limit representing work done by a service. Do not exceed this; behavior above 64 KB is undocumented and may change.
- **Subsegment** — granular detail within a segment (downstream calls, custom code blocks)
- **Inferred segment** — generated by X-Ray from subsegments for uninstrumented downstream services
### Trace ID format
```
X-Amzn-Trace-Id: Root=1-58406520-a006649127e371903a2de979;Parent=53995c3f42cd8ad8;Sampled=1
```
Format: `1-{8 hex epoch}-{24 hex unique}`. W3C trace IDs are supported (reformatted).
### Retention
- Trace data: **30 days** (not configurable)
- Service graph: **30 days**
---
## Annotations vs metadata
| Feature | Annotations | Metadata |
|---------|------------|----------|
| **Indexed** | Yes — Searchable with filter expressions | No — Not indexed |
| **Value types** | String, Number, Boolean only | Any type (objects, arrays) |
| **Limit** | **50 indexed per trace** (API accepts more, but only 50 are searchable) | No limit (within segment size) |
| **Key format** | Alphanumeric + underscore only | Any key (`AWS.` prefix reserved) |
| **Use case** | Filtering/grouping traces | Storing debug data |
**Rule of thumb**: If you need to search for it → annotation. If you just need to store it → metadata.
**WARNING**: 50 annotations per trace is a hard limit. Plan your annotation schema carefully.
---
## Sampling rules
### Default rule
- **Reservoir**: 1 request per second (shared across all instances)
- **Rate**: 5% of additional requests
- Conservative default to control costs
### Rule evaluation
- Rules evaluated in ascending **priority** order (1–9999, lower = higher priority)
- Default rule priority = 10000 (always last)
- First matching rule wins
### Rule parameters
| Parameter | Description |
|-----------|-------------|
| Priority | 1–9999 (lower = higher priority) |
| Reservoir | Fixed traces/second before applying rate |
| Rate | Percentage of additional requests (0–100 in console, 0.0–1.0 in API/JSON) |
| Service name | Wildcards `*` and `?` supported |
| Service type | e.g., `AWS::EC2::Instance`, `AWS::Lambda::Function` |
| HTTP method | GET, POST, etc. |
| URL path | Path portion of URL |
### Parent-based sampling (critical concept)
Sampling decision is made **once** by the root service. Downstream services honor the upstream decision regardless of their own rules. Custom rules only apply where no sampling decision exists yet.
### Adaptive sampling (newer)
- `SamplingRateBoost` — auto-increases rate during anomalies
- `MaxRate` — ceiling for boosted rate
- `CooldownWindowMinutes` — prevents continuous boosts (recommended when SamplingRateBoost is configured)
---
## ADOT collector configuration
### Architecture
```
[Receivers] → [Processors] → [Exporters]
```
### CloudWatch + X-Ray pipeline
> The `0.0.0.0` receivers below listen on every interface with no TLS and no authentication. Bind
> them to `127.0.0.1` when the senders are on the same host; otherwise it is recommended to restrict
> the OTLP `4317` / `4318` ports via security groups or host firewall and to avoid co-locating
> untrusted workloads. This guide does not apply those controls, so assess and configure them for
> your environment.
```yaml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 30s
send_batch_size: 8192
exporters:
awsxray:
region: us-east-1
awsemf:
namespace: MyApplication
region: us-east-1
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [awsxray]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [awsemf]
```
### EKS DaemonSet deployment
```yaml
resources:
limits:
memory: 200Mi
requests:
cpu: 250m
memory: 100Mi
```
### Cardinality prevention (three-layer defense)
1. **OTel SDK level**: Don't emit high-cardinality attributes (ContainerID, CustomerID, RequestID)
2. **ADOT Collector level**: Use Filter Processor to drop metrics by name/attribute
3. **Backend level**: Use backend-specific dimension filtering (CloudWatch: `dimension_rollup_option` + `metric_declarations`; Prometheus: `metric_relabel_configs`)
Filter as early as possible in the pipeline to reduce cost and cardinality.
---
## Instrumentation patterns
### Lambda: enable active tracing (CDK)
```typescript
import { Tracing } from 'aws-cdk-lib/aws-lambda';
const fn = new lambda.Function(this, 'MyFunction', {
runtime: lambda.Runtime.NODEJS_20_X,
handler: 'index.handler',
code: lambda.Code.fromAsset('lambda'),
tracing: Tracing.ACTIVE,
});
```
### API Gateway: enable tracing
```typescript
const api = new apigateway.RestApi(this, 'MyApi', {
deployOptions: {
tracingEnabled: true,
},
});
```
Or via CLI: `aws apigateway update-stage --rest-api-id <id> --stage-name prod --patch-operations op=replace,path=/tracingEnabled,value=true`
### Trace-log correlation
Inject trace ID into application logs for cross-pillar correlation:
```python
import logging
from opentelemetry import trace
ctx = trace.get_current_span().get_span_context()
trace_id = format(ctx.trace_id, '032x')
logging.info("Processing request", extra={"trace_id": trace_id})
```
---
## Migration constraints (X-Ray SDK → OTel)
### Annotations require explicit opt-in
In OTel, all span attributes become X-Ray **metadata** by default. To make an attribute a searchable X-Ray annotation, add its key to the `aws.xray.annotations` list:
```python
span.set_attribute("aws.xray.annotations", ["order_id", "customer_tier"])
span.set_attribute("order_id", "12345")
```
Without this, you lose all annotation-based filtering after migration.
### Centralized sampling requires a proxy
The ADOT collector config must include the `awsproxy` extension (or use the CloudWatch agent as a proxy) for X-Ray centralized sampling rules to work. Without a proxy, the SDK falls back to a default local rule (1 req/sec + 5%):
```yaml
extensions:
awsproxy:
endpoint: 127.0.0.1:2000
service:
extensions: [awsproxy]
```
SDK env vars: `OTEL_TRACES_SAMPLER=xray` and `OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:2000`
Centralized sampling language support: Java, .NET, Python, Node.js (ADOT). Vanilla OTel SDK: Java, .NET, Go.
### Mixed propagation during incremental migration
OTel defaults to W3C Trace Context; X-Ray SDK uses X-Ray trace header. During migration, configure both:
```
OTEL_PROPAGATORS=xray,tracecontext
```
Without this, traces break at service boundaries between old and new instrumentation.
### Port conflict: stop X-Ray daemon before starting ADOT
Both use port 2000. Running both simultaneously causes silent data loss.
### Lambda ADOT layer adds cold start latency
ADOT Lambda layers increase memory usage and cold start time. For latency-sensitive functions where you don't need OTel's multi-backend capabilities, X-Ray SDK may still be preferable.
### W3C trace ID version requirement
ADOT Collector 0.34.0+ (X-Ray Exporter 0.86.0+) is required to accept W3C-format trace IDs. Older versions silently reject them.
---
## Common mistakes
1. **Using X-Ray SDK for new projects** — Maintenance mode. Use ADOT/OpenTelemetry.
2. **Storing searchable data as metadata** — Metadata is NOT indexed. Use annotations for data you need to filter by.
3. **Exceeding 50 annotations per trace** — Hard limit. Plan your annotation schema.
4. **Not stripping X-Amzn-Trace-Id from untrusted requests** — Users can inject trace IDs or sampling decisions.
5. **Default sampling for all services** — 1 req/sec + 5% is too conservative for low-traffic services (may miss issues) and too aggressive for high-traffic (unnecessary cost). Tune per service.
6. **StepFunctions tracing overrides Lambda** — When StepFunction tracing is enabled, downstream Lambda tracing is always enabled regardless of Lambda's own config.
7. **Cross-account tracing** — Trace IDs propagate naturally across accounts, but unified cross-account viewing requires CloudWatch Observability Access Manager (OAM) setup with monitoring/source account links.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |