Skip to content

opensite-ai/opensite-skills / aws-observability

aws-observability

Unverified

Builds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md28.3 KB
assets/cloudwatch/alarm-template.tsscript3.9 KB
assets/cloudwatch/otel-config.yamlconfig1.9 KB
references/cloudwatch-omni/agent-evaluation.mdmarkdown98.9 KB
references/cloudwatch-omni/alerts.mdmarkdown103.3 KB
references/cloudwatch-omni/concepts.mdmarkdown28.8 KB
references/cloudwatch-omni/context-graph.mdmarkdown29.2 KB
references/cloudwatch-omni/dashboards.mdmarkdown75.5 KB
references/cloudwatch-omni/programmatic-access.mdmarkdown8.6 KB
references/cloudwatch-omni/query/promql-metrics.mdmarkdown63.9 KB
references/cloudwatch-omni/query/sql-logs-traces.mdmarkdown43.0 KB
references/cloudwatch-omni/query/views.mdmarkdown12.7 KB
references/cloudwatch/alarms.mdmarkdown11.5 KB
references/cloudwatch/cloudtrail.mdmarkdown5.3 KB
references/cloudwatch/dashboards.mdmarkdown5.8 KB
references/cloudwatch/dynamic-instrumentation.mdmarkdown39.4 KB
references/cloudwatch/dynamic-instrumentation/breakpoint-creation.mdmarkdown16.3 KB
references/cloudwatch/dynamic-instrumentation/call-tree-and-directions.mdmarkdown5.7 KB
references/cloudwatch/dynamic-instrumentation/snapshot-parsing.mdmarkdown4.2 KB
references/cloudwatch/log-insights.mdmarkdown7.0 KB
references/cloudwatch/metrics.mdmarkdown7.4 KB
references/cloudwatch/synthetics.mdmarkdown6.5 KB
references/cloudwatch/tracing.mdmarkdown9.4 KB
references/cloudwatch/troubleshooting.mdmarkdown7.5 KB
scripts/cloudwatch-omni/capture_dataset_from_traces.pyscript19.2 KB
scripts/cloudwatch-omni/evaluate_traces.pyscript41.8 KB
scripts/cloudwatch-omni/test_evaluate_traces.pyscript12.0 KB
scripts/cloudwatch/di_app_signals_client.pyscript3.3 KB
scripts/cloudwatch/di_capture.pyscript8.5 KB
scripts/cloudwatch/di_constants.pyscript560 B
scripts/cloudwatch/di_crud_rendering.pyscript13.1 KB
scripts/cloudwatch/di_crud_tools.pyscript28.8 KB
scripts/cloudwatch/di_error_translation.pyscript5.6 KB
scripts/cloudwatch/di_formatting.pyscript992 B
scripts/cloudwatch/di_gateway.pyscript7.0 KB
scripts/cloudwatch/di_instrumentation.pyscript13.0 KB
scripts/cloudwatch/di_location.pyscript15.3 KB
scripts/cloudwatch/di_logs_client.pyscript4.2 KB
scripts/cloudwatch/di_region.pyscript2.0 KB
scripts/cloudwatch/di_result.pyscript2.0 KB
scripts/cloudwatch/di_session.pyscript2.0 KB
scripts/cloudwatch/di_snapshot_parsing.pyscript11.4 KB
scripts/cloudwatch/di_snapshot_queries.pyscript2.4 KB
scripts/cloudwatch/di_snapshot_rendering.pyscript12.5 KB
scripts/cloudwatch/di_snapshot_tools.pyscript12.0 KB
scripts/cloudwatch/di_snapshots.pyscript12.8 KB
scripts/cloudwatch/di_status_assessment.pyscript4.7 KB
scripts/cloudwatch/di_status_rendering.pyscript11.8 KB
scripts/cloudwatch/di_status_tools.pyscript13.0 KB
scripts/cloudwatch/di_validation.pyscript11.2 KB

references/cloudwatch/tracing.md

View on GitHub
# Distributed Tracing: X-Ray and ADOT

X-Ray SDK is in maintenance mode. Use ADOT (OpenTelemetry) for all new projects.

## Contents
- [ADOT vs X-Ray SDK](#adot-vs-x-ray-sdk)
- [Trace structure](#trace-structure)
- [Annotations vs metadata](#annotations-vs-metadata)
- [Sampling rules](#sampling-rules)
- [ADOT collector configuration](#adot-collector-configuration)
- [Instrumentation patterns](#instrumentation-patterns)
- [Migration constraints](#migration-constraints-x-ray-sdk--otel)
- [Common mistakes](#common-mistakes)

---

## ADOT vs X-Ray SDK

| Criteria | X-Ray SDK | ADOT (OpenTelemetry) |
|----------|----------|---------------------|
| Status | **Maintenance mode** | Actively developed |
| Multi-backend | X-Ray only | CloudWatch, X-Ray, Prometheus, OpenSearch |
| Auto-instrumentation | Limited | Java, Python (compute); Node.js (Lambda layer only) |
| Vendor lock-in | AWS-specific | Vendor-neutral (OTel standard) |
| Lambda support | Built-in daemon | Lambda layer (auto-instrumentation) |
| **Recommendation** | **Legacy apps only** | **All new projects** |

**Migration path**: AWS provides migration guides from X-Ray SDK to OpenTelemetry SDK. The CloudWatch agent now also supports sending traces to X-Ray — no separate daemon needed.

---

## Trace structure

- **Trace** — collection of all segments from a single request, identified by trace ID
- **Segment** — JSON document with a **64 KB** documented limit representing work done by a service. Do not exceed this; behavior above 64 KB is undocumented and may change.
- **Subsegment** — granular detail within a segment (downstream calls, custom code blocks)
- **Inferred segment** — generated by X-Ray from subsegments for uninstrumented downstream services

### Trace ID format
```
X-Amzn-Trace-Id: Root=1-58406520-a006649127e371903a2de979;Parent=53995c3f42cd8ad8;Sampled=1
```
Format: `1-{8 hex epoch}-{24 hex unique}`. W3C trace IDs are supported (reformatted).

### Retention
- Trace data: **30 days** (not configurable)
- Service graph: **30 days**

---

## Annotations vs metadata

| Feature | Annotations | Metadata |
|---------|------------|----------|
| **Indexed** | Yes — Searchable with filter expressions | No — Not indexed |
| **Value types** | String, Number, Boolean only | Any type (objects, arrays) |
| **Limit** | **50 indexed per trace** (API accepts more, but only 50 are searchable) | No limit (within segment size) |
| **Key format** | Alphanumeric + underscore only | Any key (`AWS.` prefix reserved) |
| **Use case** | Filtering/grouping traces | Storing debug data |

**Rule of thumb**: If you need to search for it → annotation. If you just need to store it → metadata.

**WARNING**: 50 annotations per trace is a hard limit. Plan your annotation schema carefully.

---

## Sampling rules

### Default rule
- **Reservoir**: 1 request per second (shared across all instances)
- **Rate**: 5% of additional requests
- Conservative default to control costs

### Rule evaluation
- Rules evaluated in ascending **priority** order (1–9999, lower = higher priority)
- Default rule priority = 10000 (always last)
- First matching rule wins

### Rule parameters
| Parameter | Description |
|-----------|-------------|
| Priority | 1–9999 (lower = higher priority) |
| Reservoir | Fixed traces/second before applying rate |
| Rate | Percentage of additional requests (0–100 in console, 0.0–1.0 in API/JSON) |
| Service name | Wildcards `*` and `?` supported |
| Service type | e.g., `AWS::EC2::Instance`, `AWS::Lambda::Function` |
| HTTP method | GET, POST, etc. |
| URL path | Path portion of URL |

### Parent-based sampling (critical concept)
Sampling decision is made **once** by the root service. Downstream services honor the upstream decision regardless of their own rules. Custom rules only apply where no sampling decision exists yet.

### Adaptive sampling (newer)
- `SamplingRateBoost` — auto-increases rate during anomalies
- `MaxRate` — ceiling for boosted rate
- `CooldownWindowMinutes` — prevents continuous boosts (recommended when SamplingRateBoost is configured)

---

## ADOT collector configuration

### Architecture
```
[Receivers] → [Processors] → [Exporters]
```

### CloudWatch + X-Ray pipeline

> The `0.0.0.0` receivers below listen on every interface with no TLS and no authentication. Bind
> them to `127.0.0.1` when the senders are on the same host; otherwise it is recommended to restrict
> the OTLP `4317` / `4318` ports via security groups or host firewall and to avoid co-locating
> untrusted workloads. This guide does not apply those controls, so assess and configure them for
> your environment.

```yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 30s
    send_batch_size: 8192

exporters:
  awsxray:
    region: us-east-1
  awsemf:
    namespace: MyApplication
    region: us-east-1

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsxray]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [awsemf]
```

### EKS DaemonSet deployment
```yaml
resources:
  limits:
    memory: 200Mi
  requests:
    cpu: 250m
    memory: 100Mi
```

### Cardinality prevention (three-layer defense)
1. **OTel SDK level**: Don't emit high-cardinality attributes (ContainerID, CustomerID, RequestID)
2. **ADOT Collector level**: Use Filter Processor to drop metrics by name/attribute
3. **Backend level**: Use backend-specific dimension filtering (CloudWatch: `dimension_rollup_option` + `metric_declarations`; Prometheus: `metric_relabel_configs`)

Filter as early as possible in the pipeline to reduce cost and cardinality.

---

## Instrumentation patterns

### Lambda: enable active tracing (CDK)
```typescript
import { Tracing } from 'aws-cdk-lib/aws-lambda';

const fn = new lambda.Function(this, 'MyFunction', {
  runtime: lambda.Runtime.NODEJS_20_X,
  handler: 'index.handler',
  code: lambda.Code.fromAsset('lambda'),
  tracing: Tracing.ACTIVE,
});
```

### API Gateway: enable tracing
```typescript
const api = new apigateway.RestApi(this, 'MyApi', {
  deployOptions: {
    tracingEnabled: true,
  },
});
```
Or via CLI: `aws apigateway update-stage --rest-api-id <id> --stage-name prod --patch-operations op=replace,path=/tracingEnabled,value=true`

### Trace-log correlation
Inject trace ID into application logs for cross-pillar correlation:
```python
import logging
from opentelemetry import trace

ctx = trace.get_current_span().get_span_context()
trace_id = format(ctx.trace_id, '032x')
logging.info("Processing request", extra={"trace_id": trace_id})
```

---

## Migration constraints (X-Ray SDK → OTel)

### Annotations require explicit opt-in
In OTel, all span attributes become X-Ray **metadata** by default. To make an attribute a searchable X-Ray annotation, add its key to the `aws.xray.annotations` list:
```python
span.set_attribute("aws.xray.annotations", ["order_id", "customer_tier"])
span.set_attribute("order_id", "12345")
```
Without this, you lose all annotation-based filtering after migration.

### Centralized sampling requires a proxy
The ADOT collector config must include the `awsproxy` extension (or use the CloudWatch agent as a proxy) for X-Ray centralized sampling rules to work. Without a proxy, the SDK falls back to a default local rule (1 req/sec + 5%):
```yaml
extensions:
  awsproxy:
    endpoint: 127.0.0.1:2000
service:
  extensions: [awsproxy]
```
SDK env vars: `OTEL_TRACES_SAMPLER=xray` and `OTEL_TRACES_SAMPLER_ARG=endpoint=http://localhost:2000`

Centralized sampling language support: Java, .NET, Python, Node.js (ADOT). Vanilla OTel SDK: Java, .NET, Go.

### Mixed propagation during incremental migration
OTel defaults to W3C Trace Context; X-Ray SDK uses X-Ray trace header. During migration, configure both:
```
OTEL_PROPAGATORS=xray,tracecontext
```
Without this, traces break at service boundaries between old and new instrumentation.

### Port conflict: stop X-Ray daemon before starting ADOT
Both use port 2000. Running both simultaneously causes silent data loss.

### Lambda ADOT layer adds cold start latency
ADOT Lambda layers increase memory usage and cold start time. For latency-sensitive functions where you don't need OTel's multi-backend capabilities, X-Ray SDK may still be preferable.

### W3C trace ID version requirement
ADOT Collector 0.34.0+ (X-Ray Exporter 0.86.0+) is required to accept W3C-format trace IDs. Older versions silently reject them.

---

## Common mistakes

1. **Using X-Ray SDK for new projects** — Maintenance mode. Use ADOT/OpenTelemetry.

2. **Storing searchable data as metadata** — Metadata is NOT indexed. Use annotations for data you need to filter by.

3. **Exceeding 50 annotations per trace** — Hard limit. Plan your annotation schema.

4. **Not stripping X-Amzn-Trace-Id from untrusted requests** — Users can inject trace IDs or sampling decisions.

5. **Default sampling for all services** — 1 req/sec + 5% is too conservative for low-traffic services (may miss issues) and too aggressive for high-traffic (unnecessary cost). Tune per service.

6. **StepFunctions tracing overrides Lambda** — When StepFunction tracing is enabled, downstream Lambda tracing is always enabled regardless of Lambda's own config.

7. **Cross-account tracing** — Trace IDs propagate naturally across accounts, but unified cross-account viewing requires CloudWatch Observability Access Manager (OAM) setup with monitoring/source account links.

rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall