Skip to content

opensite-ai/opensite-skills / aws-observability

aws-observability

Unverified

Builds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md28.3 KB
assets/cloudwatch/alarm-template.tsscript3.9 KB
assets/cloudwatch/otel-config.yamlconfig1.9 KB
references/cloudwatch-omni/agent-evaluation.mdmarkdown98.9 KB
references/cloudwatch-omni/alerts.mdmarkdown103.3 KB
references/cloudwatch-omni/concepts.mdmarkdown28.8 KB
references/cloudwatch-omni/context-graph.mdmarkdown29.2 KB
references/cloudwatch-omni/dashboards.mdmarkdown75.5 KB
references/cloudwatch-omni/programmatic-access.mdmarkdown8.6 KB
references/cloudwatch-omni/query/promql-metrics.mdmarkdown63.9 KB
references/cloudwatch-omni/query/sql-logs-traces.mdmarkdown43.0 KB
references/cloudwatch-omni/query/views.mdmarkdown12.7 KB
references/cloudwatch/alarms.mdmarkdown11.5 KB
references/cloudwatch/cloudtrail.mdmarkdown5.3 KB
references/cloudwatch/dashboards.mdmarkdown5.8 KB
references/cloudwatch/dynamic-instrumentation.mdmarkdown39.4 KB
references/cloudwatch/dynamic-instrumentation/breakpoint-creation.mdmarkdown16.3 KB
references/cloudwatch/dynamic-instrumentation/call-tree-and-directions.mdmarkdown5.7 KB
references/cloudwatch/dynamic-instrumentation/snapshot-parsing.mdmarkdown4.2 KB
references/cloudwatch/log-insights.mdmarkdown7.0 KB
references/cloudwatch/metrics.mdmarkdown7.4 KB
references/cloudwatch/synthetics.mdmarkdown6.5 KB
references/cloudwatch/tracing.mdmarkdown9.4 KB
references/cloudwatch/troubleshooting.mdmarkdown7.5 KB
scripts/cloudwatch-omni/capture_dataset_from_traces.pyscript19.2 KB
scripts/cloudwatch-omni/evaluate_traces.pyscript41.8 KB
scripts/cloudwatch-omni/test_evaluate_traces.pyscript12.0 KB
scripts/cloudwatch/di_app_signals_client.pyscript3.3 KB
scripts/cloudwatch/di_capture.pyscript8.5 KB
scripts/cloudwatch/di_constants.pyscript560 B
scripts/cloudwatch/di_crud_rendering.pyscript13.1 KB
scripts/cloudwatch/di_crud_tools.pyscript28.8 KB
scripts/cloudwatch/di_error_translation.pyscript5.6 KB
scripts/cloudwatch/di_formatting.pyscript992 B
scripts/cloudwatch/di_gateway.pyscript7.0 KB
scripts/cloudwatch/di_instrumentation.pyscript13.0 KB
scripts/cloudwatch/di_location.pyscript15.3 KB
scripts/cloudwatch/di_logs_client.pyscript4.2 KB
scripts/cloudwatch/di_region.pyscript2.0 KB
scripts/cloudwatch/di_result.pyscript2.0 KB
scripts/cloudwatch/di_session.pyscript2.0 KB
scripts/cloudwatch/di_snapshot_parsing.pyscript11.4 KB
scripts/cloudwatch/di_snapshot_queries.pyscript2.4 KB
scripts/cloudwatch/di_snapshot_rendering.pyscript12.5 KB
scripts/cloudwatch/di_snapshot_tools.pyscript12.0 KB
scripts/cloudwatch/di_snapshots.pyscript12.8 KB
scripts/cloudwatch/di_status_assessment.pyscript4.7 KB
scripts/cloudwatch/di_status_rendering.pyscript11.8 KB
scripts/cloudwatch/di_status_tools.pyscript13.0 KB
scripts/cloudwatch/di_validation.pyscript11.2 KB

references/cloudwatch/metrics.md

View on GitHub
# CloudWatch Custom Metrics

Publishing, querying, and managing custom metrics — EMF, PutMetricData, metric filters, and retention.

## Contents
- [EMF vs PutMetricData](#emf-vs-putmetricdata)
- [Embedded Metric Format (EMF)](#embedded-metric-format-emf)
- [PutMetricData API](#putmetricdata-api)
- [Metric filters](#metric-filters)
- [Metric retention](#metric-retention)
- [Dimension design](#dimension-design)
- [Metric math](#metric-math)
- [EMF constraints](#emf-constraints)

---

## EMF vs PutMetricData

| Criteria | EMF | PutMetricData |
|----------|-----|---------------|
| Latency impact | None (async via logs) | Synchronous API call |
| Log correlation | Yes — Metrics + logs in same event | No — Separate |
| Max metrics per call | 100 per MetricDirective | 1,000 MetricDatum per request |
| High-resolution | Yes — StorageResolution=1 | Yes — StorageResolution=1 |
| Cost model | Log ingestion pricing | Per-metric API charges |
| Best for | **Lambda, containers** | Batch jobs, custom agents |

**Default recommendation**: Use EMF for Lambda and containerized workloads. Use PutMetricData for batch jobs or when you need synchronous confirmation.

---

## Embedded Metric Format (EMF)

### JSON structure
```json
{
  "_aws": {
    "Timestamp": 1574109732004,
    "CloudWatchMetrics": [{
      "Namespace": "MyService",
      "Dimensions": [["ServiceName", "Environment"]],
      "Metrics": [
        { "Name": "Latency", "Unit": "Milliseconds", "StorageResolution": 60 },
        { "Name": "RequestCount", "Unit": "Count" }
      ]
    }]
  },
  "ServiceName": "OrderService",
  "Environment": "Production",
  "Latency": 100,
  "RequestCount": 1,
  "RequestId": "abc-123"
}
```

### EMF limits
- Max **100 metrics** per MetricDirective
- Max **30 dimensions** per DimensionSet (may be empty)
- Dimension value: max **1024 characters**, must be string
- Metric value: must be numeric or array of numerics (max **100 values**)
- Max log event size: **1 MB**
- Namespace: 1–1024 characters, should not start with `AWS/`
- `Timestamp` in `_aws` is **required** per the EMF spec and JSON schema (milliseconds since epoch). In practice, if omitted, CloudWatch uses the log event's ingestion time — but explicitly setting it is recommended to avoid clock-skew issues.

### EMF libraries

For Lambda/containers, use a library that handles EMF serialization (e.g., Lambda Powertools Metrics, `aws-embedded-metrics`). These libraries manage the `_aws` metadata block, dimension limits, and metric flushing automatically.

---

## PutMetricData API

### Limits
- **500 TPS** per account per region (adjustable via Service Quotas) — NOT 150 TPS
- Up to **1,000 MetricDatum** items per request
- Up to **150 values** per MetricDatum (for percentile statistics support)
- Max **30 dimensions** per metric
- Metric name: max 255 characters
- Namespace: max 255 characters, should not start with `AWS/`

### StatisticSets (batch optimization)
Instead of publishing individual data points, aggregate into StatisticSets:
```json
{
  "MetricName": "Latency",
  "StatisticValues": {
    "SampleCount": 100,
    "Sum": 5000,
    "Minimum": 10,
    "Maximum": 200
  },
  "Unit": "Milliseconds"
}
```
Reduces API calls and cost.

---

## Metric filters

Extract metrics from log events automatically.

- **Max 100 metric filters per log group**
- Filter pattern: space-delimited terms or JSON property matching
- PutMetricFilter API: 5 TPS
- Metric filter → CloudWatch metric → alarm pipeline is the standard log-to-alert pattern

### Example: count 5xx errors from access logs
```
{ $.statusCode >= 500 }
```
Publishes a metric with value 1 for each matching log event.

---

## Metric retention

### Automatic aggregation cascade

| Data point period | Available for | Then aggregated to |
|-------------------|---------------|--------------------|
| < 60s (high-res) | **3 hours** | 1-minute |
| 60s (1 min) | **15 days** | 5-minute |
| 300s (5 min) | **63 days** | 1-hour |
| 3600s (1 hr) | **455 days (15 months)** | — |

**Key insight**: You cannot query 1-minute data from 2 months ago. It has been automatically aggregated to 5-minute resolution. High-resolution (1-second) data is only available for 3 hours.

**OTel metrics**: Only **30 days** retention (public preview) — significantly shorter than traditional CloudWatch metrics (15 months).

### Metric expiry
- Metrics with no new data for **15 months** expire
- Metrics with no data for **2 weeks** are not listed by ListMetrics (but still exist)

---

## Dimension design

**Note**: Each unique dimension combination = separate metric = separate cost.

### Anti-patterns
- Do not use `requestId`, `userId`, `sessionId` as dimensions — creates millions of metrics
- Do not publish `{InstanceId, InstanceType}` and expect to query by `InstanceId` alone — must publish both combinations separately
- Do not use inconsistent units — metrics with different units are separate data streams

### Best practices
- Use low-cardinality dimensions: `ServiceName`, `Environment`, `Operation`, `StatusCode`
- Use the `SEARCH` function for cross-dimension queries
- Always specify units consistently
- Audit custom metrics regularly — remove unused ones

---

## Metric math

Combine metrics using expressions in alarms and dashboards.

### Functions
`SUM`, `AVG`, `MIN`, `MAX`, `STDDEV`, `PERIOD`, `SEARCH`, `IF`, `FILL`, `ANOMALY_DETECTION_BAND`

### Error rate pattern
```
errors * 100 / invocations
```

### SEARCH expression (dynamic metrics)
```
SEARCH('{AWS/Lambda,FunctionName} MetricName="Errors"', 'Sum', 300)
```
Automatically includes new functions matching the pattern — useful in dashboards and graphs (SEARCH cannot be used in alarms).

### Limits
- Max **10 metrics** in a metric math alarm expression
- Use Metrics Insights queries for more (max 10,000 metrics, 500 time series returned)
- Metrics Insights alarm data window: **3 hours** only
- Max **500 metrics+expressions** per dashboard graph

### Metric math in alarms — constraints

- **`FILL` can permanently stick an alarm**: If a metric is published with slight delay, `FILL` replaces the missing latest point with the fill value, keeping the alarm in a fixed state. Use M-of-N alarms instead.
- **`RATE` on sparse metrics is unpredictable**: The evaluation range varies, causing inconsistent rate calculations. Avoid `RATE` in alarms on metrics that don't publish every period.
- **Anomaly detection restrictions** (non-exhaustive): Cannot use more than one `ANOMALY_DETECTION_BAND` per expression, cannot combine with `METRICS()` or `SEARCH`, cannot use high-resolution metrics. See [CloudWatch metric math docs](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/using-metric-math.html) for full list.

---

## EMF constraints

- **Flush interval affects alarms**: Flush EMF logs to CloudWatch at ≤5 second intervals. Longer intervals cause alarms to evaluate partial or missing data. In Lambda (where flush is automatic), use M-of-N alarms to compensate.
- **Monitor EMF parsing failures**: `AWS/Logs` namespace publishes `EMFValidationErrors` and `EMFParsingErrors` metrics. Check these if metrics aren't appearing.
- **Target values cannot be nested**: `"A.a"` matches `{ "A.a": 1 }`, NOT `{ "A": { "a": 1 } }`. Metric and dimension values must be on the root node.
- **Multiple DimensionSets multiply metrics**: `Dimensions: [["Service"], ["Service", "Operation"]]` creates 2 metrics per data point, not 1. Libraries like Powertools do this by default.
- **Dimension key max 250 chars** (per EMF schema); dimension value max 1024 chars.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall