opensite-ai/opensite-skills / aws-observability
aws-observability
UnverifiedBuilds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.
references/cloudwatch/dynamic-instrumentation/snapshot-parsing.md
View on GitHub# Snapshot retrieval and parsing
Snapshot data captured by a breakpoint lives in CloudWatch Logs
(`/aws/service-events/{service}`). Two host scripts wrap the Logs Insights queries; this file
is the recipe for analyzing what they return.
> **Reminder:** the snapshot log group (`/aws/service-events/{service}`) **must be encrypted
> at rest** with a KMS CMK (`aws logs associate-kms-key`) before capturing — captured snapshots
> may contain credentials, PII, or secrets. See Security Considerations in
> `dynamic-instrumentation.md`.
## Retrieval commands
Both require a host with `python3` + `boto3`. Region resolves as `--region` flag > `AWS_REGION` >
`AWS_DEFAULT_REGION` > `us-east-1` default (the same precedence as `di_instrumentation.py`) — it
MUST be the same region the breakpoint was created in, or searches return
empty even when the breakpoint is ACTIVE. Pass arguments via `--json-file` (or `--json -` on
stdin) so values stay off the shell command line.
- **Discover the snapshot structure first** (Step 3 rule — always do this before searching).
Write the arguments to a file, then:
```bash
# args.json:
# {"service": "<svc>", "environment": "<env>",
# "location_hash": "<16-hex>", "status_timestamp": "<ACTIVE-event-ISO8601>"}
python3 scripts/cloudwatch/di_snapshots.py sample --json-file args.json
```
Returns one nearby snapshot as JSON plus per-attribute `field_documentation`. Read the
field paths from this sample — they are authoritative; do not rely on canned paths that may
be stale.
- **Search a batch** near a status-event timestamp, narrowing with `custom_filters` when you
can name the target:
```bash
# args.json:
# {"service": "<svc>", "environment": "<env>",
# "location_hash": "<16-hex>", "status_timestamp": "<ISO8601>",
# "limit": 20, "custom_filters": ["..."]}
python3 scripts/cloudwatch/di_snapshots.py search --json-file args.json --out /tmp/snaps.json
```
`custom_filters` are raw Logs Insights fragments appended with `and`; an unbalanced double
quote is rejected. `--out` writes the result with owner-only (0600) permissions because
snapshots may contain PII/secrets.
`--print-contract` lists both ops and their exact argument schema.
## Parsing the saved output (never hand-transcribe; never `cat` a large file into context)
Large results are written to a file (use `--out`, or redirect stdout). Parse the file with
`jq`/`python` and extract only the fields you need. **Do not** retype values you see in tool
output into a script literal — a single mistyped `orderId`/`paymentRef` silently corrupts the
aggregation.
> **Encryption at rest for the saved file.** `--out` already restricts the file to owner-only
> (`0600`), but the snapshot may still contain credentials/PII. Write it only to a private
> location on an encrypted volume (e.g. an encrypted EBS volume or encrypted tmpfs), avoid
> world-readable shared temp directories, and delete it as soon as analysis is done.
The retrieval output is JSON. Snapshot records are under `results[*]`, each with an
`@message` that is itself a JSON string — `json.loads` it again to reach `body.captures.*`.
The parser already extracts the common debugging fields; key ones from a parsed snapshot:
| Field | Meaning |
| --- | --- |
| `entry_argument_names` / `entry_arguments` | method/function-entry argument names + values |
| `entry_local_names` / `entry_locals` | locals captured at entry |
| `return_value` / `throwable` | method return value or thrown exception |
| `line_numbers` / `line_locals` | line-level captured locals, keyed by line |
| `stack_preview` / `stack_frame_count` | call stack (frames use `file_path`/`line_number`) |
| `trace` | traceId/spanId for correlation |
| `duration_ms` | method duration (method-level only) |
**Java `Map`/`HashMap`** values appear as key/value `entries` (not flat `fields`); raise object
depth / collection width if map contents are truncated.
Write the jq/python against the **actual field paths from your live sample snapshot**, group by
a domain identifier (e.g. `orderId`), and surface anomalies (duplicates, outliers). When
combining results from multiple queries, deduplicate by snapshot `id` before aggregating.
After analysis, do not retain the saved snapshot file — it may contain PII/secrets.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |