Skip to content

opensite-ai/opensite-skills / aws-observability

aws-observability

Unverified

Builds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md28.3 KB
assets/cloudwatch/alarm-template.tsscript3.9 KB
assets/cloudwatch/otel-config.yamlconfig1.9 KB
references/cloudwatch-omni/agent-evaluation.mdmarkdown98.9 KB
references/cloudwatch-omni/alerts.mdmarkdown103.3 KB
references/cloudwatch-omni/concepts.mdmarkdown28.8 KB
references/cloudwatch-omni/context-graph.mdmarkdown29.2 KB
references/cloudwatch-omni/dashboards.mdmarkdown75.5 KB
references/cloudwatch-omni/programmatic-access.mdmarkdown8.6 KB
references/cloudwatch-omni/query/promql-metrics.mdmarkdown63.9 KB
references/cloudwatch-omni/query/sql-logs-traces.mdmarkdown43.0 KB
references/cloudwatch-omni/query/views.mdmarkdown12.7 KB
references/cloudwatch/alarms.mdmarkdown11.5 KB
references/cloudwatch/cloudtrail.mdmarkdown5.3 KB
references/cloudwatch/dashboards.mdmarkdown5.8 KB
references/cloudwatch/dynamic-instrumentation.mdmarkdown39.4 KB
references/cloudwatch/dynamic-instrumentation/breakpoint-creation.mdmarkdown16.3 KB
references/cloudwatch/dynamic-instrumentation/call-tree-and-directions.mdmarkdown5.7 KB
references/cloudwatch/dynamic-instrumentation/snapshot-parsing.mdmarkdown4.2 KB
references/cloudwatch/log-insights.mdmarkdown7.0 KB
references/cloudwatch/metrics.mdmarkdown7.4 KB
references/cloudwatch/synthetics.mdmarkdown6.5 KB
references/cloudwatch/tracing.mdmarkdown9.4 KB
references/cloudwatch/troubleshooting.mdmarkdown7.5 KB
scripts/cloudwatch-omni/capture_dataset_from_traces.pyscript19.2 KB
scripts/cloudwatch-omni/evaluate_traces.pyscript41.8 KB
scripts/cloudwatch-omni/test_evaluate_traces.pyscript12.0 KB
scripts/cloudwatch/di_app_signals_client.pyscript3.3 KB
scripts/cloudwatch/di_capture.pyscript8.5 KB
scripts/cloudwatch/di_constants.pyscript560 B
scripts/cloudwatch/di_crud_rendering.pyscript13.1 KB
scripts/cloudwatch/di_crud_tools.pyscript28.8 KB
scripts/cloudwatch/di_error_translation.pyscript5.6 KB
scripts/cloudwatch/di_formatting.pyscript992 B
scripts/cloudwatch/di_gateway.pyscript7.0 KB
scripts/cloudwatch/di_instrumentation.pyscript13.0 KB
scripts/cloudwatch/di_location.pyscript15.3 KB
scripts/cloudwatch/di_logs_client.pyscript4.2 KB
scripts/cloudwatch/di_region.pyscript2.0 KB
scripts/cloudwatch/di_result.pyscript2.0 KB
scripts/cloudwatch/di_session.pyscript2.0 KB
scripts/cloudwatch/di_snapshot_parsing.pyscript11.4 KB
scripts/cloudwatch/di_snapshot_queries.pyscript2.4 KB
scripts/cloudwatch/di_snapshot_rendering.pyscript12.5 KB
scripts/cloudwatch/di_snapshot_tools.pyscript12.0 KB
scripts/cloudwatch/di_snapshots.pyscript12.8 KB
scripts/cloudwatch/di_status_assessment.pyscript4.7 KB
scripts/cloudwatch/di_status_rendering.pyscript11.8 KB
scripts/cloudwatch/di_status_tools.pyscript13.0 KB
scripts/cloudwatch/di_validation.pyscript11.2 KB

references/cloudwatch/dynamic-instrumentation/snapshot-parsing.md

View on GitHub
# Snapshot retrieval and parsing

Snapshot data captured by a breakpoint lives in CloudWatch Logs
(`/aws/service-events/{service}`). Two host scripts wrap the Logs Insights queries; this file
is the recipe for analyzing what they return.

> **Reminder:** the snapshot log group (`/aws/service-events/{service}`) **must be encrypted
> at rest** with a KMS CMK (`aws logs associate-kms-key`) before capturing — captured snapshots
> may contain credentials, PII, or secrets. See Security Considerations in
> `dynamic-instrumentation.md`.

## Retrieval commands

Both require a host with `python3` + `boto3`. Region resolves as `--region` flag > `AWS_REGION` >
`AWS_DEFAULT_REGION` > `us-east-1` default (the same precedence as `di_instrumentation.py`) — it
MUST be the same region the breakpoint was created in, or searches return
empty even when the breakpoint is ACTIVE. Pass arguments via `--json-file` (or `--json -` on
stdin) so values stay off the shell command line.

- **Discover the snapshot structure first** (Step 3 rule — always do this before searching).
  Write the arguments to a file, then:

  ```bash
  # args.json:
  # {"service": "<svc>", "environment": "<env>",
  #  "location_hash": "<16-hex>", "status_timestamp": "<ACTIVE-event-ISO8601>"}
  python3 scripts/cloudwatch/di_snapshots.py sample --json-file args.json
  ```
  Returns one nearby snapshot as JSON plus per-attribute `field_documentation`. Read the
  field paths from this sample — they are authoritative; do not rely on canned paths that may
  be stale.

- **Search a batch** near a status-event timestamp, narrowing with `custom_filters` when you
  can name the target:

  ```bash
  # args.json:
  # {"service": "<svc>", "environment": "<env>",
  #  "location_hash": "<16-hex>", "status_timestamp": "<ISO8601>",
  #  "limit": 20, "custom_filters": ["..."]}
  python3 scripts/cloudwatch/di_snapshots.py search --json-file args.json --out /tmp/snaps.json
  ```
  `custom_filters` are raw Logs Insights fragments appended with `and`; an unbalanced double
  quote is rejected. `--out` writes the result with owner-only (0600) permissions because
  snapshots may contain PII/secrets.

`--print-contract` lists both ops and their exact argument schema.

## Parsing the saved output (never hand-transcribe; never `cat` a large file into context)

Large results are written to a file (use `--out`, or redirect stdout). Parse the file with
`jq`/`python` and extract only the fields you need. **Do not** retype values you see in tool
output into a script literal — a single mistyped `orderId`/`paymentRef` silently corrupts the
aggregation.

> **Encryption at rest for the saved file.** `--out` already restricts the file to owner-only
> (`0600`), but the snapshot may still contain credentials/PII. Write it only to a private
> location on an encrypted volume (e.g. an encrypted EBS volume or encrypted tmpfs), avoid
> world-readable shared temp directories, and delete it as soon as analysis is done.

The retrieval output is JSON. Snapshot records are under `results[*]`, each with an
`@message` that is itself a JSON string — `json.loads` it again to reach `body.captures.*`.
The parser already extracts the common debugging fields; key ones from a parsed snapshot:

| Field | Meaning |
| --- | --- |
| `entry_argument_names` / `entry_arguments` | method/function-entry argument names + values |
| `entry_local_names` / `entry_locals` | locals captured at entry |
| `return_value` / `throwable` | method return value or thrown exception |
| `line_numbers` / `line_locals` | line-level captured locals, keyed by line |
| `stack_preview` / `stack_frame_count` | call stack (frames use `file_path`/`line_number`) |
| `trace` | traceId/spanId for correlation |
| `duration_ms` | method duration (method-level only) |

**Java `Map`/`HashMap`** values appear as key/value `entries` (not flat `fields`); raise object
depth / collection width if map contents are truncated.

Write the jq/python against the **actual field paths from your live sample snapshot**, group by
a domain identifier (e.g. `orderId`), and surface anomalies (duplicates, outliers). When
combining results from multiple queries, deduplicate by snapshot `id` before aggregating.

After analysis, do not retain the saved snapshot file — it may contain PII/secrets.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall