Skip to content

opensite-ai/opensite-skills / aws-observability

aws-observability

Unverified

Builds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md28.3 KB
assets/cloudwatch/alarm-template.tsscript3.9 KB
assets/cloudwatch/otel-config.yamlconfig1.9 KB
references/cloudwatch-omni/agent-evaluation.mdmarkdown98.9 KB
references/cloudwatch-omni/alerts.mdmarkdown103.3 KB
references/cloudwatch-omni/concepts.mdmarkdown28.8 KB
references/cloudwatch-omni/context-graph.mdmarkdown29.2 KB
references/cloudwatch-omni/dashboards.mdmarkdown75.5 KB
references/cloudwatch-omni/programmatic-access.mdmarkdown8.6 KB
references/cloudwatch-omni/query/promql-metrics.mdmarkdown63.9 KB
references/cloudwatch-omni/query/sql-logs-traces.mdmarkdown43.0 KB
references/cloudwatch-omni/query/views.mdmarkdown12.7 KB
references/cloudwatch/alarms.mdmarkdown11.5 KB
references/cloudwatch/cloudtrail.mdmarkdown5.3 KB
references/cloudwatch/dashboards.mdmarkdown5.8 KB
references/cloudwatch/dynamic-instrumentation.mdmarkdown39.4 KB
references/cloudwatch/dynamic-instrumentation/breakpoint-creation.mdmarkdown16.3 KB
references/cloudwatch/dynamic-instrumentation/call-tree-and-directions.mdmarkdown5.7 KB
references/cloudwatch/dynamic-instrumentation/snapshot-parsing.mdmarkdown4.2 KB
references/cloudwatch/log-insights.mdmarkdown7.0 KB
references/cloudwatch/metrics.mdmarkdown7.4 KB
references/cloudwatch/synthetics.mdmarkdown6.5 KB
references/cloudwatch/tracing.mdmarkdown9.4 KB
references/cloudwatch/troubleshooting.mdmarkdown7.5 KB
scripts/cloudwatch-omni/capture_dataset_from_traces.pyscript19.2 KB
scripts/cloudwatch-omni/evaluate_traces.pyscript41.8 KB
scripts/cloudwatch-omni/test_evaluate_traces.pyscript12.0 KB
scripts/cloudwatch/di_app_signals_client.pyscript3.3 KB
scripts/cloudwatch/di_capture.pyscript8.5 KB
scripts/cloudwatch/di_constants.pyscript560 B
scripts/cloudwatch/di_crud_rendering.pyscript13.1 KB
scripts/cloudwatch/di_crud_tools.pyscript28.8 KB
scripts/cloudwatch/di_error_translation.pyscript5.6 KB
scripts/cloudwatch/di_formatting.pyscript992 B
scripts/cloudwatch/di_gateway.pyscript7.0 KB
scripts/cloudwatch/di_instrumentation.pyscript13.0 KB
scripts/cloudwatch/di_location.pyscript15.3 KB
scripts/cloudwatch/di_logs_client.pyscript4.2 KB
scripts/cloudwatch/di_region.pyscript2.0 KB
scripts/cloudwatch/di_result.pyscript2.0 KB
scripts/cloudwatch/di_session.pyscript2.0 KB
scripts/cloudwatch/di_snapshot_parsing.pyscript11.4 KB
scripts/cloudwatch/di_snapshot_queries.pyscript2.4 KB
scripts/cloudwatch/di_snapshot_rendering.pyscript12.5 KB
scripts/cloudwatch/di_snapshot_tools.pyscript12.0 KB
scripts/cloudwatch/di_snapshots.pyscript12.8 KB
scripts/cloudwatch/di_status_assessment.pyscript4.7 KB
scripts/cloudwatch/di_status_rendering.pyscript11.8 KB
scripts/cloudwatch/di_status_tools.pyscript13.0 KB
scripts/cloudwatch/di_validation.pyscript11.2 KB

references/cloudwatch/log-insights.md

View on GitHub
# CloudWatch Logs Insights

Complete query syntax reference, performance tips, and reusable query library.

## Contents
- [Commands](#commands)
- [Filter syntax](#filter-syntax)
- [Parse command](#parse-command)
- [Stats and aggregation](#stats-and-aggregation)
- [Time functions](#time-functions)
- [Advanced commands](#advanced-commands)
- [Known issues](#known-issues)
- [Reusable query library](#reusable-query-library)

---

## Commands

| Command | Description | Infrequent Access |
|---------|-------------|:-----------------:|
| `fields` | Select/transform fields, supports functions | Yes |
| `filter` | Match conditions with boolean/regex | Yes |
| `stats` | Aggregate statistics | Yes |
| `sort` | Order results `asc` or `desc` | Yes |
| `limit` | Specify max returned events (default 10,000 if omitted) | Yes |
| `parse` | Extract fields via glob or regex | Yes |
| `display` | Choose which fields to show | Yes |
| `dedup` | Remove duplicates by field | Yes |
| `unnest` | Flatten arrays into rows | Yes |
| `lookup` | Enrich with lookup table data | Yes |
| `join` | Combine events across log groups by key | Yes |
| `subqueries` | Nested queries as input | Yes |
| `anomaly` | ML anomaly detection | No |
| `pattern` | ML-based log clustering | No |
| `diff` | Compare current vs previous time period | No |
| `unmask` | Reveal data-protection masked content | No |
| `filterIndex` | Force field-index scan optimization | No |
| `SOURCE` | Programmatic log group selection (CLI/API only) | Yes |

Auto-discovered fields: `@timestamp`, `@message`, `@logStream`, `@log` (account-id:log-group-name), `@ingestionTime`, `@entity`. JSON fields auto-flattened with dot notation.

---

## Filter syntax

```
# Comparison: =, !=, <, <=, >, >=
filter statusCode >= 400

# Boolean: and, or, not
filter statusCode >= 400 and statusCode < 500

# Set membership
filter statusCode in [400, 401, 403, 404]

# Substring
filter @message like "ERROR"

# Regex
filter @message like /(?i)error/        # case-insensitive
filter @message =~ /timeout after \d+/  # regex match

# Negation
filter @message not like "DEBUG"
```

**Field index optimization**: Only `filter field = value` and `filter field IN [...]` use indexes. `filter field like` does NOT use indexes.

---

## Parse command

### Glob mode (wildcards)
```
parse @message "User * performed * on *" as user, action, resource
```

### Regex mode (named groups)
```
parse @message /User (?<user>\w+) performed (?<action>\w+)/
```

### Chaining for complex logs
```
# XML parsing
parse @message "<EventData>*</EventData>" as @EventData
| parse @EventData "<Data Name='ObjectName'>*</Data>" as ObjectName
```

---

## Stats and aggregation

```
# Basic aggregation
stats count(*), sum(duration), avg(duration), min(duration), max(duration)

# Percentiles
stats pct(duration, 50) as p50, pct(duration, 95) as p95, pct(duration, 99) as p99

# Time bucketing
stats count(*) as cnt by bin(5m)

# Group by field
stats count(*) as cnt by statusCode

# Combined
stats avg(duration) as avg_ms, pct(duration, 99) as p99 by serviceName, bin(1h)
```

---

## Time functions

- `bin(period)` — time bucketing: `bin(5m)`, `bin(1h)`, `bin(1d)`
- `datefloor(ts, period)`, `dateceil(ts, period)` — truncate/round
- `fromMillis(num)`, `toMillis(ts)` — epoch conversion
- `now()` — time query processing was started, in epoch seconds

**bin() caps**:
- ms → max 1000, s → max 60, m → max 60, h → max 24
- Use `bin(5m)` **NOT** `bin(300s)` — 300 exceeds the s→60 cap

---

## Advanced commands

### JOIN
Correlate events across log groups by a shared key:
```
filter status >= 500
| join type=inner left=api right=infra
    where api.requestId=infra.requestId
    (SOURCE '/aws/infra-logs')
```

### Subqueries
Use nested queries to filter the outer query:
```
filter requestId in (
    SOURCE '/aws/lambda/database-service'
    | filter errorType = "DatabaseConnectionTimeout"
    | fields requestId
)
```

### Anomaly detection
```
fields @timestamp, @message
| filter @message like /ERROR/
| pattern @message
| anomaly
```

### Scheduled queries
Recurring queries with results delivered to S3 and EventBridge. Configure via console or API.

---

## Known issues

1. **Backtick-escape field names with special characters**: `event-name` is interpreted as `event` minus `name`. Use `` `event-name` `` instead.

2. **100 concurrent query limit** per account (not adjustable). Partition queries by time range instead of parallelizing beyond this limit.

3. **JSON structured logs only ~10% faster** than unstructured text search. The real speedup comes from parallelizing across time ranges.

4. **Parallelization strategy**: Break queries into time-range chunks and run in parallel (14 × 12h instead of 1 × 7d). Reduces 84-minute query to ~6 minutes.

5. **`pattern`, `diff`, `unmask`, `anomaly`, and `filterIndex` don't work on Infrequent Access** log class.

6. **`head` and `tail` are deprecated** — use `limit` instead.

7. **StartQuery API**: 10 TPS (most regions). GetQueryResults: 10 TPS.

8. **Max 50 log groups** per query (API-level limit on `logGroupNames`/`logGroupIdentifiers`).

9. **No nested subqueries or correlated subqueries** — only simple subqueries.

10. **Subquery inner execution is limited to 30 seconds**. The overall query timeout is 60 minutes.

---

## Reusable query library

### Error analysis
```
# Recent errors with context
fields @timestamp, @message, @logStream
| filter @message like /ERROR/
| sort @timestamp desc
| limit 100

# Error rate by time bucket
fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) as errorCount by bin(5m)
| sort errorCount desc

# Top error patterns (ML clustering)
fields @timestamp, @message
| filter @message like /ERROR/
| pattern @message
```

### Lambda-specific
```
# Cold start analysis
filter @type = "REPORT"
| stats avg(@duration) as avg_ms, max(@duration) as max_ms,
        count(*) as invocations,
        sum(strcontains(@message, "Init Duration")) as coldStarts
  by bin(1h)

# Memory utilization
filter @type = "REPORT"
| stats max(@memorySize / 1000 / 1000) as provisioned_mb,
        max(@maxMemoryUsed / 1000 / 1000) as used_mb,
        avg(@maxMemoryUsed * 100 / @memorySize) as utilization_pct
  by bin(1h)

# Timeout detection
filter @message like /Task timed out/
| fields @timestamp, @requestId, @message
| sort @timestamp desc
| limit 20
```

### API Gateway
```
# 5xx errors by endpoint
fields @timestamp, httpMethod, resourcePath, status
| filter status >= 500
| stats count(*) as errors by resourcePath, httpMethod
| sort errors desc

# Latency percentiles by endpoint
fields @timestamp, resourcePath, responseLatency
| stats pct(responseLatency, 50) as p50,
        pct(responseLatency, 90) as p90,
        pct(responseLatency, 99) as p99
  by resourcePath
| sort p99 desc
```

### Cross-service correlation
```
# Multi-log-group error correlation (using SOURCE)
SOURCE logGroups(namePrefix: ['/app-logs', '/api-gateway-logs'])
| fields @timestamp, @message, @log
| filter @message like /ERROR/ or status >= 500
| sort @timestamp desc
| limit 200
```
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall