Skip to content

opensite-ai/opensite-skills / aws-observability

aws-observability

Unverified

Builds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md28.3 KB
assets/cloudwatch/alarm-template.tsscript3.9 KB
assets/cloudwatch/otel-config.yamlconfig1.9 KB
references/cloudwatch-omni/agent-evaluation.mdmarkdown98.9 KB
references/cloudwatch-omni/alerts.mdmarkdown103.3 KB
references/cloudwatch-omni/concepts.mdmarkdown28.8 KB
references/cloudwatch-omni/context-graph.mdmarkdown29.2 KB
references/cloudwatch-omni/dashboards.mdmarkdown75.5 KB
references/cloudwatch-omni/programmatic-access.mdmarkdown8.6 KB
references/cloudwatch-omni/query/promql-metrics.mdmarkdown63.9 KB
references/cloudwatch-omni/query/sql-logs-traces.mdmarkdown43.0 KB
references/cloudwatch-omni/query/views.mdmarkdown12.7 KB
references/cloudwatch/alarms.mdmarkdown11.5 KB
references/cloudwatch/cloudtrail.mdmarkdown5.3 KB
references/cloudwatch/dashboards.mdmarkdown5.8 KB
references/cloudwatch/dynamic-instrumentation.mdmarkdown39.4 KB
references/cloudwatch/dynamic-instrumentation/breakpoint-creation.mdmarkdown16.3 KB
references/cloudwatch/dynamic-instrumentation/call-tree-and-directions.mdmarkdown5.7 KB
references/cloudwatch/dynamic-instrumentation/snapshot-parsing.mdmarkdown4.2 KB
references/cloudwatch/log-insights.mdmarkdown7.0 KB
references/cloudwatch/metrics.mdmarkdown7.4 KB
references/cloudwatch/synthetics.mdmarkdown6.5 KB
references/cloudwatch/tracing.mdmarkdown9.4 KB
references/cloudwatch/troubleshooting.mdmarkdown7.5 KB
scripts/cloudwatch-omni/capture_dataset_from_traces.pyscript19.2 KB
scripts/cloudwatch-omni/evaluate_traces.pyscript41.8 KB
scripts/cloudwatch-omni/test_evaluate_traces.pyscript12.0 KB
scripts/cloudwatch/di_app_signals_client.pyscript3.3 KB
scripts/cloudwatch/di_capture.pyscript8.5 KB
scripts/cloudwatch/di_constants.pyscript560 B
scripts/cloudwatch/di_crud_rendering.pyscript13.1 KB
scripts/cloudwatch/di_crud_tools.pyscript28.8 KB
scripts/cloudwatch/di_error_translation.pyscript5.6 KB
scripts/cloudwatch/di_formatting.pyscript992 B
scripts/cloudwatch/di_gateway.pyscript7.0 KB
scripts/cloudwatch/di_instrumentation.pyscript13.0 KB
scripts/cloudwatch/di_location.pyscript15.3 KB
scripts/cloudwatch/di_logs_client.pyscript4.2 KB
scripts/cloudwatch/di_region.pyscript2.0 KB
scripts/cloudwatch/di_result.pyscript2.0 KB
scripts/cloudwatch/di_session.pyscript2.0 KB
scripts/cloudwatch/di_snapshot_parsing.pyscript11.4 KB
scripts/cloudwatch/di_snapshot_queries.pyscript2.4 KB
scripts/cloudwatch/di_snapshot_rendering.pyscript12.5 KB
scripts/cloudwatch/di_snapshot_tools.pyscript12.0 KB
scripts/cloudwatch/di_snapshots.pyscript12.8 KB
scripts/cloudwatch/di_status_assessment.pyscript4.7 KB
scripts/cloudwatch/di_status_rendering.pyscript11.8 KB
scripts/cloudwatch/di_status_tools.pyscript13.0 KB
scripts/cloudwatch/di_validation.pyscript11.2 KB

references/cloudwatch/cloudtrail.md

View on GitHub
# CloudTrail Operational Auditing

Using CloudTrail for operational debugging: who changed what, when. Not for security threat detection.

## Contents
- [Event types](#event-types)
- [Event history](#event-history)
- [Common operational queries](#common-operational-queries)
- [Querying CloudTrail logs](#querying-cloudtrail-logs)
- [CloudTrail → CloudWatch integration](#cloudtrail--cloudwatch-integration)

---

## Event types

| Type | Description | Default logging | Cost |
|------|-------------|:-:|------|
| **Management events** | Control plane (CreateBucket, RunInstances, IAM changes) | Yes | First copy included |
| **Data events** | Data plane (S3 GetObject, Lambda Invoke, DynamoDB GetItem) | No | Additional cost |
| **Network activity events** | VPC endpoint activity | No | Additional cost |
| **Insights events** | Unusual API call rate or error rate | No | Additional cost |

---

## Event history

- **90 days** of management events retained by default, no trail required
- Searchable in console by event name, resource type, user name, time range
- **200,000 event limit** when downloading
- Single account, single Region only
- Cannot view data events, Insights events, or network activity events

### Common lookups
```bash
# Who deleted an S3 bucket?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteBucket \
  --start-time 2026-04-20T00:00:00Z

# Who modified a security group?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngress

# Who stopped an EC2 instance?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=i-1234567890abcdef0
```

---

## Common operational queries

### "Who deleted my resource?"
1. Check Event History (90 days) for `Delete*` events
2. Filter by resource name or resource type
3. Look at `userIdentity.arn` for the actor and `sourceIPAddress` for origin

### "Who changed this configuration?"
1. Search for `Update*`, `Modify*`, `Put*` events on the resource
2. Compare `requestParameters` across events to see what changed

### "What happened during the incident?"
1. Filter by time range of the incident
2. Look for `errorCode` fields (AccessDenied, ThrottlingException)
3. Correlate with CloudWatch metrics/logs for the same time window

### "Who accessed my data?" (requires data events)
Data events must be explicitly enabled on the trail.

> **`put-event-selectors` REPLACES the trail's entire selector set — it does not append.** Sending
> only the selector below to a trail that already has selectors silently drops the existing ones,
> which can turn off data-event capture the customer relies on for audit or security. Read first,
> merge, then write:
>
> ```bash
> aws cloudtrail get-event-selectors --trail-name my-trail
> # add your selector to the returned AdvancedEventSelectors[] list, keep the rest, then put the
> # COMPLETE merged list back
> ```
>
> **Check which selector style the trail uses first.** `get-event-selectors` returns *either*
> `AdvancedEventSelectors` *or* the basic `EventSelectors`, and `put-event-selectors` accepts only one
> of the two. If the trail returned basic `EventSelectors`, writing `--advanced-event-selectors`
> converts the trail and **discards every basic selector** — the same silent loss this callout is
> about. In that case translate the existing basic selectors into advanced form as part of the merge,
> and confirm the conversion with the customer before writing.
>
> Confirm the merged set with the customer before writing. Note also that data events are billed
> per event and are typically far higher volume than management events, so say what the selector
> will match before enabling it.

```bash
# The list below must be the COMPLETE set of selectors the trail should end up with,
# i.e. existing selectors from get-event-selectors PLUS this new one.
aws cloudtrail put-event-selectors --trail-name my-trail \
  --advanced-event-selectors '[{
    "Name": "S3DataEvents",
    "FieldSelectors": [
      {"Field": "eventCategory", "Equals": ["Data"]},
      {"Field": "resources.type", "Equals": ["AWS::S3::Object"]}
    ]
  }]'
```

---

## Querying CloudTrail logs

### Recommended: Trail → S3 → Athena

For new setups, deliver CloudTrail logs to S3 and query with Amazon Athena:

```sql
SELECT eventTime, userIdentity.arn, sourceIPAddress, eventName
FROM cloudtrail_logs
WHERE eventName = 'DeleteBucket'
  AND eventTime > '2026-04-20'
ORDER BY eventTime DESC
LIMIT 100;
```

This is the long-term supported approach — works with standard SQL, scales to any volume, and integrates with existing S3-based analytics.

---

## CloudTrail → CloudWatch integration

### Alert on specific API calls
```
CloudTrail → Trail → CloudWatch Logs → Metric Filter → CloudWatch Alarm → SNS
```

1. Configure trail to deliver events to a CloudWatch Logs log group
2. Create metric filter for the event pattern (e.g., `{ $.eventName = "DeleteBucket" }`)
3. Create alarm on the metric filter
4. Configure SNS notification

### Event selectors
- **Basic**: simple include/exclude for management and data events
- **Advanced**: fine-grained filtering by event source, resource type, resource ARN
- Exclude high-volume management event sources on trails: AWS KMS, RDS Data API
- Max **250 data resources** across all basic event selectors per trail (does not apply to advanced event selectors)
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 471kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall