opensite-ai/opensite-skills / aws-observability
aws-observability
UnverifiedBuilds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.
references/cloudwatch/cloudtrail.md
View on GitHub# CloudTrail Operational Auditing
Using CloudTrail for operational debugging: who changed what, when. Not for security threat detection.
## Contents
- [Event types](#event-types)
- [Event history](#event-history)
- [Common operational queries](#common-operational-queries)
- [Querying CloudTrail logs](#querying-cloudtrail-logs)
- [CloudTrail → CloudWatch integration](#cloudtrail--cloudwatch-integration)
---
## Event types
| Type | Description | Default logging | Cost |
|------|-------------|:-:|------|
| **Management events** | Control plane (CreateBucket, RunInstances, IAM changes) | Yes | First copy included |
| **Data events** | Data plane (S3 GetObject, Lambda Invoke, DynamoDB GetItem) | No | Additional cost |
| **Network activity events** | VPC endpoint activity | No | Additional cost |
| **Insights events** | Unusual API call rate or error rate | No | Additional cost |
---
## Event history
- **90 days** of management events retained by default, no trail required
- Searchable in console by event name, resource type, user name, time range
- **200,000 event limit** when downloading
- Single account, single Region only
- Cannot view data events, Insights events, or network activity events
### Common lookups
```bash
# Who deleted an S3 bucket?
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=DeleteBucket \
--start-time 2026-04-20T00:00:00Z
# Who modified a security group?
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngress
# Who stopped an EC2 instance?
aws cloudtrail lookup-events \
--lookup-attributes AttributeKey=ResourceName,AttributeValue=i-1234567890abcdef0
```
---
## Common operational queries
### "Who deleted my resource?"
1. Check Event History (90 days) for `Delete*` events
2. Filter by resource name or resource type
3. Look at `userIdentity.arn` for the actor and `sourceIPAddress` for origin
### "Who changed this configuration?"
1. Search for `Update*`, `Modify*`, `Put*` events on the resource
2. Compare `requestParameters` across events to see what changed
### "What happened during the incident?"
1. Filter by time range of the incident
2. Look for `errorCode` fields (AccessDenied, ThrottlingException)
3. Correlate with CloudWatch metrics/logs for the same time window
### "Who accessed my data?" (requires data events)
Data events must be explicitly enabled on the trail.
> **`put-event-selectors` REPLACES the trail's entire selector set — it does not append.** Sending
> only the selector below to a trail that already has selectors silently drops the existing ones,
> which can turn off data-event capture the customer relies on for audit or security. Read first,
> merge, then write:
>
> ```bash
> aws cloudtrail get-event-selectors --trail-name my-trail
> # add your selector to the returned AdvancedEventSelectors[] list, keep the rest, then put the
> # COMPLETE merged list back
> ```
>
> **Check which selector style the trail uses first.** `get-event-selectors` returns *either*
> `AdvancedEventSelectors` *or* the basic `EventSelectors`, and `put-event-selectors` accepts only one
> of the two. If the trail returned basic `EventSelectors`, writing `--advanced-event-selectors`
> converts the trail and **discards every basic selector** — the same silent loss this callout is
> about. In that case translate the existing basic selectors into advanced form as part of the merge,
> and confirm the conversion with the customer before writing.
>
> Confirm the merged set with the customer before writing. Note also that data events are billed
> per event and are typically far higher volume than management events, so say what the selector
> will match before enabling it.
```bash
# The list below must be the COMPLETE set of selectors the trail should end up with,
# i.e. existing selectors from get-event-selectors PLUS this new one.
aws cloudtrail put-event-selectors --trail-name my-trail \
--advanced-event-selectors '[{
"Name": "S3DataEvents",
"FieldSelectors": [
{"Field": "eventCategory", "Equals": ["Data"]},
{"Field": "resources.type", "Equals": ["AWS::S3::Object"]}
]
}]'
```
---
## Querying CloudTrail logs
### Recommended: Trail → S3 → Athena
For new setups, deliver CloudTrail logs to S3 and query with Amazon Athena:
```sql
SELECT eventTime, userIdentity.arn, sourceIPAddress, eventName
FROM cloudtrail_logs
WHERE eventName = 'DeleteBucket'
AND eventTime > '2026-04-20'
ORDER BY eventTime DESC
LIMIT 100;
```
This is the long-term supported approach — works with standard SQL, scales to any volume, and integrates with existing S3-based analytics.
---
## CloudTrail → CloudWatch integration
### Alert on specific API calls
```
CloudTrail → Trail → CloudWatch Logs → Metric Filter → CloudWatch Alarm → SNS
```
1. Configure trail to deliver events to a CloudWatch Logs log group
2. Create metric filter for the event pattern (e.g., `{ $.eventName = "DeleteBucket" }`)
3. Create alarm on the metric filter
4. Configure SNS notification
### Event selectors
- **Basic**: simple include/exclude for management and data events
- **Advanced**: fine-grained filtering by event source, resource type, resource ARN
- Exclude high-volume management event sources on trails: AWS KMS, RDS Data API
- Max **250 data resources** across all basic event selectors per trail (does not apply to advanced event selectors)
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics. | Stars: 471k | Updated | Claude CodeCodex | Verified | Install |
| ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first. | Stars: 194k | Updated | Claude CodeCodex | Verified | Install |