Skip to content

opensite-ai/opensite-skills / aws-observability

aws-observability

Unverified

Builds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md28.3 KB
assets/cloudwatch/alarm-template.tsscript3.9 KB
assets/cloudwatch/otel-config.yamlconfig1.9 KB
references/cloudwatch-omni/agent-evaluation.mdmarkdown98.9 KB
references/cloudwatch-omni/alerts.mdmarkdown103.3 KB
references/cloudwatch-omni/concepts.mdmarkdown28.8 KB
references/cloudwatch-omni/context-graph.mdmarkdown29.2 KB
references/cloudwatch-omni/dashboards.mdmarkdown75.5 KB
references/cloudwatch-omni/programmatic-access.mdmarkdown8.6 KB
references/cloudwatch-omni/query/promql-metrics.mdmarkdown63.9 KB
references/cloudwatch-omni/query/sql-logs-traces.mdmarkdown43.0 KB
references/cloudwatch-omni/query/views.mdmarkdown12.7 KB
references/cloudwatch/alarms.mdmarkdown11.5 KB
references/cloudwatch/cloudtrail.mdmarkdown5.3 KB
references/cloudwatch/dashboards.mdmarkdown5.8 KB
references/cloudwatch/dynamic-instrumentation.mdmarkdown39.4 KB
references/cloudwatch/dynamic-instrumentation/breakpoint-creation.mdmarkdown16.3 KB
references/cloudwatch/dynamic-instrumentation/call-tree-and-directions.mdmarkdown5.7 KB
references/cloudwatch/dynamic-instrumentation/snapshot-parsing.mdmarkdown4.2 KB
references/cloudwatch/log-insights.mdmarkdown7.0 KB
references/cloudwatch/metrics.mdmarkdown7.4 KB
references/cloudwatch/synthetics.mdmarkdown6.5 KB
references/cloudwatch/tracing.mdmarkdown9.4 KB
references/cloudwatch/troubleshooting.mdmarkdown7.5 KB
scripts/cloudwatch-omni/capture_dataset_from_traces.pyscript19.2 KB
scripts/cloudwatch-omni/evaluate_traces.pyscript41.8 KB
scripts/cloudwatch-omni/test_evaluate_traces.pyscript12.0 KB
scripts/cloudwatch/di_app_signals_client.pyscript3.3 KB
scripts/cloudwatch/di_capture.pyscript8.5 KB
scripts/cloudwatch/di_constants.pyscript560 B
scripts/cloudwatch/di_crud_rendering.pyscript13.1 KB
scripts/cloudwatch/di_crud_tools.pyscript28.8 KB
scripts/cloudwatch/di_error_translation.pyscript5.6 KB
scripts/cloudwatch/di_formatting.pyscript992 B
scripts/cloudwatch/di_gateway.pyscript7.0 KB
scripts/cloudwatch/di_instrumentation.pyscript13.0 KB
scripts/cloudwatch/di_location.pyscript15.3 KB
scripts/cloudwatch/di_logs_client.pyscript4.2 KB
scripts/cloudwatch/di_region.pyscript2.0 KB
scripts/cloudwatch/di_result.pyscript2.0 KB
scripts/cloudwatch/di_session.pyscript2.0 KB
scripts/cloudwatch/di_snapshot_parsing.pyscript11.4 KB
scripts/cloudwatch/di_snapshot_queries.pyscript2.4 KB
scripts/cloudwatch/di_snapshot_rendering.pyscript12.5 KB
scripts/cloudwatch/di_snapshot_tools.pyscript12.0 KB
scripts/cloudwatch/di_snapshots.pyscript12.8 KB
scripts/cloudwatch/di_status_assessment.pyscript4.7 KB
scripts/cloudwatch/di_status_rendering.pyscript11.8 KB
scripts/cloudwatch/di_status_tools.pyscript13.0 KB
scripts/cloudwatch/di_validation.pyscript11.2 KB

references/cloudwatch/dynamic-instrumentation/call-tree-and-directions.md

View on GitHub
# Call Tree and Investigation Directions

Visual call tree patterns and correlation-guided direction choices.

## Visual Call Tree for Debugging

Use a visual tree structure to represent the debugging process. This helps:

1. **Visualize the call graph** - see how functions relate to each other
2. **Track investigation progress** - annotate nodes with status
3. **Communicate findings** - show the user what's been checked and what hasn't
4. **Document the path to root cause** - trace the issue through the tree

### Node Annotations

Use these annotations to mark the status of each node:

| Annotation | Meaning                                           |
| ---------- | ------------------------------------------------- |
| `OK`       | **Cleared** - No issue found in this code path    |
| `X`        | **Issue Found** - Bug or problem identified here  |
| `?`        | **Investigating** - Currently analyzing this node |
| `...`      | **Pending** - Need to investigate but haven't yet |

### Building the Call Tree

Start from the entry point and expand as you investigate. Example for a user registration service:

```
register_user() [entry point - auto-instrumented, skip]
├── validate_email(email) ...
├── check_username_available(username) ? [investigating - slow calls observed]
│   └── query_user_database(username) ...
├── hash_password(password) ...
├── create_user_record(user_data) ...
│   └── insert_into_database(record) ...
└── send_welcome_email(email) ...
```

### Detailed Node Expansion

When investigating a specific function, expand it to show internal logic:

```
check_username_available("john_doe") X [BUG FOUND - case sensitivity issue]
├── normalized = normalize_username("john_doe")
│   └── result: "john_doe" (no change)
├── query = build_query(normalized)
│   └── SQL: SELECT * FROM users WHERE username = 'john_doe'
├── result = execute_query(query)
│   └── Found: "John_Doe" exists X [case-insensitive match missed!]
├── return: True (available) X WRONG - should be False
└── Root cause: Query uses case-sensitive comparison but
    usernames should be case-insensitive
```

### Annotating with Evidence

Include snapshot data evidence directly in the tree:

```
process_checkout("order-5678", cart=[...])
├── Duration: 2,847ms X [SLOW - SLA is 500ms]
├── Input: cart = [
│   {"sku": "LAPTOP-001", "qty": 1},
│   {"sku": "MOUSE-002", "qty": 2}
│   ]
├── check_inventory("LAPTOP-001") OK
│   ├── Duration: 45ms [normal]
│   └── Evidence: Snapshot @ 14:22:03.112
├── check_inventory("MOUSE-002") OK
│   ├── Duration: 38ms [normal]
│   └── Evidence: Snapshot @ 14:22:03.157
├── calculate_shipping(address) X
│   ├── Duration: 2,651ms [SLOW!]
│   ├── Evidence: Snapshot @ 14:22:03.201
│   └── ? Need to investigate downstream calls
└── Return: {order_id: "ORD-9999", total: 1249.99}
```

### Comparing Good vs Bad Cases

Use side-by-side trees for comparison:

```
FAST REQUEST (domestic):               SLOW REQUEST (international):

calculate_shipping(addr)               calculate_shipping(addr)
├── country: "US"                      ├── country: "JP"
├── get_rates() → cache HIT OK          ├── get_rates() → cache MISS
├── duration: 12ms                     │   └── fetch_from_api()
└── return: $9.99                      │       └── duration: 2,340ms X
                                       ├── duration: 2,651ms
                                       └── return: $89.99
```

### Progressive Investigation Tree

Update the tree as you investigate deeper:

#### Step 1: Initial investigation

```
submit_payment()
├── validate_card() ? [some calls failing - investigating]
├── check_fraud() ?
├── charge_card() ?
└── send_receipt() ?
```

#### Step 2: After analyzing validate_card

```
submit_payment()
├── validate_card() ? [fails for certain card types]
│   ├── check_luhn() OK [algorithm correct]
│   ├── check_expiry() OK [date parsing correct]
│   └── check_card_type() X [fails for Amex cards]
├── check_fraud() OK [not reached when validation fails]
├── charge_card() OK [not reached when validation fails]
└── send_receipt() OK [not reached when validation fails]
```

#### Step 3: Drilling into check_card_type

```
submit_payment()
├── validate_card()
│   └── check_card_type("378282246310005") X
│       ├── Input: card_number starting with "37"
│       ├── Expected: "amex" (Amex starts with 34 or 37)
│       ├── Actual: "unknown" X
│       └── Bug: Regex pattern missing Amex prefix "37"
...
```

### Including in Reports

Include a "Call Tree" in your inline closure summary:

```markdown
## Call Tree

\`\`\`
submit_payment() [entry - auto-instrumented]
+-- validate_card(card_number) X ROOT CAUSE
| +-- check_luhn() OK
| +-- check_expiry() OK
| +-- check_card_type() X Missing Amex pattern "37"
+-- check_fraud() [not reached]
+-- charge_card() [not reached]
+-- send_receipt() [not reached]
\`\`\`

**Legend**: OK Cleared | X Issue | ? Investigating | ... Pending
```

---

## Connecting the Tree to Direction Choices

Use the call tree to choose the next move, not just to display progress. The node where the
tree first turns suspicious (`X` or `?`) determines the next direction — look it up in the
**Correlate → Decide** table in `dynamic-instrumentation.md` (Step 4). In short: an `X` on inputs sends you upstream, an
`X` on the return sends you downstream, a `?` on an intermediate value sends you line-level,
and mixed `OK`/`X` across runs sends you to multi-snapshot comparison.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall