Skip to content

opensite-ai/opensite-skills / aws-observability

aws-observability

Unverified

Builds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging or threat detection.

FileKindSize
.aws-skill-metadataother18 B
SKILL.mdskill md28.3 KB
assets/cloudwatch/alarm-template.tsscript3.9 KB
assets/cloudwatch/otel-config.yamlconfig1.9 KB
references/cloudwatch-omni/agent-evaluation.mdmarkdown98.9 KB
references/cloudwatch-omni/alerts.mdmarkdown103.3 KB
references/cloudwatch-omni/concepts.mdmarkdown28.8 KB
references/cloudwatch-omni/context-graph.mdmarkdown29.2 KB
references/cloudwatch-omni/dashboards.mdmarkdown75.5 KB
references/cloudwatch-omni/programmatic-access.mdmarkdown8.6 KB
references/cloudwatch-omni/query/promql-metrics.mdmarkdown63.9 KB
references/cloudwatch-omni/query/sql-logs-traces.mdmarkdown43.0 KB
references/cloudwatch-omni/query/views.mdmarkdown12.7 KB
references/cloudwatch/alarms.mdmarkdown11.5 KB
references/cloudwatch/cloudtrail.mdmarkdown5.3 KB
references/cloudwatch/dashboards.mdmarkdown5.8 KB
references/cloudwatch/dynamic-instrumentation.mdmarkdown39.4 KB
references/cloudwatch/dynamic-instrumentation/breakpoint-creation.mdmarkdown16.3 KB
references/cloudwatch/dynamic-instrumentation/call-tree-and-directions.mdmarkdown5.7 KB
references/cloudwatch/dynamic-instrumentation/snapshot-parsing.mdmarkdown4.2 KB
references/cloudwatch/log-insights.mdmarkdown7.0 KB
references/cloudwatch/metrics.mdmarkdown7.4 KB
references/cloudwatch/synthetics.mdmarkdown6.5 KB
references/cloudwatch/tracing.mdmarkdown9.4 KB
references/cloudwatch/troubleshooting.mdmarkdown7.5 KB
scripts/cloudwatch-omni/capture_dataset_from_traces.pyscript19.2 KB
scripts/cloudwatch-omni/evaluate_traces.pyscript41.8 KB
scripts/cloudwatch-omni/test_evaluate_traces.pyscript12.0 KB
scripts/cloudwatch/di_app_signals_client.pyscript3.3 KB
scripts/cloudwatch/di_capture.pyscript8.5 KB
scripts/cloudwatch/di_constants.pyscript560 B
scripts/cloudwatch/di_crud_rendering.pyscript13.1 KB
scripts/cloudwatch/di_crud_tools.pyscript28.8 KB
scripts/cloudwatch/di_error_translation.pyscript5.6 KB
scripts/cloudwatch/di_formatting.pyscript992 B
scripts/cloudwatch/di_gateway.pyscript7.0 KB
scripts/cloudwatch/di_instrumentation.pyscript13.0 KB
scripts/cloudwatch/di_location.pyscript15.3 KB
scripts/cloudwatch/di_logs_client.pyscript4.2 KB
scripts/cloudwatch/di_region.pyscript2.0 KB
scripts/cloudwatch/di_result.pyscript2.0 KB
scripts/cloudwatch/di_session.pyscript2.0 KB
scripts/cloudwatch/di_snapshot_parsing.pyscript11.4 KB
scripts/cloudwatch/di_snapshot_queries.pyscript2.4 KB
scripts/cloudwatch/di_snapshot_rendering.pyscript12.5 KB
scripts/cloudwatch/di_snapshot_tools.pyscript12.0 KB
scripts/cloudwatch/di_snapshots.pyscript12.8 KB
scripts/cloudwatch/di_status_assessment.pyscript4.7 KB
scripts/cloudwatch/di_status_rendering.pyscript11.8 KB
scripts/cloudwatch/di_status_tools.pyscript13.0 KB
scripts/cloudwatch/di_validation.pyscript11.2 KB

references/cloudwatch-omni/dashboards.md

View on GitHub
# Building a CloudWatch Omni Dashboard

Use this when building, saving, reading back, or fixing a CloudWatch Omni dashboard
— "build me a dashboard for checkout", "show what's most critical for this
service", "lay this out", "which chart for this signal?", "why is that panel
empty?", "why does that field have no effect after save?", "why was my save rejected?" —
and for the dashboard API itself
(`CreateOmniDashboard` and its siblings, `aws cloudwatchomni *-omni-dashboard*`).

A CloudWatch Omni dashboard is a set of **panels** laid out on a fixed 60-column
grid, most of them charts over CloudWatch Omni queries. The dashboard **body is
JSON** — a `panels[]` array (plus an optional `scope`) that the Omni UI renders. It
belongs to a **Space** and is stored, as a JSON string, in the `body` field of a
dashboard resource you create and update through the API (section 7). The
deliverable is therefore two things: a valid, grounded body, and the API call that
saves it.

**Contents**

1. [Omni dashboard vs. CloudWatch dashboard](#1-omni-dashboard-vs-cloudwatch-dashboard)
2. [Decide what the dashboard shows](#2-decide-what-the-dashboard-shows)
3. [Ground every panel query](#3-ground-every-panel-query)
4. [Author the panel bodies](#4-author-the-panel-bodies)
5. [Lay out on the 60-column grid](#5-lay-out-on-the-60-column-grid)
6. [Troubleshooting](#6-troubleshooting)
7. [API reference](#7-api-reference)
8. [Appendix — Archetype templates](#8-appendix--archetype-templates)

Two rules hold across every step:

- **Never chart a metric, field, or label you have not confirmed exists.** An
  invented name renders an **empty** panel, which on a saved dashboard reads like
  an outage — worse than no panel. Ground first (section 3); drop what you cannot
  ground.
- **Query text follows the CloudWatch Omni query dialect.** The SQL rules for logs
  and traces live in [sql-logs-traces.md](query/sql-logs-traces.md); metric
  semantics and PromQL selectors live in [promql-metrics.md](query/promql-metrics.md).
  This reference owns the dashboard body — the panel schema, chart choices, grid
  math — and the dashboard API. It does not restate the dialects.

**ALWAYS STATE — how a dashboard body fails.** It fails in several ways, and **only
one of them errors** — the rest fail silently, so call the right one out whenever you
author, explain, or debug a body (full detail in "API save semantics", section 4):

- **A structural or key error is REJECTED at save (HTTP 400 `ValidationException`).**
  The root and panel objects are schema-validated (`additionalProperties` off), so this
  covers an unknown or misspelled key there (a top-level `refreshInterval`, a per-panel
  `id`, a misspelled `title`, or a **`source`** placed at panel level instead of inside
  `config` — `panels[0]: does not satisfy the 'additionalProperties' constraint of the
  dashboard body schema`), a bad `variant` enum, a missing or non-array `panels`, a
  panel missing its required `type` or `layout`, and an out-of-bounds coordinate FIELD
  (`x`/`y` below 0, `w` below 1). The save does NOT succeed — none of these is silently
  dropped. Validate the structure and every root/panel key against section 4.
- **A bad VALUE saves with HTTP 200, then blanks at render.** The schema does not check
  cross-field invariants, and the only enum VALUE it validates is `variant` (a bad one
  is rejected at 400 — previous bullet); `visualization`, panel `type`, and `alerts`
  `state` values are unchecked. An unknown **`visualization`** VALUE blanks THAT panel
  while its siblings still render. Everything else it does not catch blanks
  the WHOLE dashboard (every sibling included): the cross-field `x + w > 60` overflow
  (per-field bounds are validated, their sum is not), a `w > 60`, a bad `h` unit
  (`80vh`), an unrecognised panel `type` VALUE, a bad `alerts` `state`, or a
  `now-30s`/`now-1y` time bound. Either way there is no error, and a `200` is not proof
  of renderability; nothing enforces the `x + w <= 60` sum.
- **A wrong key INSIDE `config` is accepted and silently ignored.** `config` (and its
  nested objects like `chartOptions`) is permissive: a misspelled or unknown key there
  is stored, comes back on read, and never takes effect. This — not root/panel keys —
  is where "a field I set doesn't take effect, with no error" actually happens.
- **`queryLanguage` must be set explicitly per panel; a mismatch fails silently.**
  Set each `explore` panel's `config.queryLanguage` to `'sql'` or `'promql'` yourself
  — it is **not** inferred from the query text. (Metrics → `promql`; log/trace
  content → `sql`.)

Work sections 2–5 in order — each depends on the one before — then save with the
API in section 7.

---

## 1. Omni dashboard vs. CloudWatch dashboard

Both are called "a CloudWatch dashboard" in ordinary speech, but they are different
resources with different bodies, different APIs, and different query languages:

| | CloudWatch Omni dashboard (this file) | CloudWatch dashboard |
|---|---|---|
| Lives in | An Omni **Space** (`spaceId` on every call) | The AWS account/Region |
| API / CLI | `CreateOmniDashboard` etc. — `aws cloudwatchomni …` | `PutDashboard` etc. — `aws cloudwatch …` |
| Body | `{ "panels": [...], "scope": {...} }` — panel `type`, `config`, `layout {x,y,w,h}` | `{ "widgets": [...] }` — widget `type`, `properties`, `x/y/width/height` |
| Grid | **60** columns; `y`/`h` in **pixels** | 24 columns; all four in grid units |
| Panel queries | Omni SQL (logs/traces) and PromQL (metrics), one query per panel | CloudWatch metric definitions, Metrics Insights, Logs Insights |
| Identity | `dashboardId` (system-minted UUID); the name is metadata | The dashboard name |

The two bodies are **not** interchangeable — a CloudWatch `widgets[]` body pasted into
`CreateOmniDashboard` is rejected, and an Omni `panels[]` body means nothing to
`PutDashboard`. CloudWatch dashboards are documented in
[../cloudwatch/dashboards.md](../cloudwatch/dashboards.md).

**Confirm Omni is enabled before acting.** The Omni dashboard API is only reachable
once a Space exists in the target Region. Before you create, update, or delete a
dashboard, probe with `aws cloudwatchomni list-spaces` in that Region. If there is
no Space, say so plainly and route the same need to the CloudWatch path — that is the
correct fallback, not a wrong answer. A **knowledge question** ("does Omni have a
dashboard API?", "what's the panel schema?") is answered from this file directly;
do not gate it behind the probe. See [concepts.md](concepts.md) for what a Space is.

---

### Facts you MUST surface when building or saving an Omni dashboard

Author and ground the panels, and surface the items below that are relevant to the task —
each is a save-time or render-time trap the API does not fully catch (only structural,
root/panel key, coordinate-field, and `variant` errors are loud; the rest are silent).
When the task is choosing a panel's chart type, surface the visualization and panel-shape
items; when saving or reading a dashboard back, surface the save-boundary behavior for
their config. Each item points to the section that details it; state what fits the task,
not a re-paste of every detail.

- **Pick a `visualization` value from §4's supported set deliberately, there is no safe
  default.** Any value outside that set, plus a missing or empty value, saves `200` and
  silently blanks that one panel at render, with no fallback that draws.
- **`visualization` and `queryLanguage` are both REQUIRED on every `explore` (data)
  panel, and a `queryLanguage` mismatch is a silent blank rather than an error.** They
  are `config` fields of the `explore` panel type only. The non-explore panel types
  (such as `markdown`, `divider`, `alerts`, `spans`, and `application-map`) carry
  neither. Set each `explore` panel's `config.queryLanguage`
  to `sql` or `promql` yourself. It is never inferred from the query text (§4 preamble,
  §4 "explore", §4 "API save semantics").
- **The two panel shapes authors most often mis-pick:** a single scalar aggregate →
  `number`, not `table`; a top-N ranking (`ORDER BY … LIMIT` / `topk`) → `table`, not
  `line` (§4, "Visualizations").
- **The `number`-tile single-column product bug.** A `number` panel whose SQL selects
  **one column** renders the result-set ROW COUNT, not the value — so every single-row
  scalar aggregate reads `1`. This is not specific to `COUNT(*)`, and making the aggregate
  the only/first selected column is exactly what triggers it. Put a scalar KPI in a
  `table` panel (§4, "Visualizations", "`number` tile trap").
- **The save boundary — where the API is loud vs silent.** SILENT (saves HTTP 200,
  fails only at render): an unknown key INSIDE `config` is stored, round-trips on read,
  and is silently ignored; and a cross-field `x + w > 60` overflow blanks the whole
  canvas. LOUD (REJECTED at save with HTTP 400 `ValidationException`): a structural,
  top-level or per-panel key, per-field coordinate-BOUND (negative `x`/`y`, `w < 1`), or
  `variant` violation. And `GetOmniDashboard` returns the body byte-for-byte (§4, "API
  save semantics").

---

## 2. Decide what the dashboard shows

Pick the panels, order them into sections, and choose a visualization per signal.

### Start focused — a tight dashboard beats an exhaustive one

- **The default is a FOCUSED ~6–8 panels:** roughly **4 KPI tiles + 2–3
  golden-signal trends + at most one breakdown table.** A single-signal dashboard
  is smaller still (**3–6 panels**). Don't emit dozens of panels for a vague ask.
- **But build the COMPLETE core — never collapse to one row.** A real dashboard
  reads as summary → trend → breakdown (below): a KPI row AND at least one trend
  chart AND a breakdown. If some metric-based panels cannot ground, fill those
  tiers from traces/logs (section 3) rather than shipping a lone KPI row.
- **Cap the KPI row at 4 tiles** (`number` panels). Four is also the natural grid
  fit — `15 × 4 = 60` columns — so a fifth tile forces an awkward wrap.
- **Always include a saturation panel in the core when the grounded catalog has
  one** — backlog / queue depth, consumer lag / liveness, or circuit-open /
  write-denied. This is the incident-visibility panel; the broader "infra / USE
  health" sweep below remains a deferrable follow-up, but the single saturation
  signal belongs in the core.
- **Build the core, then offer depth as a follow-up "add section."** Defer these
  rather than placing them up front: per-service latency, error / status-code
  breakdown, infra / USE health, and log signals. Name the one the user most likely
  wants next instead of pre-building all of them.

### Ordering principle — headline first, detail last

Every dashboard reads top-to-bottom as **summary → trend → breakdown → raw**:

1. **KPI row** — a few `number` panels with the headline figures (availability,
   error rate, p99 latency, request rate). The fast "is it healthy?" read.
2. **Trend charts** — `line` / `area` / `bar` time series for the golden signals.
3. **Ranked breakdown** — a `table` (top-N by the primary signal) so the worst
   offenders surface.
4. **Raw / drill-down** — logs, spans, alerts, or an application map. Optional
   depth, offered as a follow-up rather than built by default.

Open a logically distinct group with a `divider`, and open an on-call / incident
dashboard with a short `markdown` runbook panel.

### If the ask names a resource type, start from a template

When the request names a known resource TYPE — an EC2 fleet, a Lambda function, a
Kubernetes/EKS workload, an RDS/database instance, an ECS/web service, or an API
Gateway API — don't classify from scratch. Start from that archetype's ready panel
set in the [Appendix](#8-appendix--archetype-templates), then ground and lay it out.

The panel *content* for a resource type — which signals, how each is read
(gauge vs. counter, `max` for a 0/1 check, `min` for a credit balance), which exist
only under a configuration (an installed agent, a burstable instance type, Container
Insights), and the identity label to group by — is owned by the per-service catalog in
[promql-metrics.md](query/promql-metrics.md) (its "Service-health question — facts you
MUST surface" section and section 5). When the ask is which signals or panels a named
resource type needs, state those catalog facts as the proposed panel set; this file then
supplies the body and the grid. Do not decide the content by enumerating resources
through the service's control plane.

### Recipes (for a general intent)

- **Golden-signals service health (the default when the ask is vague).** For
  "build a dashboard for `<service>`" / "how is `<service>` doing?". Use RED
  (rate/errors/duration) for a request-serving service, USE (utilization,
  saturation, errors) for an infrastructure resource. CORE (~6–8): a KPI row
  (availability %, error rate, p99 latency, request rate); a request-rate `line`;
  an errors `bar` (stacked) or `area`; a latency `line` (p50/p90/p99) **when a real
  percentile source exists** (section 3); and a top-N `table`. Build each family
  from the service's OWN grounded metric first, falling back to a generic OTel
  metric, then a span / log derivation, only when no higher tier grounds it.
- **Errors-focused.** A headline error-rate `number`; a stacked `bar`/`area` of the
  status-code or fault/error breakdown over time; a `table` of top error sources.
  Offer a recent-error log panel and — if the space has alerts — an `alerts` panel
  filtered to `CRITICAL` (or `WARNING`) as a follow-up.
- **Latency-focused.** A p99 `number`; a `line` of p50/p90/p99 together; a `table`
  of the slowest operations. If the service exposes no percentile-capable source,
  chart the average with an honest title — never a fake p99.
- **Throughput-focused.** A request/message-rate `number`; a `line` over time
  grouped by the identity label — read the aggregation off the counter's
  temporality (`sum_over_time` for a delta Sum, `rate()` / `increase()` for a
  cumulative Sum), not a uniform `rate()`; for a queue, pair inbound vs. outbound
  counters and add the backlog gauges (depth, age).
- **Saturation / capacity.** Utilization gauges (`solidgauge` for a bounded %
  against a threshold, `line` for a trend): CPU, memory, disk/queue depth, credit
  balances; a `table` of the resources closest to their limit. An empty series
  often means "not applicable," not zero.
- **Incident review (a fixed past window).** Open with a `markdown` summary, set a
  **fixed** `scope.timeRange` (absolute start and end) on the dashboard so every
  panel covers the incident, then golden-signal charts plus a `spans` panel — or
  a `dashboard` panel with `source.id: "trace-details"` — for the offending path.
  The fixed scope is what lets a pinned trace resolve at all, not a nicety of
  presentation — such a `dashboard` panel outside it reports the trace as not found.
  Find the offending trace with a span query per
  [sql-logs-traces.md](query/sql-logs-traces.md) and pin its `traceId` in that
  panel's `source.params.traceConfig`.
- **Multi-service overview (a fleet or whole space).** One compact row per service
  — a `number` KPI plus a small `line` — under a `divider` per service, or a single
  wide `table` ranking all services by health. Prefer a `table` or `topk(...)` over
  a `line` with more than ~25 series.

One query per `explore` panel. To compare two signals, emit two side-by-side
panels (section 5), never one overlaid panel. If the intent is genuinely one
number, a single `number` panel is a complete answer — don't pad it.

---

## 3. Ground every panel query

Ground each data panel's query against live telemetry **before** you write the
body. The query languages themselves are deferred to
[sql-logs-traces.md](query/sql-logs-traces.md) and
[promql-metrics.md](query/promql-metrics.md); this section is the discipline.

**The grounding loop, per data panel:**

1. **Discover what exists.** Run discovery queries against the Space — don't guess
   service names, field names, or metric names.
   - Logs and traces: `EXPLAIN (ANALYZE_FIELDS)` or `SELECT `@record` … LIMIT 10`,
     narrowed to the service and telemetry type (see *Schema Discovery* in
     [sql-logs-traces.md](query/sql-logs-traces.md)). Interactive discovery queries
     **need** a `` `@timestamp` `` bound — only the final panel query is windowless
     (below).
   - Metrics: enumerate the real metric names and their identity labels per
     [promql-metrics.md](query/promql-metrics.md) — which metrics a service emits,
     each metric's instrument type (Sum / Gauge / Histogram) and temporality
     (delta / cumulative), and the labels that scope it.
   - Follow a service's dependencies (the services it calls, from span data) to
     find neighbors worth a panel.
2. **List the real names.** For each signal a panel will chart, list the actual
   metric / field / label names for that data set and signal kind (LOGS, TRACES,
   METRICS). Use those names **exactly** — don't shorten, alias, pluralize, or
   invent them. Nouns in the request ("throttles", "5xx", "timeouts") describe
   *what to aggregate*, not field names — map them to a real grounded name.
   **Metric names are emitter/OTel-specific — they are NOT Prometheus
   conventions.** Do not assume `requests_total`, `http_requests_total`, `*_total`,
   `*_seconds`, `*_count`, or `*_bucket`; a PromQL selector MUST name a metric from
   the grounded list verbatim. If the list lacks the metric you wanted, it does not
   exist here — use a fallback surface (step 4), never a guessed name.
   **Pick the SOURCE per signal by a strict 3-tier priority — reach for a lower
   tier only when no higher tier grounds the signal:** (1) the service's OWN domain
   metric it emits (a business counter / gauge / histogram) FIRST; (2) a generic
   OTel metric (e.g. `Count` / `Errors`) as a fallback; (3) a trace-span / log
   derivation (span `durationNano` percentiles, log counts) LAST.
3. **Pick the surface and set `queryLanguage` to match.** Metrics → **PromQL**; log
   or trace content → **SQL**. Metrics execute as NATIVE PromQL
   (`queryLanguage: "promql"`), NEVER SQL — there is **no** `metrics.default` SQL
   table; a SQL `FROM` is only `default`, `logs.default`, or `traces.default`. Set
   the panel's `queryLanguage` explicitly — it is **not** inferred from the query
   text, and a mismatch is a silent failure.
4. **Fall back to a lower tier before dropping.** The same RED signal can come from
   more than one place: request rate, error rate, and latency can be derived from
   **TRACES/spans (SQL)** — counts, error status, duration percentiles — and
   error/volume counts from **LOGS (SQL)**, not only from METRICS. If the metric you
   wanted is not grounded, re-express the SAME panel from a lower-tier surface
   rather than dropping it. A service with sparse metrics but rich traces/logs
   should still yield a FULL dashboard.
5. **Self-check the draft.** Scan every field / metric / label reference in the
   drafted `queryString` against the grounded lists. If a reference cannot be
   confirmed, re-ground; if it still cannot — on any surface — **drop that panel**
   rather than shipping an invented query. Optionally run the final windowless
   query once with a temporary time bound added, to confirm it returns rows.

**Bind panels to the grounded metric identity.** Vended AWS metrics carry an
instrumentation scope (`@instrumentation.@name="cloudwatch.aws/<service>"`) plus a
datapoint attribute (e.g. `FunctionName`); OTLP-native signals carry
`@resource.*` labels. Select the metric by name — use an `{__name__="<exact>"}`
matcher for a name that starts with a digit or contains `%`, `.`, or spaces (e.g.
`4xxErrors`, `traces.span.metrics.*`); a plain non-dotted name may be used bare.
A bare Prometheus `job=` / `service=` selector is an ungrounded reference — drop
panels that use one. The per-service scope + datapoint-attribute matrix lives in
[promql-metrics.md](query/promql-metrics.md).

**Pick the aggregation from the metric's type.** Read it off the grounded
instrument type and temporality — `sum_over_time(<m>[w])` for a DELTA Sum,
`rate()` / `increase()` for a CUMULATIVE Sum, `avg()` / `max()` for a Gauge, and
`histogram_quantile(0.99, rate(<base>[5m]))` for a Histogram (use the BASE metric
name — NO `_bucket`, NO `by (le)`). Never label an average as a percentile, and
never chart a counter as a raw value.

**Two grounding cautions:**

- **Keep every panel query WINDOWLESS.** Never embed a time window in the
  `queryString` — no `` `@timestamp` `` bound, no `NOW() - INTERVAL`, no PromQL
  range that pins an absolute time. The panel (or dashboard) `scope` drives the
  time range; a hard-coded window fights the time picker and freezes the panel.
  (This is the one place a dashboard query differs from an interactive one, which
  *requires* a time bound — so don't copy an interactive query verbatim.)
- **An empty series is often correct.** A signal that only exists under a specific
  configuration (an agent installed, request metrics enabled, a burstable
  instance) returns nothing when that condition is absent. Confirm a metric is
  expected to have data before making it a headline panel.

---

## 4. Author the panel bodies

This is the round-tripping JSON schema — the format a user hand-edits in the
source editor and the one the UI persists. Every field here survives save → reopen.
In a body you author now, a key that is not in the schema at the **root or panel**
level is rejected at save (see "API save semantics"), not dropped; a key inside
`config` is accepted but ignored. Only a **pre-existing** legacy body (stored before
the schema was enforced) has non-schema keys dropped or rewritten on read. The whole
object is what you serialize into the API's `body` string (section 7).

### Top-level shape

```jsonc
{
  "panels": [ /* Panel[] — required, may be empty */ ],
  "scope": { "timeRange": { "start": "now-3h", "end": "now" } }  // optional
}
```

- **`panels`** is the only required key. Panels sit on a **fixed 60-column grid**;
  the column count is not a field and is not configurable.
- **`scope.timeRange`** is the window the whole dashboard opens under. Omit it to
  inherit whatever range is already active.

> There is no `views`, no `layout: { columns: 60 }`, no per-panel `id`, no
> `refreshInterval`, and no top-level `owner`. The root and panel objects are
> schema-validated with `additionalProperties` off, so sending one of these is **not**
> silently dropped — it **fails the save with a `ValidationException`** (see "API save
> semantics"). Author new bodies in the `panels[]` form only. A body authored under the
> current schema round-trips **exactly**: `GetOmniDashboard` returns its `body`
> **byte-for-byte** (stored verbatim as an opaque string; whitespace and key order
> preserved). Only a **pre-existing legacy** body is upgraded on read (see the carve-out
> above), so those do not round-trip byte-for-byte.

### Panel

```jsonc
{
  "type": "explore",                 // required — selects the panel and its config shape
  "layout": { "x": 0, "y": 0, "w": 30, "h": 320 },  // required — rejected at save if absent (section 5)
  "title": "Read IOPS",              // optional
  "description": "…",                // optional, one line
  "config": { /* shape depends on type */ },
  "scope": { "timeRange": { "start": "now-7d", "end": "now" } }, // optional per-panel override
  "variant": "transparent"           // optional — drop the card frame (headings, spacers, prose)
}
```

Both `type` and `layout` are required — a panel missing either is rejected at save
with a `ValidationException` (`panels[N].type` / `panels[N].layout` `required`). The
renderer does not reflow overlaps (section 5), so give every panel a deliberate
`layout` to keep placement deterministic. An untitled panel takes
a heading from its content. `variant` accepts `"transparent"` (drops the card frame)
or `"default"` (explicit no-op) — omit the key for the default card frame. `variant`
is a panel-level enum, so any other value is rejected at save with a
`ValidationException` (see "API save semantics" below), not silently blanked; other
`variant` values you read back in product-authored bodies are private — do not author
them.

### Scope and time range

`scope.timeRange` has the same shape at the dashboard and panel level; a panel's
scope overrides the dashboard's for that panel only (e.g. a 7-day baseline beside a
1-hour detail view).

```jsonc
"timeRange": { "start": "now-1h", "end": "now" }               // rolling window
"timeRange": { "start": "2026-01-01T00:00:00Z", "end": "now" } // growing window (since incident)
"timeRange": { "start": "2026-01-01T00:00:00Z",
               "end":   "2026-02-01T00:00:00Z" }               // fixed window
```

Each bound is independently an ISO-8601 instant, the literal `now`, or a relative
`now-<N><unit>`. Units: `m` minutes, `h` hours, `d` days, `w` weeks, `mo` months.
There is no seconds or years unit and no rounding syntax — use an absolute instant
for a calendar boundary. The scope drives a query's time range; never embed a
window in a `queryString` (section 3).

### Panel types

`alerts` · `explore` · `markdown` · `divider` · `spans` · `application-map` ·
`dashboards-list` · `investigation-list` · `dashboard` · `agent-kpi-strip` ·
`recent-error-traces-table` · `agent-health-table` · `agent-playground`. **Author
only these thirteen** — this is the public authoring contract. `dashboard` and
`agent-playground` are full-bleed surfaces (public and hand-authorable, but not
offered for free composition; a `dashboard` panel is always `x: 0, w: 60`). There
is no `navigation` type. `alert-detail` and `trace-details` are **withdrawn**: a
saved body that already holds one still reads back, but do not author them in a
new body — see below for the `dashboard`-panel replacement. Other product-authored
types you read back are private; leave them as they are and do not author new ones.
A **missing** `type` is rejected at save (`panels[N].type` `required`, HTTP 400). An
**unrecognised** `type` VALUE is not caught by the schema — it saves with HTTP 200 and
blanks the canvas at render (see "API save semantics" below).

#### explore — charts and query results (the workhorse)

```jsonc
"config": {
  "queryString": "sum by (FunctionName) (sum_over_time({__name__=\"Invocations\", \"@instrumentation.@name\"=\"cloudwatch.aws/lambda\"}[5m]))",
  "queryLanguage": "promql",      // 'sql' | 'promql' — REQUIRED; SET IT EXPLICITLY; not inferred; a mismatch silently blanks the panel
  "visualization": "line",         // REQUIRED; one of the nine below; missing or unknown silently blanks the panel (no 'table' fallback)
  "autoRun": true,                 // run on open — set this on any data panel
  "promqlQueryOptions": { "step": 300 },  // PromQL step in SECONDS (default 60)
  "chartOptions": { /* appearance — see below */ }
}
```

- **Always set `queryLanguage` explicitly** — it is **not** inferred from the query
  text; a mismatch is a silent failure. The telemetry source is the query's `FROM`
  clause (SQL) or metric selector (PromQL); there is no separate source field.
- **Always set `autoRun: true`** on a panel meant to show data without a click —
  otherwise it opens showing its query, not its result.
- **Match `step` to the range vector.** A `[5m]` window sampled at the default 60s
  over-samples 5×; set `promqlQueryOptions.step` to `300` (SECONDS).
- **`inputMode`.** Omit for a chart tile (the default). Set `"inputMode": "editor"`
  **only** when authoring a full Explore-surface panel — that is the one accepted
  value, and it is load-bearing on read (a saved Explore surface with `inputMode`
  dropped reopens as a tile).
- The `@`-labels are double-quoted PromQL selectors, so they are JSON-escaped
  (`\"`) once more inside the `queryString`; and because the whole body is itself
  passed to the API as a JSON **string**, it is escaped a third time on the wire —
  build the body as an object and let your JSON library serialize it (section 7)
  rather than hand-escaping.

#### markdown — formatted text

```jsonc
"config": { "content": "## On call\n\n1. Check error rate.\n2. Open the failing trace." }
```

Pair with `"h": "auto"` and `"variant": "transparent"` for prose that reads as
part of the canvas.

#### divider — collapsible section heading

```jsonc
"config": { "title": "Service health", "collapsed": false }
```

Always full width (`x: 0, w: 60`); give it a small fixed height (`40`). Groups the
panels beneath it, down to the next divider, into a collapsible section.

#### alerts

```jsonc
{ "type": "alerts", "config": { "state": "CRITICAL" } }  // OK | WARNING | CRITICAL | NODATA (the Omni alert states); omit for all. NOT the classic CloudWatch alarm states `ALARM` / `INSUFFICIENT_DATA` (and `ALERT` is not a state at all) — an unknown state value silently blanks the canvas
```

**One alert's detail** is not a panel type of its own any more — `alert-detail` is
withdrawn (readable, not authorable). Author a `dashboard` panel instead:

```jsonc
{ "type": "dashboard",
  "config": {   // `source` must sit INSIDE config — at panel level the save is rejected
    "source": { "kind": "system", "id": "alert-details", "version": 1,
                "params": { "alertName": "checkout-error-rate-high", "alertId": "6c89…" } } },  // alertName required; alertId optional
  "layout": { "x": 0, "y": 0, "w": 60, "h": "auto" } }
```

The required parameter is the alert's **name**, but alert names are **not** unique
within a space — two alerts can share one name. The stable identity is the
`alertId` (see [alerts.md](alerts.md)); pass it too so the panel is unambiguous.
Resolve a name to its `alertId`s with `ListAlerts` (`filterCriteria.names`) first,
and rename one via `UpdateAlert` if two collide.

#### application-map / spans

Both take a public `config`; either also renders correctly with no `config` and
picks up the dashboard's time range.

```jsonc
{ "type": "application-map", "config": { "focusServiceName": "checkout" } }  // opens focused on one service; a one-time opening intent the console drops on its next save
{ "type": "spans", "config": { "lens": "application",   // application | agent (default application)
                               "grain": "traces",       // application lens: traces | spans (default traces)
                               "agentGrain": "traces",  // agent lens: traces | sessions (default traces)
                               "agentName": "…" } }     // agent lens only: scope to one agent; absent = all
```

An absent or unrecognised selector falls back to its default. Everything else —
service/operation/status/duration/trace-id filters, camera, node selection, rail
state — is per-user console state, not saved.

#### dashboards-list

`"config": {}` — a list of the Space's saved dashboards.

#### one trace's detail — a `dashboard` panel, not `trace-details`

`trace-details` is a **withdrawn** panel type: a saved body that holds one still
reads back, but do not author it in a new body — like any unrecognised `type` it
saves with HTTP 200 and blanks the canvas. Author a `dashboard` panel whose
`source.id` is `trace-details`:

```jsonc
{ "scope": { "timeRange": { "start": "2024-06-01T14:00:00Z", "end": "2024-06-01T15:00:00Z" } },
  // ^ load-bearing: the trace is looked up inside this window (see below)
  "panels": [
    { "type": "dashboard",
      "config": {   // `source` must sit INSIDE config — at panel level the save is rejected
        "source": { "kind": "system", "id": "trace-details", "version": 1,
                    "params": { "traceConfig": {
                      "traceId": "…",                                    // required
                      "initialMode": "waterfall",  // waterfall | graph | flame | raw (`flamegraph` is a legacy alias read as `graph`)
                      "focusSpanId": "…" } } } },  // optional — deep-link to a specific span
      "layout": { "x": 0, "y": 0, "w": 60, "h": "auto" } } ] }
```

**The dashboard's own `scope.timeRange` is what finds the trace — give it one that
covers the trace's timestamps.** Whatever window the viewer is on is the window the
lookup uses, so a trace older than that window reports `Trace not found — This trace
has no spans in the selected time range` even though the id is right and the spans are
retained. Pin an absolute `start`/`end` around the incident, exactly as the
Incident-review recipe above does; a relative default such as the viewer's last 30
minutes will lose the trace as soon as it ages past it.

`traceConfig` may also carry `startTime`/`endTime` (epoch **milliseconds**), and a body
that already has them still saves and reads back — but they do not select the lookup
window and setting them is not a substitute for the scope. A pinned trace is useful only
while it is still retained; prefer a `spans` panel for a durable dashboard.

### Visualizations — exactly nine legal values

`line` · `area` · `bar` · `scatter` · `pie` · `table` · `number` · `solidgauge` ·
`heatmap`.

> `stacked-bar`, `single-metric`, and `donut` are **not** valid — use `bar` with
> `chartOptions.plotOptions.style.barOptions.stacked: true` (there is no
> `plotOptions.stacked`; see Chart options), `number`, and `pie` respectively. `visualization` is REQUIRED and
> there is no fallback default that renders: a missing value, an unknown value such
> as `stacked-bar`, or the empty string `""` saves with HTTP 200 but silently blanks
> the panel at render.

**`number` tile trap (known product bug).** A `number` panel whose SQL result set has
**exactly one column** renders the ROW COUNT of that result set instead of the value,
and labels it with the literal legend `value` rather than the column alias. Measured on
a live Space: a one-column query returning 5 rows renders `5`, one returning 3 rows
renders `3`, and any single-row scalar aggregate therefore renders `1` —
`SELECT max(severityNumber) AS m` and `SELECT count(*) AS n` both show `1` while the
same queries return `9` and `810972`.

The trigger is the **single column**, not `COUNT(*)`, so do **not** try to fix this by
making the aggregate the only or first selected column — that is the shape that breaks.

- **A single scalar KPI belongs in a `table` panel.** Verified: the same two queries in
  `table` panels render `9` and `810972` under their own aliases. This is the only form
  that reliably shows the value you asked for.
- **A trend-shaped KPI works in a `number` panel** because the bucket column makes the
  result two columns: `SELECT date_trunc('minute', \`@timestamp\`) AS t, count(*) AS n …
  GROUP BY t ORDER BY t` renders the latest bucket's value, legend `n`.
- Adding a second column does make a real value appear, but **which** column it reads is
  not predictable from the query, so do not build a KPI tile on it: measured,
  `SELECT max(…) AS m, count(*) AS n` rendered `n`, `SELECT count(*) AS n, 0 AS pad`
  rendered `n`, and `SELECT max(…) AS m, 'x' AS pad` rendered `m`.

Pick by signal kind:

| Signal | Visualization |
|--------|---------------|
| Time series, ≤ ~25 series, comparable units | `line` |
| Additive series / discrete buckets (2xx/4xx/5xx, per-AZ) | `bar` (stacked) or `area` |
| Correlation point cloud (no connecting line) | `scatter` |
| One number, with a delta | `number` |
| Resources ranked by a metric | `table` (default sort: descending on the primary signal) |
| Genuine part-of-whole | `pie` |
| A bounded ratio against thresholds (availability %) | `solidgauge` |

Two shapes authors most often get wrong — check them explicitly: a **scalar
aggregate** (one number, no `by` / no `GROUP BY` over time) should be a `number`
tile, not a table; a **top-N ranking** (`ORDER BY … LIMIT` / `topk(...)`) should be
a `table`, not a line.

Anti-patterns: no `line` past ~25 series (use `topk(...)` or a `table`); don't
stack a `bar` whose series go negative; don't put two units on one chart — split
into two `w: 30` panels. `heatmap` does not yet ingest histogram-bucket output —
for a latency distribution use `line` with a `histogram_quantile`.

### Chart options

`chartOptions` sets appearance and is a discriminated union keyed on `view`, which
must match the panel's `visualization` (when both are present, `view` wins — so
setting only `visualization` is the simpler, recommended form).

```jsonc
"chartOptions": {
  "view": "line",
  "title": { "text": "Requests/sec", "show": true },
  "plotOptions": {
    "legend": { "show": true, "position": "bottom" },
    "yAxis": [ { "min": 0, "title": "req/s" } ]   // TUPLE of 1 or 2 axes (2nd = right-hand)
  }
}
```

- **Cartesian** (`line`/`area`/`bar`/`scatter`): `legend`, `stacking`, `xAxis`, and
  `yAxis` as a **tuple** of one or two axes (the second is the right-hand axis). Use
  `type: 'datetime'` for a time axis (not `'time'`).
- **`solidgauge`**: `yAxis` is an **object** `{ min, max }` (not a tuple), plus
  `plotBands: [{ from, to, color }]`.
- **`heatmap`**: `xAxis`/`yAxis` with `categories`, and `colorScale`.
- **`table`**: `hiddenColumns`, `summaryColumns` (`MIN`/`MAX`/`SUM`/`AVG`),
  `layout` (`horizontal`/`vertical`), `stickySummary`, `showTimeSeriesData`,
  `formatJson` (`true` | `'raw'` | `'raw-single-line'`). Column order is **not** a
  chart option — the console does not save it. `number` and `pie` take little
  beyond the base fields.
- **Stacking depends on the view.** For `line`/`area`, set
  `plotOptions.stacking: "normal"` (or `"percent"`) — an **enum**, not a boolean,
  keyed directly on `plotOptions`; it wins over the equivalent
  `plotOptions.style.lineOptions.stacked: true`. A **`bar`** panel does not read
  `stacking` at all — the only stacking field the bar renderer reads is
  `plotOptions.style.barOptions.stacked: true`. There is **no** `plotOptions.stacked`
  field; a flag placed there is an unknown `config`-level key — it saves without error
  but is ignored at render (it survives read-back yet never takes effect), so the
  chart will not stack.

### API save semantics — what happens to a body on read/write

The save API **schema-validates the body's structure, its root/panel keys, each
coordinate FIELD's bounds, and the `variant` enum** — but not cross-field invariants or
the other enum values (`visualization`, panel `type`, `alerts` state). A body
fails in three distinct ways, and **only the first errors**:

- **Rejected at save (HTTP 400 `ValidationException`).** The root and panel objects
  have `additionalProperties` **off**, so an unknown or misspelled key there — a
  top-level `refreshInterval` / `owner` / `layout`, a per-panel `id`, a misspelled
  `title` — fails with `does not satisfy the 'additionalProperties' constraint`. The
  schema also rejects: a bad **`variant`** value (`enum` constraint); a missing or
  non-array `panels` (`required` / `type`); a panel missing its required `type` or
  `layout` (`required`); and an out-of-bounds coordinate FIELD — `layout.x` or
  `layout.y` below 0, or `layout.w` below 1 (`minimum`). The dashboard is **not**
  created or updated.
- **Accepted then blank (HTTP 200).** A value the schema does NOT check persists with a
  `200` and a `dashboardId`, then fails at render — the blast radius depends on the kind:
  an unknown `visualization` VALUE blanks **that panel** while its siblings render;
  everything else the schema does not catch — an unrecognised panel `type` VALUE, a bad
  `alerts` `state`, the cross-field `x + w > 60` overflow (each field is in bounds, but
  their sum is not validated), a `w > 60` (no maximum on `w`), a bad `h` unit (`80vh`, a
  fraction), or a `now-30s` / `now-1y` bound (no seconds/years unit) — blanks the
  **WHOLE dashboard** (valid siblings included). A `200`
  proves only that the body parsed and passed the schema, not that it renders.
- **Accepted then ignored (HTTP 200, renders fine minus that key).** An unknown or
  misspelled key **inside `config`** (or its nested objects like `chartOptions`) is
  stored, comes back on read, and simply never takes effect — this is the one place
  the "a field I set doesn't take effect, with no error" behavior lives (e.g. a
  British `visualisation`, or a `plotOptions.stacked` that should be
  `plotOptions.stacking` for line/area or `plotOptions.style.<view>Options.stacked`
  for bar).

Consequences:

- **Do not rely on a 200 to confirm renderability.** Validate the body against this
  reference before the save call; a `200` + a `dashboardId` proves only that the JSON
  parsed and passed the schema (structure, keys, and per-field bounds), not that it
  renders.
- **Structural / key / field-bound / `variant` errors are loud; other enum-value and
  cross-field errors are silent.** The schema catches missing structure, unknown keys,
  per-field coordinate bounds, and the `variant` enum with a `ValidationException`; it
  does NOT catch an unknown `visualization` value — which blanks that one panel — or a
  bad panel `type` /
  `alerts` `state`, the `x + w <= 60` sum, and bad `h` units — which blank the whole
  canvas — all only at render.
- **`layout` is required on every panel** — a panel with no `layout` is rejected at
  save (`panels[N].layout` `required`), not drawn full-width.
- **Read-back is byte-for-byte for a current-schema body.** The `body` is stored
  verbatim as an opaque string; `GetOmniDashboard` returns it exactly as written
  (whitespace and key order preserved), so it round-trips exactly. (Only a pre-existing
  legacy body is upgraded on read and so does not round-trip; section 6.)
- **`queryLanguage` mismatches are silent failures**, not errors. Set it per `explore` panel.
- **Dashboard names** (the API's `name`, not part of the body) must be 1–256 chars
  matching `^[a-zA-Z0-9_.@~()-]+$`. Nothing is trimmed: surrounding whitespace,
  spaces, `/`, `:`, `#`, `+`, accented letters and emoji all fail the pattern, and
  257+ chars fails the length check. The body itself must be 1–1,048,576 bytes
  once serialized.

---

## 5. Lay out on the 60-column grid

Assign each panel's `layout: { x, y, w, h }` so panels tile cleanly with no overlap
and no overflow. This is the single most error-prone part of a body.

> **ALWAYS STATE when laying out a grid — two different failure modes.** (1) A
> per-field bound violation is **rejected at save** (`ValidationException`, HTTP 400):
> a negative `x` or `y`, or a `w` below 1, trips the schema's `minimum` constraint and
> the save does not succeed. (2) The cross-field `x + w > 60` overflow, a `w > 60`, and
> a bad `h` (a fraction or a `vh`/`%` unit) are **NOT** validated: they save with HTTP
> 200 and a `dashboardId` and then render the entire dashboard — every otherwise-valid
> sibling panel included — as a blank page, with no error. Nothing enforces the
> `x + w <= 60` sum, and the renderer does not reflow or auto-arrange overlaps. Keep
> `x + w <= 60` on every panel yourself; a `200` is not proof the layout renders.

### The two-unit rule (memorize this)

`x`/`w` and `y`/`h` do **not** share a unit:

| Field | Unit | Rule |
|-------|------|------|
| `x` | grid **column** | integer `0`–`59`; `x: 0` is the left edge. A **negative** `x` is **rejected at save** (`minimum`, HTTP 400) |
| `w` | grid **columns** | integer, `w >= 1` is **enforced at save** (`w: 0` is rejected, `minimum`, HTTP 400); there is **no** enforced maximum (`w: 61` saves). **`x + w` must not exceed `60`**, but that sum is **not** enforced — an overflow saves with HTTP 200 and blanks the canvas |
| `y` | **pixels** | integer pixels from the top, not a row index; a **negative** `y` is **rejected at save** (`minimum`, HTTP 400) |
| `h` | **pixels** or `'auto'` | integer pixels within roughly `100`–`2000` (`divider` `40` is the sanctioned sub-100 row), or the literal `'auto'`; never fractional and never a relative CSS unit (`vh` / `%` / `vw`) — those are **not** rejected at save, they save with HTTP 200 and blank the canvas at render |

Mixing the units (treating `y` as a row index, or `w` as pixels) is the most common
authoring bug. Prefer `'auto'` for content-sized panels (markdown, lists, nested
`dashboard` panels) and a fixed pixel height where the content needs one; a relative
CSS unit such as `'80vh'` is an unknown value that blanks the canvas.

### Width conventions

Every row MUST match one of the width-per-count patterns below. Author the correct
shape yourself — nothing re-shapes it for you.

| Row count | Width per panel (`w`) | Use for |
|-----------|-----------------------|---------|
| 1 panel | `60` | full-width strip (hero chart, table, status strip, divider, application-map, `dashboard`) |
| 2 panels | `30` each | two halves side-by-side (most time-series pairs) |
| 3 panels | `20` each | three thirds (read/write/idle, 2xx/4xx/5xx) |
| 4 panels | `15` each | four quarters (KPI row) |

**Hard rules — all MUST:**

- A row's widths sum to **EXACTLY 60**, with `x` as the cumulative sum from 0
  (`0, 30`; `0, 20, 40`; `0, 15, 30, 45`).
- Every panel in a row shares the **same `y`** AND the **same `h`**.
- **Never more than 4 panels in one row.** A 5th starts a new row below: 5 panels =
  3 then 2; 6 = 3+3; 7 = 4+3. Five across does not fit the 60-column grid and the
  overflow lands on top of its neighbours.
- **No mixed-width rows** except an intentional hero-supporting layout, which uses
  TWO rows: row 1 is one `w: 60` hero, row 2 is `w: 30`×2 or `w: 20`×3 supporting
  panels — never side-by-side with the hero.
- **No heterogeneous chart/table rows.** A `line` chart and a `table` on the same
  row read badly — split them into two rows (chart row above, table row below).

### Stacking rows with `y`

`y` is the pixel offset from the top and it orders rows. Give successive rows
**increasing `y`** equal to the running sum of prior row heights:

- KPI row (`h: 120`) at `y: 0`
- first chart row (`h: 320`) at `y: 120`
- next chart row at `y: 440`, and so on.

When a row uses `'auto'` heights, still give the next row a larger `y`; the renderer
resolves final positions from each row's height, so approximate but
monotonically-increasing `y` values are fine.

### Height guidance by panel kind

| Panel kind | Suggested `h` |
|------------|---------------|
| `number` KPI tile | `120` (short — one figure) |
| Time-series (`line`/`area`/`bar`/`scatter`) | ~`320` |
| `table` | ~`320`–`400`, taller for many rows |
| `pie` / `solidgauge` | ~`280`–`320` (roughly square reads best) |
| `divider` | `40` (always full width, `x: 0, w: 60`) |
| `dashboard` (nested system dashboard) | `'auto'` (always full width, `x: 0, w: 60`) |
| `markdown` runbook | `'auto'` (pair with `variant: 'transparent'`) |
| `application-map` / `spans` | full width, tall (~`480`) |

### Layout shortcuts

- **Full-width strip** — `{ x: 0, y: <row>, w: 60, h: <h> }`.
- **Two halves** — `{ x: 0, …, w: 30 }` and `{ x: 30, …, w: 30 }`, same `y`.
- **Three thirds** — `x: 0`, `x: 20`, `x: 40`, each `w: 20`, same `y`.
- **KPI row of four** then a wide chart — four tiles at `y: 0`, `w: 15`,
  `x: 0/15/30/45`, `h: 120`; one chart at `y: 120`, `x: 0`, `w: 60`, `h: 320`.
- **Divider-led section** — a full-width `divider` (`h: 40`) at the row's `y`, then
  the section's panels at a larger `y` beneath it.

### Overflow and overlap

- A panel that overflows the grid (`x + w > 60`) is never reflowed into what you
  meant — it saves with HTTP 200 and blanks the whole canvas (section 4, API save
  semantics). Keep `x + w <= 60` on every panel yourself.
- The renderer does not reflow overlaps. Two panels sharing a `y` must not share
  columns — give them non-overlapping `[x, x+w)` ranges. Follow the
  width-per-count patterns above and give each row a larger `y` and you will never
  overlap.

---

## 6. Troubleshooting

### A panel rendered empty

Work down this list in order; the first match is usually the cause.

1. **An ungrounded name.** A metric, field, or label that does not exist returns
   nothing, not an error. Re-run discovery (section 3) for exactly the names in
   the `queryString`; a `*_total` / `*_seconds` Prometheus-convention name, or a
   bare `job=` / `service=` selector, is the CloudWatch tell.
2. **`queryLanguage` mismatch.** A PromQL selector under `"queryLanguage": "sql"`
   (or SQL under `promql`) fails silently. Check the language against the text.
3. **A time window embedded in the query.** A `` `@timestamp` `` bound, a
   `NOW() - INTERVAL`, or an absolute PromQL range pins the panel to a window the
   time picker no longer covers. Remove it; the `scope` drives the range.
4. **The panel's or dashboard's `scope` excludes the data.** A fixed incident-window
   scope, or a per-panel override, can legitimately show nothing outside its range.
5. **`autoRun` is not `true`.** The panel opens showing its query, not its result —
   it looks empty until clicked.
6. **The metric selector's identity is wrong.** A vended metric without its
   `@instrumentation.@name` scope, or the wrong datapoint attribute
   (`FunctionName` vs. `ApiName`), matches nothing. Compare against the archetype
   identity forms and [promql-metrics.md](query/promql-metrics.md).
7. **The series is genuinely empty — and that is correct.** Conditional signals
   (`IteratorAge` for a non-stream consumer, `RunningTaskCount` without Container
   Insights, burst-credit metrics on a non-burstable instance) return nothing when
   the condition is absent. Say so, and either drop the panel or retitle it.
8. **A `step` far coarser than the window.** With a very large
   `promqlQueryOptions.step` over a short scope there may be no evaluation point.

### A field has no effect, or the canvas is blank

- **The field is not in the schema, or its name is misspelled.** Where it lands
  decides the symptom. A misspelled or unknown key **inside `config`** (e.g.
  `plotOptions.stacked`, which should be `plotOptions.stacking: "normal"` or
  `plotOptions.style.<view>Options.stacked`) saves without error and is ignored at
  render — that is the "field has no effect" case. A wrong key at the **root or
  panel level** (`layout.columns`, per-panel `id`, `refreshInterval`, `owner`,
  `views`) does NOT reach this state — it is rejected at save with a
  `ValidationException` (section 4), so its symptom is a failed save, not a
  silently-ignored field.
- **`inputMode` was omitted from an Explore-surface panel.** It reopens as a tile.
- **The whole canvas is blank after a 200.** Nothing was rejected — the save returned
  HTTP 200 — but the body carries a value the schema does not check that breaks the
  whole render: a cross-field `x + w > 60` overflow or a `w > 60`, a bad `h` (fractional
  or a `vh` unit), an unrecognised panel `type` VALUE, a bad `alerts` `state`, a
  `timeRange` bound with a seconds/years unit, a `source` on a non-`dashboard` panel, or
  a `dashboard` panel without one. No sibling renders until it is fixed — validate each
  panel against section 4. (An unknown `visualization` VALUE instead blanks only its own
  panel; see the next bullet. A missing/non-array `panels`, a panel missing `type` or
  `layout`, or a negative `x`/`y` or `w: 0` is **rejected at save** with a
  `ValidationException`, not this silent-blank symptom.)
- **A panel is blank but the rest render.** A missing or invalid `visualization` VALUE
  (`stacked-bar`, `donut`, `single-metric`, `""`; there is no `table` fallback) blanks
  that panel alone while its siblings render.
- **A pre-existing legacy body was rewritten on read.** In bodies stored before the
  schema was enforced, `views` became `panels` and old relative ranges became
  `{ start, end }` on read — that is the read-time upgrade of grandfathered bodies,
  not data loss. It does NOT apply to a body you author now: sending `views` (or any
  unknown root/panel key) in a new body is rejected at save with a `ValidationException`
  (see "API save semantics"), not upgraded.

### Reading back a body (pasted into chat, or fetched with GetOmniDashboard)

1. **Get the JSON.** From the API, `omniDashboard.body` is a JSON **string** —
   parse it before inspecting it. From a paste, strip any surrounding prose.
2. **Check the top level.** `panels` must be an array; note any `scope.timeRange`
   (a fixed absolute window means an incident dashboard).
3. **Walk each panel** and describe it in the user's terms — what it shows, from
   which telemetry, over what range — while checking, for each:
   `type` is one of the thirteen authorable types; `layout` uses the two units correctly
   and stays inside 60 columns; rows share `y` and `h`; for `explore`,
   `queryLanguage` matches the query text, `visualization` is one of the nine, the
   query is windowless, and every name in it is plausible for the grounded catalog;
   `chartOptions.view` matches `visualization`; no unknown keys.
4. **Explain the two silent failures** (an unknown key inside `config` that is stored
   but ignored at render; `queryLanguage` not inferred) whenever you find an instance
   of either, because the user will not have seen an error. An unknown root- or
   panel-level key cannot appear in a read-back body from a current-schema save — it
   would have been rejected at write with a ValidationException.
5. **Do not narrate the schema when the user asked what the dashboard shows** —
   lead with the panels' meaning; surface the defects as a short list after.

---

## 7. API reference

Five operations manage dashboards. They are exposed by the CloudWatch Omni service
(endpoint prefix `cloudwatch-omni`, SigV4 signing name `cloudwatch` — see
[programmatic-access.md](programmatic-access.md)) and reachable from the AWS CLI as
`aws cloudwatchomni <kebab-case-operation>`, from every AWS SDK, or through an AWS
MCP `aws___call_aws` tool with the same operation and parameter names.

**Common to all five:**

- **Space-scoped.** `spaceId` (the Space's UUID) is required on every call. There
  is no account-wide list of dashboards — you list per Space.
- **The identity is `dashboardId`**, a system-minted lowercase UUID
  (`^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$`). The
  caller-supplied `name` is metadata, is **not** unique, and is **not** part of the
  id — renaming never changes the id or the ARN.
- **ARN:** `arn:<partition>:cloudwatch:<region>:<account-id>:omni-dashboard/<dashboardId>`
  — the `cloudwatch` vendor namespace with an `omni-dashboard/` resource type, the
  same convention as alerts. Neither the Space nor the name appears in it.
- **`body` is a string**, 1–1,048,576 bytes, containing the serialized JSON from
  section 4. The API **schema-validates the root and per-panel objects** on write
  (`additionalProperties` off) and stores the string **verbatim**, returning a
  current-schema body **byte-for-byte** on read (only pre-existing legacy bodies are
  upgraded on read; section 6). An API `ValidationException` on `body` can therefore be a
  length/type problem OR a body-schema one — an unknown root/panel key, a bad `variant`
  enum, a missing/non-array `panels`, a panel missing `type`/`layout`, or an
  out-of-bounds coordinate field (negative `x`/`y`, `w: 0`) (section 4). What is NOT
  enforced at the API — a bad enum VALUE (`visualization`, panel `type`), the
  `x + w > 60` sum, or unknown `config` keys — surfaces only when the UI renders.
- **`name`:** 1–256 chars, `^[a-zA-Z0-9_.@~()-]+$` (section 4's rule).
  **`description`:** 1–1024 chars, optional.
- **Errors** every operation can return: `ValidationException`,
  `AccessDeniedException`, `InternalServerException`, `ThrottlingException`.
  Per-operation additions are noted below.
- **Authorization** is the Space's access grants, exactly as for a console user. A
  correctly-signed call returning `AccessDeniedException` usually means the caller
  has no grant on the Space — check that before suspecting the request. The IAM
  action names are the operation names (`cloudwatch:CreateOmniDashboard`, etc.).

### CreateOmniDashboard

Creates a dashboard in a Space. Returns the full **`omniDashboard`** — read
`dashboardId` and `arn` from it; you do not need a follow-up `GetOmniDashboard`.

| Input | Required | Notes |
|---|---|---|
| `spaceId` | ✔ | The Space to create the dashboard in |
| `name` | ✔ | Display name, 1–256 chars, `^[a-zA-Z0-9_.@~()-]+$`. Not unique |
| `body` | ✔ | The serialized `{ "panels": [...] }` JSON, 1–1,048,576 bytes |
| `description` | | 1–1024 chars |
| `tags` | | Key/value map |
| `clientToken` | | Idempotency token — a retry with the same token returns the original result instead of creating a duplicate. SDKs and the CLI fill it automatically |

Errors beyond the common set: `ServiceQuotaExceededException` (the Space's
dashboard limit), `ResourceNotFoundException` (unknown Space), `ConflictException`.

```bash
# body.json holds the panels[] object from section 4; file:// passes its text as the string value
aws cloudwatchomni create-omni-dashboard \
  --region us-east-1 \
  --space-id 3f2c1e9a-7b4d-4c0e-9a1b-2d3e4f5a6b7c \
  --name checkout-service-health \
  --description "RED golden signals for checkout" \
  --body file://body.json
```

Build `body.json` with a JSON library from the panel objects — do not hand-escape
the nested `\"` in PromQL selectors a second time; the CLI/SDK serializes the string
for the wire.

### GetOmniDashboard

Retrieves one dashboard **including its body**. This is the only way to read a
saved body — list results do not carry it.

| Input | Required |
|---|---|
| `spaceId` | ✔ |
| `dashboardId` | ✔ |

Returns **`omniDashboard`**: `dashboardId`, `arn`, `name`, `body` (a JSON string —
parse it), `createdBy` (the creating principal), `description`, `tags`, `createdAt`,
`updatedAt`. Read-only. Errors beyond the common set: `ResourceNotFoundException`.

```bash
aws cloudwatchomni get-omni-dashboard --region us-east-1 \
  --space-id <spaceId> --dashboard-id <dashboardId> \
  --query 'omniDashboard.body' --output text > body.json
```

### ListOmniDashboards

Enumerates the dashboards in a Space, optionally filtered by name prefix, with
pagination.

| Input | Notes |
|---|---|
| `spaceId` | Required |
| `namePrefix` | 1–256 chars, same character set as `name`; matches names that start with it |
| `maxResults` | 1–100 per page |
| `nextToken` | Pagination token |

**Read the summaries from `items`** and page with `nextToken` — `items` is the only
list member the response carries. Each entry is an
`OmniDashboardSummary`: `dashboardId`, `arn`, `name`, `createdBy`, `description`,
`tags`, `createdAt`, `updatedAt` — **no `body`**. Because names are not unique, a
`namePrefix` (or an exact-name match on the client side) can return several
dashboards; this is the usual way to resolve a name to a `dashboardId` before
`Get`/`Update`/`Delete`. Follow `nextToken` rather than treating the first page as
the complete set. Errors beyond the common set: `ResourceNotFoundException`
(unknown Space).

```bash
aws cloudwatchomni list-omni-dashboards --region us-east-1 \
  --space-id <spaceId> --name-prefix checkout- --max-results 50
```

### UpdateOmniDashboard

Modifies a dashboard in place, addressed by `dashboardId`. **PATCH semantics:**
only the fields you send change; an omitted field is left unchanged. Returns the
full updated **`omniDashboard`**, so you can confirm the result without a second
call. `@idempotent`.

| Input | Required | Notes |
|---|---|---|
| `spaceId` | ✔ | |
| `dashboardId` | ✔ | The dashboard to update |
| `body` | | Full replacement body, 1–1,048,576 bytes. There is no partial-panel patch — send the complete `panels[]` |
| `name` | | Renaming is supported; the id and ARN do not change |
| `description` | | 1–1024 chars |

`tags` cannot be changed through this operation. Errors beyond the common set:
`ResourceNotFoundException`, `ConflictException`, `ServiceQuotaExceededException`.

**To edit one panel:** `GetOmniDashboard` → parse `body` → change the panel →
serialize → `UpdateOmniDashboard` with the whole body. Because a body is replaced,
not merged, an update built from a stale copy silently discards someone else's
edits — fetch immediately before you write.

**Confirm before updating.** Draft the exact change, state that `body` replaces the
whole panel set, and confirm with the user before calling.

```bash
aws cloudwatchomni update-omni-dashboard --region us-east-1 \
  --space-id <spaceId> --dashboard-id <dashboardId> \
  --body file://body.json
```

### DeleteOmniDashboard

Permanently removes a dashboard. Returns an empty response.

| Input | Required |
|---|---|
| `spaceId` | ✔ |
| `dashboardId` | ✔ |

**Idempotent** — deleting a dashboard that has already been removed succeeds
without error (the operation does not return `ResourceNotFoundException`), so a
successful response does not prove the dashboard existed. Deletion cannot be undone;
if the body might be wanted later, `GetOmniDashboard` and keep it first.

**Confirm before deleting.** This is destructive and irreversible. Name the specific
dashboard (id and name) and confirm with the user before calling — never delete on
inference, and because a success does not tell you whether you deleted the one you
meant, confirm the target up front rather than checking after.

```bash
aws cloudwatchomni delete-omni-dashboard --region us-east-1 \
  --space-id <spaceId> --dashboard-id <dashboardId>
```

### End-to-end: build, save, verify

1. Probe: `aws cloudwatchomni list-spaces` — pick the `spaceId` (section 1).
2. Plan, ground, author, lay out (sections 2–5) → a `panels[]` object.
3. Serialize it to `body.json`; `create-omni-dashboard` with a valid `name`.
4. Read back with `get-omni-dashboard`, parse `body`, and run the read-back checks
   in section 6 — the API schema-checks only root/panel keys and per-field bounds,
   so this is where a `config`-level slip (bad enum value, unknown key) would surface.
5. Report the `dashboardId`, `arn`, and a one-line description of each panel.

**CLI says `cloudwatchomni` is not a valid choice.** The local AWS CLI predates
Omni's service model; upgrade it. The failure is client-side argument parsing and
says nothing about whether Omni is enabled. `aws cloudwatch <omni-operation>`
will never work at any version — CloudWatch is a different service that
shares only the signing name.

---

## 8. Appendix — Archetype templates

When the ask names a known resource TYPE, start from its ready panel set below,
then **ground** every query (section 3) and **lay it out** (section 5). Build the
CORE subset by default and offer the optional-depth panels as a follow-up "add
section." Look up each panel's real metric and reading (gauge vs. counter, derived
formulas, whether a real percentile source exists) in
[promql-metrics.md](query/promql-metrics.md); the query strings are TEMPLATES — the
identity form is canonical, but the metric name and statistic MUST be grounded. If
a signal cannot be grounded, **drop that panel** and re-sequence `y` so no blank row
is left.

Every template binds to the `@`-label identity (section 3) — vended AWS metrics on
`@instrumentation.@name="cloudwatch.aws/<svc>"` + datapoint attribute, OTLP /
span-RED on `@resource.*`. Set `queryLanguage` explicitly, `autoRun: true` on every
data panel, and `promqlQueryOptions.step` to match the range vector; keep every
query windowless. `…` in a template stands for the identity labels shown in that
archetype's first row.

### EC2 fleet

Infrastructure resource — **USE / saturation**; scope `cloudwatch.aws/ec2` by
`InstanceId` (the `CWAgent` guest-OS metrics share the identity). Reading rules from the
catalog: the vended EC2 metrics reach this surface as **delta exponential histograms**
(`CPUUtilization`, `StatusCheckFailed*`, `CPUCreditBalance`) and **delta Sums**
(`NetworkIn`/`NetworkOut`), so a max is `histogram_quantile(1, …)`, a min
`histogram_quantile(0, …)`, an average `histogram_sum(…) / histogram_count(…)`, and a
network volume `sum_over_time(…[5m])` — bare `max()`/`avg()` and `rate()` render empty
panels here. Every `StatusCheckFailed*` metric is a per-period **0/1 flag read as a max**
(an average hides a failed check), and `_System` (AWS hardware) is a different fault from
`_Instance` (the guest OS or network) — show them side by side; `CPUCreditBalance` exists
only on burstable (T-family) instances, and near zero at baseline CPU means the instance is
being throttled; `mem_used_percent` / `disk_used_percent` exist only where the CloudWatch
agent is installed (an empty panel means "no agent", so ground it and drop it if absent);
`DiskRead*` / `DiskWrite*` are instance-store only — EBS volume I/O is the `AWS/EBS`
family keyed by `VolumeId`, a separate row if the fleet's disks matter. `__name__` takes no
regex, so a panel that shows several check types is one selector per name joined with `or`
and tagged with `label_replace`. `S` below stands for `"@instrumentation.@name"="cloudwatch.aws/ec2"`.

**CORE (~8 panels — the default build; every query below returns series on an enriched Space, except the two configuration-gated tiles noted inline):**

| Panel | Viz | Query template | Layout |
|-------|-----|----------------|--------|
| Instances failing any check | `number` | `count(histogram_quantile(1, {__name__="StatusCheckFailed", S}) == 1) or vector(0)` — `count()` over an empty match is itself empty, so without `or vector(0)` a healthy fleet renders blank; with it the tile reads `0` both when the fleet is healthy and when the selector matches no instance, so confirm the bare selector returns series before trusting a `0` | `{x:0,y:0,w:15,h:120}` |
| Peak CPU % | `number` | `max(histogram_quantile(1, {__name__="CPUUtilization", S}))` | `{x:15,y:0,w:15,h:120}` |
| Lowest CPU credits | `number` | `min(histogram_quantile(0, {__name__="CPUCreditBalance", S}))` — burstable instances only; drop if empty | `{x:30,y:0,w:15,h:120}` |
| Peak memory % | `number` | `max({__name__="mem_used_percent", S})` — CloudWatch agent only, a plain gauge (switch to `histogram_quantile(1, …)` only if its `__type__` is ExponentialHistogram); drop if empty | `{x:45,y:0,w:15,h:120}` |
| CPU % by instance | `line` | `histogram_quantile(1, {__name__="CPUUtilization", S})` — one series per `InstanceId` | `{x:0,y:120,w:30,h:320}` |
| Status checks by instance and type | `line` | `label_replace(histogram_quantile(1, {__name__="StatusCheckFailed_System", S}), "check", "System", "", "") or label_replace(histogram_quantile(1, {__name__="StatusCheckFailed_Instance", S}), "check", "Instance", "", "") or label_replace(histogram_quantile(1, {__name__="StatusCheckFailed_AttachedEBS", S}), "check", "AttachedEBS", "", "")` — one 0/1 series per instance and check type, never averaged | `{x:30,y:120,w:30,h:320}` |
| Network in/out by instance | `line` | `label_replace(sum by (InstanceId) (sum_over_time({__name__="NetworkIn", S}[5m])), "direction", "in", "", "") or label_replace(sum by (InstanceId) (sum_over_time({__name__="NetworkOut", S}[5m])), "direction", "out", "", "")` — bytes per 5 minutes; append `/ 300` for bytes per second | `{x:0,y:440,w:60,h:320}` |
| Hottest instances | `table` | `topk(10, histogram_quantile(1, {__name__="CPUUtilization", S}))` | `{x:0,y:760,w:60,h:360}` |

**Optional depth (follow-up "infra / USE health"):** `CPUCreditBalance` by instance
(`histogram_quantile(0, …)`, `line`); `disk_used_percent` by instance (`CWAgent`); the EBS
volume side (`VolumeQueueLength`, `VolumeAvgReadLatency` / `VolumeAvgWriteLatency` on
`cloudwatch.aws/ebs` by `VolumeId` — check their `__type__` first). Append two-up and
re-sequence `y`.

### Lambda

Serverless function — RED-shaped; scope `cloudwatch.aws/lambda` by `FunctionName`. The vended
Lambda counters (`Invocations`, `Errors`, `Throttles`) are **delta Sums**, so every count panel
uses `sum_over_time(...[5m])`; `rate()` on them renders `NaN` (see promql-metrics.md section 4).
`Duration` is a gauge with no percentile label; the error-rate tile is the derived
`Errors / Invocations`.

**CORE (~8 panels — the default build):**

| Panel | Viz | Query template | Layout |
|-------|-----|----------------|--------|
| Invocations (5m) | `number` | `sum(sum_over_time({__name__="Invocations", "@instrumentation.@name"="cloudwatch.aws/lambda", FunctionName="<fn>"}[5m]))` | `{x:0,y:0,w:15,h:120}` |
| Error rate | `number` | `sum(sum_over_time({__name__="Errors", …, FunctionName="<fn>"}[5m])) / sum(sum_over_time({__name__="Invocations", …, FunctionName="<fn>"}[5m]))` | `{x:15,y:0,w:15,h:120}` |
| Throttles (5m) | `number` | `sum(sum_over_time({__name__="Throttles", …, FunctionName="<fn>"}[5m]))` | `{x:30,y:0,w:15,h:120}` |
| Concurrency | `number` | `max({__name__="ConcurrentExecutions", …, FunctionName="<fn>"})` | `{x:45,y:0,w:15,h:120}` |
| Invocations over time | `line` | `sum by (FunctionName) (sum_over_time({__name__="Invocations", …, FunctionName="<fn>"}[5m]))` | `{x:0,y:120,w:30,h:320}` |
| Errors over time | `line` | `sum by (FunctionName) (sum_over_time({__name__="Errors", …, FunctionName="<fn>"}[5m]))` | `{x:30,y:120,w:30,h:320}` |
| Duration (ms) | `line` | `{__name__="Duration", …, FunctionName="<fn>"}` | `{x:0,y:440,w:60,h:320}` |
| Recent errors | `table` (SQL) | scope to `/aws/lambda/<fn>` with a grounded log-group filter; `queryLanguage:'sql'` | `{x:0,y:760,w:60,h:360}` |

**Optional depth (follow-up):** Throttles `line`; Concurrent executions `line`;
Stream-consumer lag `line` (`IteratorAge` — only for a stream/event-source
consumer, empty otherwise). Append two-up (`w:30,h:320`) and re-sequence `y`.

### Kubernetes/EKS

OTLP-native — no vended scope. **RED** from the span metrics grouped by
`@resource.service.name`; **USE** from the OTLP Kubernetes resource metrics grouped
by `@resource.k8s.*`. The span-metric names, span-status attribute, and `k8s.*`
resource-metric names are ingestion-path-dependent — ground them against the live
catalog (section 3).

**CORE (~8 panels — the default build):**

| Panel | Viz | Query template | Layout |
|-------|-----|----------------|--------|
| Request rate | `number` | `sum(sum_over_time({__name__="traces.span.metrics.calls", "@resource.service.name"="<svc>"}[5m])) / 300` (delta Sum — `rate()` under-reads it) | `{x:0,y:0,w:15,h:120}` |
| Error rate | `number` | ratio of `…calls, "status.code"="ERROR"` to all calls | `{x:15,y:0,w:15,h:120}` |
| p99 latency | `number` | `histogram_quantile(0.99, rate({__name__="traces.span.metrics.duration", …}[5m]))` | `{x:30,y:0,w:15,h:120}` |
| Running pods | `number` | `count({"k8s.pod.phase", "@resource.k8s.namespace.name"="<ns>", "@resource.k8s.deployment.name"="<deploy>"})` | `{x:45,y:0,w:15,h:120}` |
| Request rate | `line` | `sum by ("@resource.service.name") (sum_over_time({__name__="traces.span.metrics.calls", …}[5m])) / 300` | `{x:0,y:120,w:30,h:320}` |
| Errors by status | `bar` | `sum by ("status.code") (sum_over_time({__name__="traces.span.metrics.calls", …}[5m]))` | `{x:30,y:120,w:30,h:320}` |
| Latency p50/p90/p99 | `line` | `histogram_quantile(0.99, rate({__name__="traces.span.metrics.duration", …}[5m]))` — one series per quantile (0.50, 0.90, 0.99) | `{x:0,y:440,w:60,h:320}` |
| Top services by errors | `table` | `topk(10, sum by ("@resource.service.name") (sum_over_time({__name__="traces.span.metrics.calls", "status.code"="ERROR"}[5m])))` | `{x:0,y:760,w:60,h:360}` |

**Optional depth (follow-up "infra / USE health"):** Pod CPU usage, Pod memory
usage, Container restarts — grouped by `@resource.k8s.*` (`k8s.pod.cpu.usage`,
`k8s.pod.memory.usage`, `increase(k8s.container.restarts[15m])`). Append two-up and
re-sequence `y`.

### RDS/database

Infrastructure resource — USE/saturation; scope `cloudwatch.aws/rds` by
`DBInstanceIdentifier`. Reading rules: IOPS is already per-second → never `rate()`;
latency is in seconds → × 1000 for ms; mind `FreeStorageSpace` / `FreeableMemory`
polarity (higher is healthier).

**CORE (~7 panels — the default build):**

| Panel | Viz | Query template | Layout |
|-------|-----|----------------|--------|
| CPU % | `number` | `max({__name__="CPUUtilization", "@instrumentation.@name"="cloudwatch.aws/rds", DBInstanceIdentifier="<db>"})` | `{x:0,y:0,w:15,h:120}` |
| Connections | `number` | `max({__name__="DatabaseConnections", …, DBInstanceIdentifier="<db>"})` | `{x:15,y:0,w:15,h:120}` |
| Free storage | `number` | `min({__name__="FreeStorageSpace", …, DBInstanceIdentifier="<db>"})` | `{x:30,y:0,w:15,h:120}` |
| Read latency (ms) | `number` | `max({__name__="ReadLatency", …, DBInstanceIdentifier="<db>"}) * 1000` | `{x:45,y:0,w:15,h:120}` |
| CPU % | `line` | `{__name__="CPUUtilization", …, DBInstanceIdentifier="<db>"}` | `{x:0,y:120,w:30,h:320}` |
| Connections | `line` | `{__name__="DatabaseConnections", …, DBInstanceIdentifier="<db>"}` | `{x:30,y:120,w:30,h:320}` |
| Top instances by CPU | `table` | `topk(10, max by (DBInstanceIdentifier) ({__name__="CPUUtilization", …}))` | `{x:0,y:440,w:60,h:360}` |

**Optional depth (follow-up "infra / USE health"):** Read IOPS `line`; Read latency
(ms) `line`; add the `WriteIOPS` / `WriteLatency` counterparts for a full
read+write view. Append two-up and re-sequence `y`.

### ECS/web service

Cross-signal — **USE** from the ECS service tier combined with **RED** from the
load balancer in front (or from Application Signals). Identity + reading rules: ECS
scopes by `ClusterName` + a non-empty `ServiceName`; ALB `TargetResponseTime` is a
gauge in seconds with no percentile label; the target status-code metrics carry the
`_Count` suffix.

**CORE (~7 panels — the default build; this archetype has no ranked table):**

| Panel | Viz | Query template | Layout |
|-------|-----|----------------|--------|
| Request rate | `number` | `sum(rate({__name__="RequestCount", "@instrumentation.@name"="cloudwatch.aws/applicationelb", LoadBalancer="<lb>"}[5m]))` | `{x:0,y:0,w:15,h:120}` |
| 5xx rate | `number` | `sum(rate({__name__="HTTPCode_Target_5XX_Count", …, LoadBalancer="<lb>"}[5m]))` | `{x:15,y:0,w:15,h:120}` |
| Peak response (ms) | `number` | `max({__name__="TargetResponseTime", …, LoadBalancer="<lb>"}) * 1000` | `{x:30,y:0,w:15,h:120}` |
| Service CPU % | `number` | `max({__name__="CPUUtilization", "@instrumentation.@name"="cloudwatch.aws/ecs", ClusterName="<cluster>", ServiceName="<svc>"})` | `{x:45,y:0,w:15,h:120}` |
| Request rate | `line` | `sum(rate({__name__="RequestCount", …, LoadBalancer="<lb>"}[5m]))` | `{x:0,y:120,w:30,h:320}` |
| Status codes | `bar` | `sum by (__name__) (rate({__name__=~"HTTPCode_Target_..._Count", …, LoadBalancer="<lb>"}[5m]))` | `{x:30,y:120,w:30,h:320}` |
| Response time (ms) | `line` | `{__name__="TargetResponseTime", …, LoadBalancer="<lb>"} * 1000` | `{x:0,y:440,w:60,h:320}` |

**Optional depth (follow-up "infra / USE health"):** Memory util % `line`; Running
tasks `line` (`RunningTaskCount` — needs ECS Container Insights or OTel enrichment,
empty otherwise). If the space runs Application Signals, swap the RED panels for
`Service`/`Fault`/`Error` and the availability formula `(1 − Fault/Total) × 100`.
Append two-up and re-sequence `y`.

### API Gateway

Request-serving — RED. Identity split: REST v1 on `ApiName` with `4XXError` /
`5XXError`, HTTP v2 on `ApiId` with `4xx` / `5xx`; gateway overhead is the formula
`Latency − IntegrationLatency`.

**CORE (~8 panels — the default build):**

| Panel | Viz | Query template | Layout |
|-------|-----|----------------|--------|
| Request count/s | `number` | `sum(rate({__name__="Count", "@instrumentation.@name"="cloudwatch.aws/apigateway", ApiName="<api>"}[5m]))` | `{x:0,y:0,w:15,h:120}` |
| 5xx rate | `number` | `sum(rate({__name__="5XXError", …, ApiName="<api>"}[5m]))` | `{x:15,y:0,w:15,h:120}` |
| 4xx rate | `number` | `sum(rate({__name__="4XXError", …, ApiName="<api>"}[5m]))` | `{x:30,y:0,w:15,h:120}` |
| Peak latency (ms) | `number` | `max({__name__="Latency", …, ApiName="<api>"})` | `{x:45,y:0,w:15,h:120}` |
| Request count | `line` | `sum(rate({__name__="Count", …, ApiName="<api>"}[5m]))` | `{x:0,y:120,w:30,h:320}` |
| Errors 4xx/5xx | `bar` | `sum by (__name__) (rate({__name__=~"[45]XXError", …, ApiName="<api>"}[5m]))` | `{x:30,y:120,w:30,h:320}` |
| Latency (ms) | `line` | `{__name__="Latency", …, ApiName="<api>"}` | `{x:0,y:440,w:60,h:320}` |
| Top resources by 5xx | `table` | `topk(10, sum by (Resource) (rate({__name__="5XXError", …, ApiName="<api>"}[5m])))` | `{x:0,y:760,w:60,h:360}` |

**Optional depth (follow-up "per-service latency"):** Integration latency (ms)
`line` (`IntegrationLatency`) — chart it beside `Latency` so the gap reads as
gateway overhead. Append two-up and re-sequence `y`.
rea-analysis-optimizationmorluto/reaOptimize REA providers and shared analysis workflows for agent context, latency, CPU, and memory while preserving correctness and Evidence. Use for analysis bottlenecks and cache or query design; CI/test-suite optimization belongs to rea-ci-tests-audit.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-ci-tests-auditmorluto/reaAudit REA CI and test suites for correctness, fixture drift, false passes, and measured performance; use for suite or workflow investigations and efficiency changes, rather than ordinary feature testing or a single test failure.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
rea-tool-designmorluto/reaAudit, investigate, design, or change REA investigation tools, CLI/MCP contracts, provider capabilities, and Evidence semantics.Stars: 472kUpdated Claude CodeCodexVerifiedInstall
ponytail-auditDietrichGebert/ponytailQuality audit of the whole repo: bugs, security, real load, missing tests, speed, and what to delete. Most important first.Stars: 194kUpdated Claude CodeCodexVerifiedInstall