Skip to content

oliverv/caro / .agents/skills/oma-observability

oma-observability

Unverified

Design or troubleshoot telemetry, SLOs, and incident diagnostics. Route to the relevant signal, system boundary, or vendor guide.

FileKindSize
SKILL.mdskill md14.6 KB
resources/anti-patterns.mdmarkdown45.7 KB
resources/boundaries/cross-application.mdmarkdown19.5 KB
resources/boundaries/multi-tenant.mdmarkdown17.9 KB
resources/boundaries/release.mdmarkdown12.4 KB
resources/boundaries/slo.mdmarkdown6.5 KB
resources/checklist.mdmarkdown15.0 KB
resources/examples.mdmarkdown15.7 KB
resources/execution-protocol.mdmarkdown9.5 KB
resources/incident-forensics.mdmarkdown23.3 KB
resources/intent-rules.mdmarkdown11.8 KB
resources/layers/L3-network.mdmarkdown17.6 KB
resources/layers/L4-transport.mdmarkdown14.6 KB
resources/layers/L7-application/crash-analytics.mdmarkdown17.6 KB
resources/layers/L7-application/mobile-rum.mdmarkdown15.8 KB
resources/layers/L7-application/waf.mdmarkdown16.6 KB
resources/layers/L7-application/web-rum.mdmarkdown17.3 KB
resources/layers/mesh.mdmarkdown20.0 KB
resources/matrix.mdmarkdown21.1 KB
resources/meta-observability.mdmarkdown23.5 KB
resources/observability-as-code.mdmarkdown21.8 KB
resources/signals/audit.mdmarkdown13.9 KB
resources/signals/cost.mdmarkdown17.2 KB
resources/signals/logs.mdmarkdown10.2 KB
resources/signals/metrics.mdmarkdown19.6 KB
resources/signals/privacy.mdmarkdown17.5 KB
resources/signals/profiles.mdmarkdown15.2 KB
resources/signals/traces.mdmarkdown19.1 KB
resources/standards.mdmarkdown12.7 KB
resources/transport/collector-topology.mdmarkdown11.9 KB
resources/transport/otlp-grpc-vs-http.mdmarkdown7.6 KB
resources/transport/sampling-recipes.mdmarkdown11.0 KB
resources/transport/udp-statsd-mtu.mdmarkdown6.3 KB
resources/vendor-categories.mdmarkdown17.7 KB

resources/examples.md

View on GitHub
# oma-observability Examples

> End-to-end walkthroughs demonstrating the skill's value.
> Each scenario ties multiple files together into a single executable story.

---

## Scenario 1: Greenfield: OSS Full-Stack on Kubernetes

**Situation:** Series A startup, new k8s cluster. Team wants OTel-native metrics + logs + traces + Grafana
dashboards at zero licensing cost.

**Intent:** `setup`

**Files referenced:** `vendor-categories.md §(a)`, `transport/collector-topology.md §2`, `observability-as-code.md §5`, `checklist.md`

### Walkthrough

1. Invoke `/oma-observability "set up OTel stack on k8s"`.
2. Intent classifier routes to `setup`.
3. `vendor-categories.md §(a) OSS Full-Stack` selects **Grafana LGTM+** (Mimir, Loki, Tempo, Grafana)
   for a team already familiar with Grafana dashboards.
4. `transport/collector-topology.md §2 Two-Tier Hybrid` determines the deployment pattern:
   DaemonSet agent per node + Deployment gateway.

**Agent CRD; DaemonSet (node-level collection):**

```yaml
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-agent
  namespace: observability
spec:
  mode: daemonset
  config:
    receivers:
      hostmetrics:
        collection_interval: 30s
        scrapers: { cpu: {}, memory: {}, filesystem: {} }
      filelog:
        include: [/var/log/pods/*/*/*.log]
        include_file_path: true
      kubeletstats:
        collection_interval: 20s
        auth_type: serviceAccount
    processors:
      memory_limiter:            # mandatory: first processor in every pipeline
        limit_mib: 400
        check_interval: 1s
      k8sattributes:
        extract:
          metadata: [k8s.pod.name, k8s.namespace.name, k8s.node.name]
    exporters:
      otlp:
        endpoint: otel-gateway.observability.svc:4317
    service:
      pipelines:
        metrics:
          receivers: [hostmetrics, kubeletstats]
          processors: [memory_limiter, k8sattributes]
          exporters: [otlp]
        logs:
          receivers: [filelog]
          processors: [memory_limiter, k8sattributes]
          exporters: [otlp]
```

**Gateway CRD; Deployment (batching, `mode: deployment`, 2 replicas):**

```yaml
# Key fields only — full config in observability-as-code.md §5
spec:
  mode: deployment
  replicas: 2
  config:
    receivers:
      otlp: { protocols: { grpc: { endpoint: 0.0.0.0:4317 } } }
    processors:
      memory_limiter: { check_interval: 1s, limit_percentage: 75 }
      batch: {}
    exporters:
      otlphttp/tempo: { endpoint: http://tempo.observability.svc:4318 }
      prometheusremotewrite: { endpoint: http://mimir.observability.svc:9009/api/v1/push }
      loki: { endpoint: http://loki.observability.svc:3100/loki/api/v1/push }
    service:
      pipelines:
        traces: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/tempo] }
        metrics: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheusremotewrite] }
        logs: { receivers: [otlp], processors: [memory_limiter, batch], exporters: [loki] }
```

5. Collector config is committed to git and applied via Argo CD app-of-apps per `observability-as-code.md §7`.
6. Validation checklist (from `checklist.md §1, §3`):
   - [ ] `memory_limiter` is the first processor in every pipeline
   - [ ] NTP drift < 100 ms on all nodes (`chronyc tracking`)
   - [ ] Cardinality budget set before first data ingest
   - [ ] Gateway has `≥ 2` replicas with PodDisruptionBudget

**Outcome:** 9 pods, 2 services, first trace visible in Grafana Tempo within 15 minutes of `kubectl apply`.

---

## Scenario 2: Incident Forensics: Payment Service 5xx Spike

**Situation:** "ap-northeast-2 payment service 5xx spike at 14:20 UTC"; alert fires with no `trace_id`
provided. On-call SRE must localize root cause in under 15 minutes.

**Intent:** `investigate`

**Files referenced:** `incident-forensics.md §3–§5 Scenario A`, `boundaries/slo.md §7`, `signals/traces.md`, `signals/logs.md §7`, `boundaries/release.md`

### Walkthrough

1. **Alert source:** multi-window burn-rate alert from `boundaries/slo.md §7` fires; 2 % budget
   consumed in 1 h at 14.4× rate. `severity: critical`.
2. Invoke `/oma-observability --investigate "5xx spike in ap-northeast-2"`.
3. Router routes to `incident-forensics.md §3 Six-Dimension Narrowing Flow`.

**Step 1; Acquire `trace_id` via metric exemplar (`incident-forensics.md §2`):**

```promql
histogram_quantile(0.99,
  sum by (le, service_name) (
    rate(http_server_request_duration_seconds_bucket{
      deployment_environment="prod",
      cloud_region="ap-northeast-2"
    }[5m])
  )
)
# Click a spike data point → Grafana Mimir exemplar panel shows trace_id link
```

Exemplar retrieved: `trace_id = 9a3f1c8e2b7d4e50a1c3f9d2b8e7a4c0`.

**Step 2; Six-dimension narrowing:**

| Dimension | Finding |
|-----------|---------|
| Region | `cloud.region = ap-northeast-2` only. `ap-southeast-1` healthy. Single-region blast radius. |
| Server | Errors on 3 of 6 pods, all on `k8s.node.name = ip-10-0-2-11.ec2.internal`. |
| Service | Earliest ERROR span: `service.name = payments-checkout`, `service.version = v2.4.1`. |
| Layer | `span.kind = CLIENT`, `db.system = redis`; L7 DB call, not a network-layer issue. |
| Code | `exception.type = TimeoutException`, `exception.message = "Redis pool exhausted after 5000ms"`, `code.function.name = processPayment`, `code.line.number = 287`. |

**Step 3; Cross-signal validation (`incident-forensics.md §4`):**

```
# Grafana Tempo — TraceQL lookup
{ trace:id = "9a3f1c8e2b7d4e50a1c3f9d2b8e7a4c0" }

# Filter logs by trace_id to find correlated warnings
# Loki LogQL
{service_name="payments-checkout"} | json | trace_id="9a3f1c8e2b7d4e50a1c3f9d2b8e7a4c0"
```

Logs reveal: `WARN: connection pool at capacity (256/256)` at 14:21. Redis
`redis_pool_active_connections / redis_pool_max_connections` metric at 100 %.

**Step 4; Release correlation:**

`service.version = v2.4.1` deployed at 13:57 UTC; 25 minutes before incident.
Changelog: Redis connection pool reduced from 512 → 256 in this version.

**Step 5; Remediation:**

Flagger canary analysis `successRate` already < 99 % → auto-rollback to `v2.4.0` triggers
per `boundaries/release.md §Flagger Canary CR`. Error rate returns to baseline in 3 minutes.
Deployment event chain preserved in `signals/audit.md` for post-mortem.

---

## Scenario 3: Multi-Tenant Cost Attribution

**Situation:** B2B SaaS now has 50+ tenants. CFO asks: "Which tenants cost most to serve, and
how does our observability bill break down per customer?"

**Intent:** `route` + `tune`

**Files referenced:** `boundaries/multi-tenant.md §7`, `signals/cost.md §3–§4`, `vendor-categories.md §(f)`, `meta-observability.md §Cardinality Guardrails`

### Walkthrough

1. Tag every pod at deploy time with `tenant.id`; applied via Helm values or admission webhook
   per `boundaries/multi-tenant.md §7`:

```yaml
metadata:
  labels:
    tenant.id: "acme-corp"
    tenant.tier: "enterprise"
```

2. Deploy OpenCost (CNCF Incubating) with Prometheus scrape per `signals/cost.md §3`:

```yaml
# OTel Collector gateway — OpenCost scrape
receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: opencost
          scrape_interval: 60s
          static_configs:
            - targets: ["opencost.opencost.svc.cluster.local:9003"]
```

3. Per-tenant cost PromQL (daily, `signals/cost.md §5`):

```promql
sum by (tenant_id) (
  increase(opencost_workload_cost_total[24h])
  * on(namespace) group_left(tenant_id)
  kube_namespace_labels
)
```

4. Cardinality guard: keep top-1 000 tenants labeled, bucket overflow as `"other"` to prevent
   TSDB series explosion (`meta-observability.md §Cardinality Guardrails`, `boundaries/multi-tenant.md §13`):

```yaml
# OTel Collector transform processor — cardinality cap
processors:
  transform/cardinality:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["tenant_id"], "other")
              where attributes["tenant_id"] not in ${env:TOP_TENANT_LIST}
```

5. Per-tier sampling policy at the gateway to control observability bill per tenant
   (`boundaries/multi-tenant.md §5`): `enterprise` = 100 %, `pro` = 20 %, `free` = 2 %.
   Full `tail_sampling` YAML (with `and` sub-policies) lives in `transport/sampling-recipes.md §4`.

6. Include the observability bill itself: instrument the collector pipeline with throughput counters
   per `tenant.id` to apportion Grafana Cloud / Datadog usage proportionally
   (`signals/cost.md §4`, `vendor-categories.md §(f)`).

**Outcome:** Identified 3 free-tier tenants consuming 40 % of egress cost. Applied egress rate
limit at collector ingress (`boundaries/multi-tenant.md §11 Noisy Neighbor Protection`).
Those tenants were moved to a stricter free-tier cap, reducing monthly egress spend by 35 %.

---

## Scenario 4: Migrating off Fluentd to Fluent Bit

**Situation:** Legacy Fluentd DaemonSet is consuming 400 MB RAM per node vs. Fluent Bit's typical
80 MB. The CNCF 2025-10 migration guide is now the normative reference. Team needs zero-downtime
migration with log parity verification before decommission.

**Intent:** `migrate`

**Files referenced:** `vendor-categories.md §(h)`, `signals/logs.md §6`, `transport/collector-topology.md §2`, `observability-as-code.md §3`

### Walkthrough

1. Invoke `/oma-observability --migrate "Fluentd to Fluent Bit"`.
2. Router routes to `vendor-categories.md §(h) Log Pipeline` → **Fluent Bit** (CNCF Graduated,
   drop-in config compatibility with Fluentd, C runtime, 5–15 MB RAM typical).

**Migration phases (zero-downtime):**

**Phase 1; Dual-send:** Deploy Fluent Bit DaemonSet alongside the existing Fluentd DaemonSet.
Fluent Bit ships logs to the same backend; Fluentd continues as primary. Confirm both streams
reach the backend without duplicates by comparing log record counts.

```ini
# Fluent Bit DaemonSet — core pipeline (fluent-bit.conf)
[INPUT]
    Name           tail
    Path           /var/log/pods/*/*/*.log
    Parser         cri
    Tag            kube.*
    Mem_Buf_Limit  5MB

[FILTER]
    Name        kubernetes
    Match       kube.*
    Merge_Log   On
    Labels      On

[OUTPUT]
    Name      opentelemetry
    Match     kube.*
    Host      otel-gateway.observability.svc
    Port      4318
    Logs_uri  /v1/logs
```

**Phase 2; Parity verification:** query log record count per `service.name` in both pipelines
over a 1-hour window. Expect < 0.1 % divergence accounting for timing windows.

```promql
# Fluent Bit throughput
sum by (service_name) (rate(fluentbit_output_proc_records_total[1h]))

# Fluentd throughput (comparison)
sum by (service_name) (rate(fluentd_output_emit_records_total[1h]))
```

**Phase 3; Decommission Fluentd:** once parity is confirmed over 24 h, scale down Fluentd
DaemonSet to 0 and remove its resources.

```bash
kubectl scale daemonset fluentd -n logging --replicas=0
kubectl delete daemonset fluentd -n logging
```

3. Commit Fluent Bit ConfigMap and DaemonSet manifests to git; apply via Argo CD per
   `observability-as-code.md §7`.
4. Validate with `otelcol validate` on the gateway config; run `promtool check rules` on any
   log-based alerting rules.

**Outcome:** Node-level log collection RAM drops from 400 MB (Fluentd/Ruby) to ~80 MB (Fluent Bit/C),
a 5× reduction typical per the CNCF 2025-10 guide. CPU overhead at steady state also reduces by
~60 %. No log gaps observed during the dual-send window.

---

## Scenario 5: SLO with Burn-Rate Alerts (GitOps End-to-End)

**Situation:** Platform team adopts the SLO practice for the checkout service. Target: 99.9 %
availability on a 28-day rolling window. Everything must be versioned in git and applied
via Argo CD; no manual UI edits.

**Intent:** `alert`

**Files referenced:** `boundaries/slo.md §5, §7, §8`, `observability-as-code.md §4, §6, §7`, `boundaries/release.md`

### Walkthrough

1. Define the SLO as OpenSLO YAML and commit to the observability repo (`boundaries/slo.md §5`):

```yaml
# Key fields — full spec in boundaries/slo.md §5
apiVersion: openslo.com/v1
kind: SLO
metadata:
  name: checkout-availability
spec:
  service: checkout
  sloType: Request-Based
  objectives:
    - target: 0.999
      window: 28d
  indicator:
    spec:
      ratioMetric:
        good:
          metricSource:
            type: Prometheus
            spec:
              query: >
                sum(rate(http_requests_total{service="checkout",status=~"2..|3.."}[{{.Window}}]))
        total:
          metricSource:
            type: Prometheus
            spec:
              query: sum(rate(http_requests_total{service="checkout"}[{{.Window}}]))
```

2. Generate PrometheusRule CRD via Sloth (`observability-as-code.md §4.2`):

```bash
sloth generate -i sloth/checkout-slo.yaml | kubectl apply -f -
```

3. The generated `PrometheusRule` includes multi-window burn-rate alerts from `boundaries/slo.md §7`
   and `observability-as-code.md §6`:

```yaml
# Fast burn — 2 % budget in 1 h (14.4× rate), gated by 5 m short window → page
- alert: CheckoutSLOBurnRateFast
  expr: |
    (sum(rate(http_requests_total{service="checkout",status=~"5.."}[1h]))
      / sum(rate(http_requests_total{service="checkout"}[1h]))) / (1 - 0.999) > 14.4
    and
    (sum(rate(http_requests_total{service="checkout",status=~"5.."}[5m]))
      / sum(rate(http_requests_total{service="checkout"}[5m]))) / (1 - 0.999) > 14.4
  for: 2m
  labels: { severity: critical, slo: checkout-availability }
  annotations:
    summary: "Checkout SLO fast burn: budget exhausting in < 1h"

# Slow burn — 5 % budget in 6 h (6× rate), gated by 30 m → ticket
# Full PromQL in observability-as-code.md §6 and boundaries/slo.md §7.4
```

4. GitOps commit flow (`observability-as-code.md §7`):

```
git PR (checkout-slo.yaml + PrometheusRule)
  → CI: promtool check rules alerts/checkout-slo-burn-rate.yaml
  → CI: promtool test rules alerts/checkout_slo_test.yaml
  → peer review (SRE CODEOWNERS approval)
  → merge to main
  → Argo CD sync → Prometheus Operator applies PrometheusRule to cluster
```

5. Validate via synthetic error injection:

```bash
# Inject 20 % error rate for 3 minutes to trigger fast-burn alert
kubectl run error-injector --image=curlimages/curl --restart=Never -- \
  sh -c 'for i in $(seq 1 200); do curl -s -o /dev/null -w "%{http_code}" \
    http://checkout-api/error-inject; done'
```

Fast-burn alert fires in < 2 minutes.

6. Error budget policy (`boundaries/slo.md §8`):

| Budget remaining | Action |
|-----------------|--------|
| > 50 % | Normal feature velocity |
| 25–50 % | Review reliability vs feature ratio |
| < 25 % | Reliability sprint |
| 0 % exhausted | Deploy freeze until budget recovers |

When budget reaches 0 %, Flagger canary promotion gates use the same SLI error rate to block
new deploys (`boundaries/release.md`).

---

## Appendix: Vendor Query Cross-Reference

From `incident-forensics.md §4`; trace lookup by `trace_id` across backends:

| Vendor | Query syntax |
|--------|-------------|
| Grafana Tempo | `{ trace:id = "4bf92f3577b34da6a3ce929d0e0e4736" }` |
| Honeycomb | `trace:4bf92f3577b34da6a3ce929d0e0e4736` |
| Datadog APM | `@trace_id:4bf92f3577b34da6a3ce929d0e0e4736` |
| Jaeger HTTP API | `GET /api/traces/4bf92f3577b34da6a3ce929d0e0e4736` |
| Elastic APM | `trace.id: "4bf92f3577b34da6a3ce929d0e0e4736"` |
| Sentry | `trace_id:4bf92f3577b34da6a3ce929d0e0e4736` |

---

## Contribution Protocol

- Add new scenarios only if they demonstrate cross-file value spanning 3+ skill files.
- Each scenario must cite at least 3 other skill files by relative path from `resources/`.
- Scenarios are for human learning: keep them narrative, not exhaustive.
- Code and YAML snippets are illustrative excerpts; for authoritative full config see the
  referenced files.
- Do not add future skill references: if a domain is out of scope, cite the external tool
  from `SKILL.md §When NOT to use`.
brainstormingobra/superpowersYou MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).Stars: 322kUpdated Claude CodeCodexVerifiedInstall
brand-guidelinesanthropics/skillsApplies Anthropic's official brand colors and typography to any sort of artifact that may benefit from having Anthropic's look-and-feel. Use it when brand colors or style guidelines, visual formatting, or company design standards apply.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
frontend-designanthropics/skillsGuidance for distinctive, intentional visual design when building new UI or reshaping an existing one. Helps with aesthetic direction, typography, and making choices that don't read as templated defaults.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
mcp-builderanthropics/skillsGuide for creating high-quality MCP (Model Context Protocol) servers that enable LLMs to interact with external services through well-designed tools. Use when building MCP servers to integrate external APIs or services, whether in Python (FastMCP) or Node/TypeScript (MCP SDK).Stars: 198kUpdated Claude CodeCodexVerifiedInstall