Skip to content

hoangminhung/devops-ai-kit / claude/skills/playbook-monitoring-stack

playbook-monitoring-stack

Verified

Playbook for deploying, upgrading or extending a self-hosted Kubernetes observability stack — kube-prometheus-stack (Prometheus Operator, Prometheus, Alertmanager, Grafana), Loki, Grafana Alloy as the collector, persistent storage on Longhorn or cloud disks, retention and capacity sizing, alert routing, dashboards as code, and migrating off Promtail/Grafana Agent. Use whenever the task is to build, change, scale, upgrade or troubleshoot monitoring, logging or alerting infrastructure, even if only one component (e.g. just Loki or just Alloy) is mentioned.

FileKindSize
SKILL.mdskill md6.9 KB
---
name: playbook-monitoring-stack
description: Playbook for deploying, upgrading or extending a self-hosted Kubernetes observability stack — kube-prometheus-stack (Prometheus Operator, Prometheus, Alertmanager, Grafana), Loki, Grafana Alloy as the collector, persistent storage on Longhorn or cloud disks, retention and capacity sizing, alert routing, dashboards as code, and migrating off Promtail/Grafana Agent. Use whenever the task is to build, change, scale, upgrade or troubleshoot monitoring, logging or alerting infrastructure, even if only one component (e.g. just Loki or just Alloy) is mentioned.
when_to_use: Use when building, upgrading, extending or debugging a monitoring / logging / alerting system (Prometheus, Grafana, Loki, Alloy, Alertmanager), even if only one component is mentioned. Vietnamese triggers — "xây dựng, nâng cấp, mở rộng hoặc debug hệ thống monitoring / logging / alerting", "kể cả khi chỉ nhắc một thành phần".
---

# Playbook: Observability stack on Kubernetes

<!-- devops-ai-kit · Last reviewed: 2026-10-28 · Detailed config standards: rules `observability.md`, `helm.md`; storage: skill `k8s-storage`. -->

This playbook is a **checklist that the plan must answer**. It does not replace the official docs: field names, deployment modes and defaults change between chart versions, so ask `infra-researcher` to check them for the version you use.

## 1. Design questions (the plan must answer each one)

**Scope & placement**

- [ ] What to collect: infrastructure metrics, app metrics, logs, (traces?), from how many clusters?
- [ ] Where the stack runs: inside the monitored cluster, or in a separate monitoring cluster? If the cluster dies, who sends the alert?

**Components & versions**

- [ ] Which charts and versions (pin exact versions):
  - kube-prometheus-stack;
  - Loki (chart `grafana/loki`);
  - Alloy (chart `grafana/alloy`, or the combined chart `grafana/k8s-monitoring`);
  - Grafana (bundled with kube-prometheus-stack, or separate).
- [ ] Does the old system have Promtail / Grafana Agent? Both are EOL, so add the move to Alloy to the scope (`alloy convert` helps convert the config) or record it as a follow-up.

**Loki**

- [ ] Deployment mode: monolithic / single-binary for small log volume; other modes for bigger scale. Choose based on the Loki docs for the **exact version**, because the recommended mode has changed between versions.
- [ ] Loki storage: filesystem on a PVC is only for single-binary. To scale you need object storage (Azure Blob, S3, GCS or self-hosted S3-compatible). For a self-hosted option, check the product's maintenance / license status before you choose it.

**Alloy**

- [ ] Collect logs through the Kubernetes API (`loki.source.kubernetes`), or read `/var/log/pods` files with a DaemonSet. Big clusters should prefer file-based, so the API server does not get extra load.
- [ ] Does Alloy also collect metrics (taking over part of the scraping), or only logs?

**Capacity**

- [ ] Retention: how many days for metrics, how many days for logs. Set both `retention` **and** `retentionSize` for Prometheus.
- [ ] Estimate the size:
  - Prometheus ≈ `retention_seconds × samples/second × 1–2 bytes/sample`;
  - Loki ≈ `GB of logs/day × days ÷ compression ratio`;
  - then multiply by the replication factor of the app **and** of Longhorn.

  Write the calculation in the plan.
- [ ] Longhorn: which StorageClass for Prometheus / Loki / Alertmanager / Grafana. For components that already run HA (Prometheus with 2 replicas, Loki replication), think about a lower `numberOfReplicas` (see `k8s-storage`).

**Access & security**

- [ ] Grafana: SSO / OIDC; the admin password comes from a Secret through ESO (not in values).
- [ ] Prometheus / Alertmanager UI are not public.
- [ ] Do you need tenants / permissions per team?

**Alerting**

- [ ] Alertmanager receivers: Slack / Teams / email / Telegram / PagerDuty. Webhook credentials come through ESO.
- [ ] Routing by `severity`, `team`.
- [ ] Watchdog goes to an external dead man's switch.

**Content**

- [ ] Dashboards and alert rules are code: which folder in Git, and how they are provisioned (sidecar label `grafana_dashboard`, PrometheusRule).

**Deployment through GitOps**

- [ ] Argo CD with `ServerSideApply=true` for the large CRDs of kube-prometheus-stack.
- [ ] Sync waves: CRDs / operator → stack → dashboards / rules.

## 2. Common risks

| Risk | How to avoid it |
|---|---|
| Prometheus PVC is full → Prometheus crash loop | `retentionSize` < PVC size (keep ~15–20% free); `predict_linear` alert on the disk |
| kube-prometheus-stack chart upgraded but CRDs are old | Read the upgrade notes for each major; apply CRDs first as the notes say |
| Cardinality explosion makes Prometheus run out of RAM | Review the labels of new ServiceMonitors; watch `prometheus_tsdb_head_series` |
| Loki rejects logs (rate limit / stream limit) | Set `limits_config` from real volume; keep labels minimal |
| Alerts reach nobody and nobody knows | Watchdog + dead man's switch; test receivers after every change |
| The monitoring stack dies together with the monitored cluster | Heartbeat to an outside service; think about a separate monitoring cluster when there are many clusters |

## 3. Suggested deployment order

1. Namespace (+ PSA labels), StorageClass (if a new one is needed), secrets through ESO.
2. Prometheus Operator CRDs, then kube-prometheus-stack.
3. Loki, then Alloy (logs into Loki).
4. Grafana datasources (Prometheus, Loki), dashboards, PrometheusRules.
5. Alertmanager receivers, then send a test alert.
6. For an old stack (Promtail / Agent): run both side by side, compare, then remove the old one.

## 4. Verification checklist (put it in section 8 of the plan)

| Check | Suggested command | Expected |
|---|---|---|
| Pods healthy | `kubectl -n <ns> get pods` | All Running/Ready, restarts not going up |
| PVC | `kubectl -n <ns> get pvc` | Bound, correct StorageClass and size |
| Longhorn volumes | `kubectl -n longhorn-system get volumes.longhorn.io` | robustness `healthy` |
| Scrape targets | Query `count(up == 0)` through the Prometheus API / Grafana | 0, or only targets known to be not needed |
| Rules load | Query `count(ALERTS)` + check the Rules page / `promtool` before | No evaluation errors |
| Logs reach Loki | Query `{namespace="<ns>"}` for the last 5 minutes | New log lines |
| Alerts reach receivers | Send a test alert (`amtool alert add` or wait for Watchdog) | Message arrives in the right channel |
| Dead man's switch | Check the heartbeat service | It is receiving signals |
| Retention | Check the running args / config | Same values as in the plan |
| Grafana | Log in with SSO, datasource "Test" | Works |

## 5. Docs to leave behind

- A runbook for each main component (Prometheus, Loki, Alloy, Alertmanager): alert → action, how to add capacity, how to upgrade.
- A concept note "How a log line / a metric travels from a pod to Grafana" for newcomers.
- An ADR if there are architecture choices (Loki mode, where the stack runs, object storage).
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall