Skip to content

hoangminhung/devops-ai-kit / claude/skills/playbook-monitoring-stack

playbook-monitoring-stack

Verified

Playbook for deploying, upgrading or extending a self-hosted Kubernetes observability stack — kube-prometheus-stack (Prometheus Operator, Prometheus, Alertmanager, Grafana), Loki, Grafana Alloy as the collector, persistent storage on Longhorn or cloud disks, retention and capacity sizing, alert routing, dashboards as code, and migrating off Promtail/Grafana Agent. Use whenever the task is to build, change, scale, upgrade or troubleshoot monitoring, logging or alerting infrastructure, even if only one component (e.g. just Loki or just Alloy) is mentioned.

Playbook: Observability stack on Kubernetes

This playbook is a checklist that the plan must answer. It does not replace the official docs: field names, deployment modes and defaults change between chart versions, so ask infra-researcher to check them for the version you use.

1. Design questions (the plan must answer each one)

Scope & placement

  • What to collect: infrastructure metrics, app metrics, logs, (traces?), from how many clusters?
  • Where the stack runs: inside the monitored cluster, or in a separate monitoring cluster? If the cluster dies, who sends the alert?

Components & versions

  • Which charts and versions (pin exact versions):
    • kube-prometheus-stack;
    • Loki (chart grafana/loki);
    • Alloy (chart grafana/alloy, or the combined chart grafana/k8s-monitoring);
    • Grafana (bundled with kube-prometheus-stack, or separate).
  • Does the old system have Promtail / Grafana Agent? Both are EOL, so add the move to Alloy to the scope (alloy convert helps convert the config) or record it as a follow-up.

Loki

  • Deployment mode: monolithic / single-binary for small log volume; other modes for bigger scale. Choose based on the Loki docs for the exact version, because the recommended mode has changed between versions.
  • Loki storage: filesystem on a PVC is only for single-binary. To scale you need object storage (Azure Blob, S3, GCS or self-hosted S3-compatible). For a self-hosted option, check the product's maintenance / license status before you choose it.

Alloy

  • Collect logs through the Kubernetes API (loki.source.kubernetes), or read /var/log/pods files with a DaemonSet. Big clusters should prefer file-based, so the API server does not get extra load.
  • Does Alloy also collect metrics (taking over part of the scraping), or only logs?

Capacity

  • Retention: how many days for metrics, how many days for logs. Set both retention and retentionSize for Prometheus.

  • Estimate the size:

    • Prometheus ≈ retention_seconds × samples/second × 1–2 bytes/sample;
    • Loki ≈ GB of logs/day × days ÷ compression ratio;
    • then multiply by the replication factor of the app and of Longhorn.

    Write the calculation in the plan.

  • Longhorn: which StorageClass for Prometheus / Loki / Alertmanager / Grafana. For components that already run HA (Prometheus with 2 replicas, Loki replication), think about a lower numberOfReplicas (see k8s-storage).

Access & security

  • Grafana: SSO / OIDC; the admin password comes from a Secret through ESO (not in values).
  • Prometheus / Alertmanager UI are not public.
  • Do you need tenants / permissions per team?

Alerting

  • Alertmanager receivers: Slack / Teams / email / Telegram / PagerDuty. Webhook credentials come through ESO.
  • Routing by severity, team.
  • Watchdog goes to an external dead man's switch.

Content

  • Dashboards and alert rules are code: which folder in Git, and how they are provisioned (sidecar label grafana_dashboard, PrometheusRule).

Deployment through GitOps

  • Argo CD with ServerSideApply=true for the large CRDs of kube-prometheus-stack.
  • Sync waves: CRDs / operator → stack → dashboards / rules.

2. Common risks

RiskHow to avoid it
Prometheus PVC is full → Prometheus crash loopretentionSize < PVC size (keep ~15–20% free); predict_linear alert on the disk
kube-prometheus-stack chart upgraded but CRDs are oldRead the upgrade notes for each major; apply CRDs first as the notes say
Cardinality explosion makes Prometheus run out of RAMReview the labels of new ServiceMonitors; watch prometheus_tsdb_head_series
Loki rejects logs (rate limit / stream limit)Set limits_config from real volume; keep labels minimal
Alerts reach nobody and nobody knowsWatchdog + dead man's switch; test receivers after every change
The monitoring stack dies together with the monitored clusterHeartbeat to an outside service; think about a separate monitoring cluster when there are many clusters

3. Suggested deployment order

  1. Namespace (+ PSA labels), StorageClass (if a new one is needed), secrets through ESO.
  2. Prometheus Operator CRDs, then kube-prometheus-stack.
  3. Loki, then Alloy (logs into Loki).
  4. Grafana datasources (Prometheus, Loki), dashboards, PrometheusRules.
  5. Alertmanager receivers, then send a test alert.
  6. For an old stack (Promtail / Agent): run both side by side, compare, then remove the old one.

4. Verification checklist (put it in section 8 of the plan)

CheckSuggested commandExpected
Pods healthykubectl -n <ns> get podsAll Running/Ready, restarts not going up
PVCkubectl -n <ns> get pvcBound, correct StorageClass and size
Longhorn volumeskubectl -n longhorn-system get volumes.longhorn.iorobustness healthy
Scrape targetsQuery count(up == 0) through the Prometheus API / Grafana0, or only targets known to be not needed
Rules loadQuery count(ALERTS) + check the Rules page / promtool beforeNo evaluation errors
Logs reach LokiQuery {namespace="<ns>"} for the last 5 minutesNew log lines
Alerts reach receiversSend a test alert (amtool alert add or wait for Watchdog)Message arrives in the right channel
Dead man's switchCheck the heartbeat serviceIt is receiving signals
RetentionCheck the running args / configSame values as in the plan
GrafanaLog in with SSO, datasource "Test"Works

5. Docs to leave behind

  • A runbook for each main component (Prometheus, Loki, Alloy, Alertmanager): alert → action, how to add capacity, how to upgrade.
  • A concept note "How a log line / a metric travels from a pod to Grafana" for newcomers.
  • An ADR if there are architecture choices (Loki mode, where the stack runs, object storage).
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 322kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 198kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall