Skip to content

kr7shnasomani/kairos / .agents/skills/alerting-irm

alerting-irm

Verified

Configure Grafana Alerting, Incident Response Management (IRM), and SLOs end-to-end — provisions Grafana-managed and data-source-managed alert rules, contact points (Slack/PagerDuty/email/webhook), notification policies with hierarchical matchers, silences, mute timings, on-call schedules and escalation chains, incident-management integrations, and SLOs with multi-window burn-rate alerts. Use when configuring alerts, debugging notification routing, setting up on-call rotations, declaring or managing incidents, defining SLOs, provisioning alerting via YAML or API, picking matchers for a notification policy, building a PagerDuty/Slack webhook receiver, or troubleshooting why an alert isn't firing — even when the user says "page me on errors", "alert me when X happens", "route this to the platform team", or "set up an SLO" without naming Alerting or IRM.

FileKindSize
SKILL.mdskill md7.6 KB
references/alerting.mdmarkdown4.6 KB
references/irm.mdmarkdown2.2 KB
references/slo.mdmarkdown2.1 KB

references/slo.md

View on GitHub
# SLOs and burn-rate alerts

Grafana auto-generates recording rules, dashboards, and burn-rate alerts when you define an SLO via the UI or API. Reference YAML for the generated artifacts:

## Generated recording rules

```yaml
groups:
  - name: slo_availability
    interval: 1m
    rules:
      - record: slo:availability:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{status!~"5.."}[5m])) by (service)
          / sum(rate(http_requests_total[5m])) by (service)

      - record: slo:error_budget:remaining
        expr: |
          (slo:availability:ratio_rate30d - 0.999) / (1 - 0.999)
```

## Burn-rate alerts

```yaml
- alert: SLOBurnRateHigh
  expr: |
    slo:burn_rate:ratio_rate1h > 14.4      # 1h window, 5% budget in 1h
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "SLO burn rate critical for {{ $labels.service }}"
```

## Multi-window burn-rate alerts (recommended)

The single 1h-window check above fires on transient noise. Better to require burn rate above threshold in BOTH a long window (1h, the "sustained" signal) AND a short window (5m, the "still burning right now" signal):

```yaml
- alert: SLOBurnRateFast
  expr: |
    slo:burn_rate:ratio_rate1h > 14.4
    and
    slo:burn_rate:ratio_rate5m > 14.4
  for: 2m
  labels:
    severity: critical
```

The `and` filters out: (a) a single 5m spike that wasn't actually a sustained issue (1h window stays low), and (b) a slow burn that already recovered (5m window now low). This is the Google SRE "multi-window, multi-burn-rate" pattern — see [SRE Workbook ch. 5](https://sre.google/workbook/alerting-on-slos/).

## Validating SLO config

After creating an SLO:
1. Check recording rules are running: `GET /api/v1/rules?type=record` should include the SLO's recording rules
2. Wait for the rules to evaluate at least once (default interval 1m)
3. Verify burn rate is computed: `GET /api/v1/query?query=slo:burn_rate:ratio_rate1h` should return a numeric value (not empty)
4. Test the alert: force a failure (e.g. inject errors in staging) and confirm the alert fires within the `for` duration
receiving-code-reviewobra/superpowersUse when receiving code review feedback, before implementing suggestions, especially if feedback seems unclear or technically questionable - requires technical rigor and verification, not performative agreement or blind implementationStars: 322kUpdated Claude CodeCodexVerifiedInstall
requesting-code-reviewobra/superpowersUse when completing tasks, implementing major features, or before merging to verify work meets requirementsStars: 322kUpdated Claude CodeCodexVerifiedInstall
using-git-worktreesobra/superpowersUse when starting feature work that needs isolation from current workspace or before executing implementation plans - ensures an isolated workspace exists via native tools or git worktree fallbackStars: 322kUpdated Claude CodeCodexVerifiedInstall
verification-before-completionobra/superpowersUse when about to claim work is complete, fixed, or passing, before committing or creating PRs - requires running verification commands and confirming output before making any success claims; evidence before assertions alwaysStars: 322kUpdated Claude CodeCodexVerifiedInstall