arjunprabhulal/agent-skills / skills/sre/monitoring-setup
monitoring-setup
VerifiedInstruments a service so problems are visible before users report them — metrics, logs, traces, dashboards, and alerts people will actually act on. Use this whenever the user is adding monitoring, mentions Prometheus, Grafana, OpenTelemetry, SLOs, dashboards, or alerting, has a service nobody is watching, or is drowning in alerts nobody reads. For instrumenting an AI agent specifically, use agent-observability.
| File | Kind | Size |
|---|---|---|
| SKILL.md | skill md | 5.3 KB |
SKILL.md
View on GitHub---
name: monitoring-setup
description: Instruments a service so problems are visible before users report them — metrics, logs, traces, dashboards, and alerts people will actually act on. Use this whenever the user is adding monitoring, mentions Prometheus, Grafana, OpenTelemetry, SLOs, dashboards, or alerting, has a service nobody is watching, or is drowning in alerts nobody reads. For instrumenting an AI agent specifically, use agent-observability.
license: MIT
---
# Monitoring setup
Two failure modes, and most teams manage both simultaneously: **nothing is monitored**, so
problems surface as customer complaints; and **everything is alerted**, so the channel is noise
and real alerts are missed.
The target is a small number of signals that mean something, and an even smaller number of alerts
that always warrant waking someone.
## 1. Instrument the four signals that matter
For any request-serving system, start here and resist adding more until these are solid:
- **Rate:** requests per second, by endpoint and status
- **Errors:** failures per second, by type. Distinguish client errors from server errors; a rise
in 400s is a different problem from a rise in 500s
- **Duration:** latency as a **histogram**, not an average. You cannot recover percentiles from
a mean, and the mean hides the tail where the pain is
- **Saturation:** how full the constrained resource is: connection pool, thread pool, queue
depth, memory, disk
For queues and batch systems, substitute queue depth, age of the oldest item, and processing lag.
**Age of the oldest unprocessed item** is usually the single best health signal for a queue —
better than depth, which can look fine while one item is stuck.
**Done when:** you can answer "is it healthy?" from four graphs.
## 2. Use the standard instrumentation
Use OpenTelemetry rather than a vendor SDK. Instrumentation is expensive to write and painful to
replace; the standard keeps you able to change backends without re-instrumenting.
Take the automatic instrumentation for your framework, HTTP client, and database first — it
covers most of what you need for no effort. Add manual spans only for meaningful business
operations.
**Propagate trace context** across service boundaries from the start. Retrofitting it means
touching every call site, and without it you cannot follow a request through the system.
**Done when:** one request produces one connected trace across every service it touches.
## 3. Keep cardinality under control
The most common way monitoring becomes expensive and then gets switched off.
Every unique label combination is a separate time series. A label with a user ID, a request ID,
or a raw URL path creates unbounded series and will eventually take down the metrics backend or
the budget.
- **Labels must be bounded:** endpoint *template* (`/orders/:id`), status class, region, version
- **Never** user ID, request ID, email, session, or full path
- High-cardinality detail belongs in **traces and logs**, which are built for it
**Done when:** every label has a knowable, small set of values.
## 4. Alert on symptoms, not causes
Alert on what users experience. Cause-based alerts fire constantly and often for conditions
nobody needs to act on.
- **Good:** error rate above X for 5 minutes; p99 latency above target; queue age above the SLA;
a job that did not run
- **Bad:** CPU above 80%. Sometimes that is fine. If high CPU is not causing user-visible
problems, it is a dashboard line, not a page
Every alert needs:
- **A runbook:** what to check, what to do. An alert without one is an interruption, not
information
- **A severity that matches reality.** Page only for things that need a human *now*. Everything
else is a ticket
- **A tuned threshold and duration.** An alert that fires on a 30-second blip trains people to
ignore it
**If an alert has fired ten times and nobody acted, delete it.** That is the discipline that
keeps the channel meaningful.
**Done when:** every alert would change someone's behaviour at 3am.
## 5. Build dashboards for a question
A dashboard with forty panels is used by nobody. Build one per question:
- **Service health:** the four signals, one screen, readable in ten seconds during an incident
- **Dependency health:** what this service calls, and how those calls are doing
- **Business metrics:** orders, signups, whatever the system exists to do. Frequently the
fastest indicator that something is wrong
Put the most important graph top-left, and annotate deploys onto the timeline. Correlating a
change with a deploy is the most common thing anyone does with a dashboard.
**Done when:** the health dashboard answers "is it broken?" without scrolling.
## 6. Define what good means
Pick one or two SLOs — a target and a window — for the things users actually care about, and set
an error budget. This turns "is it slow?" into a question with an answer, and turns alerting
thresholds into something derived rather than guessed.
Do not define an SLO for every metric. Two meaningful ones beat twelve nobody references.
**Done when:** you can say whether the service met its target last month.
## Report
State what is instrumented, what is alerted and at what threshold, what the dashboards answer,
and — importantly — what is *not* covered. The uninstrumented path is where the next surprise
comes from, and naming it is more useful than a longer list of what is covered.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| reverse-engineer-anythingmorluto/reaReverse engineer native, managed, Electron/JavaScript, packaged, firmware, and browser targets with REA, and build agent environments (AgentEnv bundles) of web applications from a live instance or a capture, and of Linux desktop apps with a task from a request. Use shipped-artifact or requested runtime evidence to explain features, compare versions, decompile code, or guide a reconstruction. Skip REA for ordinary source-repository architecture analysis. | Stars: 472k | Updated | Claude CodeCodex | Verified | Install |
| diagnosing-superpowersobra/superpowersUse when a superpowers session went wrong and your human partner wants to know why — repeated work, ignored plans, stumbles, poor results, a skill that didn't fire, "it took too long", "why is it so expensive", "what is it doing" — or wants to build a bug report for the superpowers maintainers, for the current session or a past one identified by id or path, on any harness. | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| dispatching-parallel-agentsobra/superpowersUse when facing 2+ independent tasks that can be worked on without shared state or sequential dependencies | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| executing-plansobra/superpowersUse when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |