artubss/SKILLS-CLAUDE-CODE / favorites/tpl-devops-monitoring-observability
tpl-devops-monitoring-observability
VerifiedTemplate do pack (devops/06-monitoring-observability.md). Orienta o agente em infra, deploy, CI/CD e operacao alinhado a esse contexto.
PROJECT: Monitoring & Observability Stack (Prometheus + Grafana + Loki + OpenTelemetry)
Skill gerado a partir do pack templates-claude-code. Arquivo de origem: devops/06-monitoring-observability.md. Use como baseline e adapte ao projeto antes de mudancas grandes.
Conteudo do template
STACK
- Metrics: Prometheus 2.50+ + Alertmanager
- Visualization: Grafana 10+ (dashboard-as-code with Grafonnet/JSON)
- Logs: Loki 3 + Promtail / Alloy
- Tracing: OpenTelemetry Collector + Tempo (or Jaeger)
- App instrumentation: Node.js +
@opentelemetry/sdk-node+prom-client - Deployment: Docker Compose (dev) / Helm kube-prometheus-stack (prod)
ARCHITECTURE RULES
- Instrument at the boundary — measure HTTP in/out, DB calls, queue produce/consume. Not every internal function.
- RED method for services — Rate (req/s), Errors (error rate %), Duration (latency p50/p95/p99). Always.
- USE method for resources — Utilization, Saturation, Errors for CPU/memory/disk/network.
- Metric naming is permanent — rename = breaking change. Use
{service}_{noun}_{unit}_{type}format. - Logs are for events, metrics for aggregates — never log
request_count++; emit a counter metric. - Structured JSON logs only — no
console.log("User logged in"). Uselogger.info({ userId, action }). - Alert on symptoms, not causes — alert on high error rate (symptom), not "CPU > 80%" (cause).
- Dashboards are code — store as JSON in
monitoring/dashboards/; never click-to-build only.
NODE.JS INSTRUMENTATION
// src/telemetry.ts — import BEFORE everything else in src/index.ts
import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http'
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics'
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http'
import { ExpressInstrumentation } from '@opentelemetry/instrumentation-express'
import { PgInstrumentation } from '@opentelemetry/instrumentation-pg'
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT || 'http://otel-collector:4318/v1/traces',
}),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: process.env.OTEL_EXPORTER_OTLP_METRICS_ENDPOINT || 'http://otel-collector:4318/v1/metrics',
}),
exportIntervalMillis: 15000,
}),
instrumentations: [
new HttpInstrumentation({
ignoreIncomingPaths: ['/healthz', '/metrics'], // Don't trace health checks
}),
new ExpressInstrumentation(),
new PgInstrumentation(),
],
serviceName: process.env.OTEL_SERVICE_NAME || 'my-app',
})
sdk.start()
process.on('SIGTERM', () => sdk.shutdown())
PROMETHEUS METRICS (Node.js)
// src/metrics.ts
import { Registry, Counter, Histogram, Gauge } from 'prom-client'
export const registry = new Registry()
registry.setDefaultLabels({ service: 'my-app', env: process.env.NODE_ENV || 'dev' })
// ---- HTTP metrics (RED) ----
export const httpRequestsTotal = new Counter({
name: 'http_requests_total',
help: 'Total HTTP requests',
labelNames: ['method', 'route', 'status_code'],
registers: [registry],
})
export const httpRequestDuration = new Histogram({
name: 'http_request_duration_seconds',
help: 'HTTP request duration in seconds',
labelNames: ['method', 'route', 'status_code'],
buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
registers: [registry],
})
// ---- Business metrics ----
export const activeUsers = new Gauge({
name: 'app_active_users',
help: 'Currently active authenticated users',
registers: [registry],
})
export const ordersProcessed = new Counter({
name: 'app_orders_processed_total',
help: 'Total orders processed',
labelNames: ['status'], // success | failed
registers: [registry],
})
// ---- DB connection pool ----
export const dbPoolSize = new Gauge({
name: 'pg_pool_size',
help: 'PostgreSQL connection pool size',
labelNames: ['state'], // idle | active | waiting
registers: [registry],
})
MIDDLEWARE: METRICS + LOGGING
// src/middleware/observability.ts
import { Request, Response, NextFunction } from 'express'
import { httpRequestsTotal, httpRequestDuration } from '../metrics'
import logger from '../logger'
export function metricsMiddleware(req: Request, res: Response, next: NextFunction) {
const start = Date.now()
const route = req.route?.path || req.path
res.on('finish', () => {
const duration = (Date.now() - start) / 1000
const labels = { method: req.method, route, status_code: String(res.statusCode) }
httpRequestsTotal.inc(labels)
httpRequestDuration.observe(labels, duration)
logger.info({
type: 'http_request',
method: req.method,
path: req.path,
status: res.statusCode,
duration_ms: Math.round(duration * 1000),
request_id: req.headers['x-request-id'],
user_id: (req as any).user?.id,
})
})
next()
}
STRUCTURED LOGGER
// src/logger.ts — always use this, never console.log()
import pino from 'pino'
const logger = pino({
level: process.env.LOG_LEVEL || 'info',
...(process.env.NODE_ENV === 'development'
? { transport: { target: 'pino-pretty' } }
: {}
),
base: {
service: process.env.OTEL_SERVICE_NAME || 'my-app',
env: process.env.NODE_ENV,
version: process.env.APP_VERSION,
},
redact: ['req.headers.authorization', 'body.password', 'body.token'],
serializers: {
err: pino.stdSerializers.err,
},
})
export default logger
// Usage:
// logger.info({ userId: '123', action: 'login' }, 'User authenticated')
// logger.error({ err, orderId }, 'Order processing failed')
// NEVER: logger.info('User ' + userId + ' logged in')
METRIC NAMING CONVENTIONS
Format: {service}_{noun}_{unit}_{type}
✅ http_requests_total (counter)
✅ http_request_duration_seconds (histogram)
✅ pg_pool_connections (gauge)
✅ queue_messages_total (counter, label: status=processed|failed)
✅ cache_hit_ratio (gauge, 0.0-1.0)
❌ requestCount (no service prefix, no unit, camelCase)
❌ error_rate (rate is computed by PromQL, not stored)
❌ my_app_latency_ms (use _seconds, Prometheus convention)
ALERT RULES
# monitoring/alerts/app.yml
groups:
- name: app.rules
rules:
# Alert on symptoms, not causes
- alert: HighErrorRate
expr: |
rate(http_requests_total{status_code=~"5.."}[5m])
/ rate(http_requests_total[5m]) > 0.05
for: 2m
labels:
severity: critical
team: backend
annotations:
summary: "High error rate on {{ $labels.service }}"
description: "Error rate is {{ $value | humanizePercentage }} (threshold: 5%)"
runbook: "https://runbooks.internal/high-error-rate"
- alert: HighLatencyP99
expr: |
histogram_quantile(0.99,
rate(http_request_duration_seconds_bucket[5m])
) > 2.0
for: 5m
labels:
severity: warning
annotations:
summary: "P99 latency > 2s on {{ $labels.service }}"
- alert: ServiceDown
expr: up{job="my-app"} == 0
for: 1m
labels:
severity: critical
annotations:
summary: "{{ $labels.instance }} is down"
# Log-based alert via Loki ruler
- alert: PanicInLogs
expr: |
count_over_time({app="my-app"} |= "panic" [5m]) > 0
labels:
severity: critical
PROMETHEUS SCRAPE CONFIG
# prometheus/prometheus.yml
global:
scrape_interval: 15s
evaluation_interval: 15s
rule_files:
- /etc/prometheus/rules/*.yml
alerting:
alertmanagers:
- static_configs:
- targets: ['alertmanager:9093']
scrape_configs:
- job_name: my-app
static_configs:
- targets: ['app:3000']
metrics_path: /metrics
relabel_configs:
- source_labels: [__address__]
target_label: instance
- job_name: node-exporter
static_configs:
- targets: ['node-exporter:9100']
- job_name: postgres-exporter
static_configs:
- targets: ['postgres-exporter:9187']
GRAFANA DASHBOARD AS CODE (JSON snippet)
{
"title": "App RED Dashboard",
"uid": "app-red-v1",
"tags": ["app", "red"],
"panels": [
{
"title": "Request Rate",
"type": "timeseries",
"targets": [{
"expr": "sum(rate(http_requests_total[5m])) by (route)",
"legendFormat": "{{ route }}"
}]
},
{
"title": "Error Rate",
"type": "stat",
"targets": [{
"expr": "sum(rate(http_requests_total{status_code=~'5..'}[5m])) / sum(rate(http_requests_total[5m]))",
"legendFormat": "Error %"
}],
"thresholds": {
"steps": [
{"color": "green", "value": null},
{"color": "yellow", "value": 0.01},
{"color": "red", "value": 0.05}
]
}
},
{
"title": "P95 Latency",
"type": "timeseries",
"targets": [{
"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, route))",
"legendFormat": "p95 {{ route }}"
}]
}
]
}
OTEL COLLECTOR CONFIG
# otel-collector/config.yml
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
send_batch_size: 1024
timeout: 5s
memory_limiter:
limit_mib: 512
check_interval: 5s
exporters:
prometheus:
endpoint: 0.0.0.0:8889
loki:
endpoint: http://loki:3100/loki/api/v1/push
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch, memory_limiter]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [batch]
exporters: [loki]
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 119k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |