Skip to content

cogni-dao/cogni / .claude/skills/monitoring-expert

monitoring-expert

Verified

Configures monitoring systems, implements structured logging pipelines, creates Prometheus/Grafana dashboards, defines alerting rules, and instruments distributed tracing. Implements Prometheus/Grafana stacks, conducts load testing, performs application profiling, and plans infrastructure capacity. Use when setting up application monitoring, adding observability to services, debugging production issues with logs/metrics/traces, running load tests with k6 or Artillery, profiling CPU/memory bottlenecks, or forecasting capacity needs.

Monitoring Expert

Observability and performance specialist implementing comprehensive monitoring, alerting, tracing, and performance testing systems.

Core Workflow

  1. Assess — Identify what needs monitoring (SLIs, critical paths, business metrics)
  2. Instrument — Add logging, metrics, and traces to the application (see examples below)
  3. Collect — Configure aggregation and storage (Prometheus scrape, log shipper, OTLP endpoint); verify data arrives before proceeding
  4. Visualize — Build dashboards using RED (Rate/Errors/Duration) or USE (Utilization/Saturation/Errors) methods
  5. Alert — Define threshold and anomaly alerts on critical paths; validate no false-positive flood before shipping

Quick-Start Examples

Structured Logging (Node.js / Pino)

import pino from "pino";

const logger = pino({ level: "info" });

// Good — structured fields, includes correlation ID
logger.info(
  { requestId: req.id, userId: req.user.id, durationMs: elapsed },
  "order.created"
);

// Bad — string interpolation, no correlation
console.log(`Order created for user ${userId}`);

Prometheus Metrics (Node.js)

import { Counter, Histogram, register } from "prom-client";

const httpRequests = new Counter({
  name: "http_requests_total",
  help: "Total HTTP requests",
  labelNames: ["method", "route", "status"],
});

const httpDuration = new Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP request latency",
  labelNames: ["method", "route"],
  buckets: [0.05, 0.1, 0.3, 0.5, 1, 2, 5],
});

// Instrument a route
app.use((req, res, next) => {
  const end = httpDuration.startTimer({ method: req.method, route: req.path });
  res.on("finish", () => {
    httpRequests.inc({
      method: req.method,
      route: req.path,
      status: res.statusCode,
    });
    end();
  });
  next();
});

// Expose scrape endpoint
app.get("/metrics", async (req, res) => {
  res.set("Content-Type", register.contentType);
  res.end(await register.metrics());
});

OpenTelemetry Tracing (Node.js)

import { NodeSDK } from "@opentelemetry/sdk-node";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-http";
import { trace } from "@opentelemetry/api";

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({ url: "http://jaeger:4318/v1/traces" }),
});
sdk.start();

// Manual span around a critical operation
const tracer = trace.getTracer("order-service");
async function processOrder(orderId) {
  const span = tracer.startSpan("order.process");
  span.setAttribute("order.id", orderId);
  try {
    const result = await db.saveOrder(orderId);
    span.setStatus({ code: SpanStatusCode.OK });
    return result;
  } catch (err) {
    span.recordException(err);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw err;
  } finally {
    span.end();
  }
}

Prometheus Alerting Rule

groups:
  - name: api.rules
    rules:
      - alert: HighErrorRate
        expr: |
          rate(http_requests_total{status=~"5.."}[5m])
          / rate(http_requests_total[5m]) > 0.05
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Error rate above 5% on {{ $labels.route }}"

k6 Load Test

import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  stages: [
    { duration: "1m", target: 50 }, // ramp up
    { duration: "5m", target: 50 }, // sustained load
    { duration: "1m", target: 0 }, // ramp down
  ],
  thresholds: {
    http_req_duration: ["p(95)<500"], // 95th percentile < 500 ms
    http_req_failed: ["rate<0.01"], // error rate < 1%
  },
};

export default function () {
  const res = http.get("https://api.example.com/orders");
  check(res, { "status is 200": (r) => r.status === 200 });
  sleep(1);
}

Reference Guide

Load detailed guidance based on context:

TopicReferenceLoad When
Loggingreferences/structured-logging.mdPino, JSON logging
Metricsreferences/prometheus-metrics.mdCounter, Histogram, Gauge
Tracingreferences/opentelemetry.mdOpenTelemetry, spans
Alertingreferences/alerting-rules.mdPrometheus alerts
Dashboardsreferences/dashboards.mdRED/USE method, Grafana
Performance Testingreferences/performance-testing.mdLoad testing, k6, Artillery, benchmarks
Profilingreferences/application-profiling.mdCPU/memory profiling, bottlenecks
Capacity Planningreferences/capacity-planning.mdScaling, forecasting, budgets
Cogni Fleet Diagnosishub guideRed fleet status, node 52x, shared-substrate attribution

Fleet availability and deployment identity

Continuous Serve probes measure availability: TLS, HTTP 200, a healthy response, and the configured deadline. Never pin a release SHA in their body assertions, labels, or required inputs: normal flights and promotions must not require a separate monitoring apply to remain available.

Check actual build identity through /version and /readyz diagnostics. Keep exact expected-versus-actual SHA verification in flight, promotion, and candidate validation gates. A wrong build fails deployment verification; it is not, by itself, evidence of an availability outage. Incident-scoped recovery proofs retain their exact build, native database, and incident identity assertions.

When HTTP 200 produces probe_success=0, inspect probe_failed_due_to_regex and the configured assertions before attributing an outage to a database or host. Follow the existing fleet diagnosis guide.

Constraints

MUST DO

  • Use structured logging (JSON)
  • Include request IDs for correlation
  • Set up alerts for critical paths
  • Monitor business metrics, not just technical
  • Use appropriate metric types (counter/gauge/histogram)
  • Implement health check endpoints

MUST NOT DO

  • Log sensitive data (passwords, tokens, PII)
  • Alert on every error (alert fatigue)
  • Use string interpolation in logs (use structured fields)
  • Skip correlation IDs in distributed systems
finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 322kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 322kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 322kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 198kUpdated Claude CodeCodexVerifiedInstall