Skip to content

EliasOulkadi/shokunin / .pack/skills/error-handler

error-handler

Verified

Design error handling, structured logging, and observability with OpenTelemetry (traces, metrics, logs), error classification, recovery patterns (retry with jitter, circuit breaker, bulkhead, timeout), error budgets/SLOs with burn rate alerts, and production incident triage. Use when user asks to implement error handling, logging, monitoring, observability, OpenTelemetry, error boundaries, circuit breakers, retry logic, or SLO tracking. Do NOT use for incident runbooks (use runbook-gen), vendor-specific APM setup (Datadog, Sentry agent config), or K8s debugging.

Error Handler

Observability that makes debugging fast and production predictable. Based on Google SRE (Site Reliability Engineering), OpenTelemetry semantic conventions, and production patterns from Sentry, Honeycomb, and Datadog.

Sub-Commands

CommandDescription
setupSet up OpenTelemetry SDK + instrumentation for a service
classifyClassify errors by HTTP status, severity, and alert priority
recoverImplement recovery patterns: retry with jitter, circuit breaker, timeout
sloDefine error budget, SLO, and burn rate alerts
auditAudit existing error handling against best practices

The 3 Signals (OpenTelemetry)

SignalPurposeExample
TracesFollow a request across servicesUser request ? API Gateway ? Auth ? DB
MetricsAggregate measurements over timeRequest count, error rate, latency p50/p95/p99
LogsDiscrete events with context"User login failed: invalid credentials for user_abc123"

All three signals share trace_id + span_id for correlation.

Workflow

Step 1: Classify errors

CategoryHTTPSeverityLog levelSLO impactAlert?
Validation (user error)400LowinfoNoNo
Authentication401MediumwarnNoOnly if spike detected
Authorization403MediumwarnNoOnly if spike detected
Not Found404LowdebugNoNo
Conflict409MediuminfoNoNo
Rate Limit429LowwarnNoNo
Internal500HigherrorYesYes
Downstream failure502/503HigherrorYesYes
Timeout504HigherrorYesYes

Step 2: Set up OpenTelemetry (exact configuration)

import { NodeSDK } from '@opentelemetry/sdk-node'
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http'
import { Resource } from '@opentelemetry/resources'
import { ATTR_SERVICE_NAME } from '@opentelemetry/semantic-conventions'

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'api-service',
    'deployment.environment': process.env.NODE_ENV,
  }),
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT,
  }),
  instrumentations: [
    new HttpInstrumentation(),
    new ExpressInstrumentation(),
    new PgInstrumentation(),
    new RedisInstrumentation(),
  ],
})

sdk.start()

process.on('SIGTERM', async () => {
  await sdk.shutdown()
})

Sampling strategy:

  • Dev/staging: AlwaysOn (100%)
  • Production: ParentBased + TraceIdRatioBased(0.1) (10% head-based)
  • Never sample errors out. Use AlwaysOn for traces with status=ERROR.

Import note: ParentBasedSampler is imported from @opentelemetry/sdk-trace-base, NOT from @opentelemetry/sdk-node. If using @opentelemetry/sdk-node, configure it via the sampler option in NodeSDK constructor.

Step 3: Implement structured error middleware

import { trace, SpanStatusCode } from '@opentelemetry/api'
import { randomUUID } from 'crypto'

app.use((err: Error, req: Request, res: Response, next: NextFunction) => {
  const span = trace.getActiveSpan()
  const requestId = req.headers['x-request-id'] || randomUUID()

  span?.recordException(err)
  span?.setStatus({ code: SpanStatusCode.ERROR, message: err.message })

  logger.error({
    message: err.message,
    errorName: err.name,
    stack: err.stack,
    requestId,
    trace_id: span?.spanContext().traceId,
    span_id: span?.spanContext().spanId,
    path: req.path,
    method: req.method,
    userId: req.user?.id,
  })

  if (err instanceof ValidationError) {
    return res.status(400).json({
      error: {
        code: 'VALIDATION_ERROR',
        message: err.message,
        details: err.details,
        requestId,
      },
    })
  }

  if (err instanceof NotFoundError) {
    return res.status(404).json({
      error: { code: 'NOT_FOUND', message: err.message, requestId },
    })
  }

  // Generic fallback - never leak internal state
  res.status(500).json({
    error: {
      code: 'INTERNAL_ERROR',
      message: 'An unexpected error occurred',
      requestId,
    },
  })
})

Step 4: Recovery patterns (exact implementations)

Retry with exponential backoff + jitter

async function withRetry<T>(
  fn: () => Promise<T>,
  options: { maxRetries?: number; baseMs?: number; maxMs?: number } = {}
): Promise<T> {
  const { maxRetries = 3, baseMs = 200, maxMs = 10000 } = options

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn()
    } catch (error) {
      if (attempt === maxRetries) throw error
      if (!isRetryable(error)) throw error

      const exponentialDelay = baseMs * Math.pow(2, attempt)
      const jitter = Math.random() * 0.3 * exponentialDelay
      const delay = Math.min(exponentialDelay + jitter, maxMs)

      await new Promise(r => setTimeout(r, delay))
    }
  }
  throw new Error('Unreachable')
}

function isRetryable(error: any): boolean {
  const retryableStatuses = [408, 429, 502, 503, 504]
  if (error?.status && retryableStatuses.includes(error.status)) return true
  if (error?.code === 'ETIMEDOUT' || error?.code === 'ECONNRESET') return true
  return false
}

// Never retry: 400, 401, 403, 404, 409, 422

Circuit breaker

class CircuitBreaker {
  private failures = 0
  private state: 'closed' | 'open' | 'half-open' = 'closed'
  private lastFailureTime = 0

  constructor(
    private failureThreshold: number = 5,
    private resetTimeout: number = 30000
  ) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'open') {
      if (Date.now() - this.lastFailureTime > this.resetTimeout) {
        this.state = 'half-open'
      } else {
        throw new Error('Circuit breaker is open')
      }
    }

    try {
      const result = await fn()
      if (this.state === 'half-open') {
        this.state = 'closed'
        this.failures = 0
      }
      return result
    } catch (error) {
      this.failures++
      this.lastFailureTime = Date.now()

      if (this.failures >= this.failureThreshold) {
        this.state = 'open'
      }
      throw error
    }
  }
}

Timeout

function withTimeout<T>(promise: Promise<T>, timeoutMs: number): Promise<T> {
  const timeout = new Promise<T>((_, reject) =>
    setTimeout(() => reject(new Error(`Operation timed out after ${timeoutMs}ms`)), timeoutMs)
  )
  return Promise.race([promise, timeout])
}

// Usage
const result = await withTimeout(fetch('https://api.example.com/data'), 5000)

Step 5: Error budgets + SLO

SLOError budget (monthly)Burn rate criticalBurn rate warning
99.9%43m 50s6x (exhausted in 4.6d)3x (exhausted in 9.3d)
99.5%3h 39m10x (exhausted in 1.1d)5x (exhausted in 2.2d)
99%7h 18m15x (exhausted in 0.5d)7.5x (exhausted in 1d)

Burn rate alert (PromQL for 99.9% SLO):

sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
> 0.001 * 6

Multi-window, multi-burn-rate approach (Google SRE):

  • Short window (5m) + high burn rate (6-10x) ? Page on-call immediately
  • Long window (1h) + low burn rate (2-3x) ? Ticket for next business day

Structured Logging Schema

Every log line must be JSON with these fields:

{
  "timestamp": "2026-05-16T10:21:27.000Z",
  "level": "error",
  "message": "Failed to process payment",
  "trace_id": "0af7651916cd43dd8448eb211c80319c",
  "span_id": "b7ad6b7169203331",
  "request_id": "req_a1b2c3d4",
  "service": "payment-service",
  "environment": "production",
  "userId": "user_abc123",
  "path": "/api/payments",
  "method": "POST",
  "duration_ms": 234,
  "error": {
    "name": "PaymentFailedError",
    "message": "Card declined",
    "code": "CARD_DECLINED"
  }
}

Rules:

  • Never log PII, tokens, passwords, or full credit card numbers
  • Always include trace_id, span_id, request_id for correlation
  • duration_ms on every request. p95 > 200ms ? investigate.

Production Checklist

  • OpenTelemetry SDK initialized at app startup (before any imports)
  • Auto-instrumentations for HTTP, DB, messaging, Redis
  • Every route handler wrapped in try/catch or error middleware
  • Structured JSON logging (never plain text)
  • trace_id + span_id + request_id in every log line
  • Recovery: retry with jitter on external calls. Timeout on every external call.
  • Circuit breaker for critical downstream services
  • SLO defined for every service (99.9% critical, 99.5% standard)
  • Burn rate alerts: multi-window (5m short, 1h long)
  • No PII, tokens, or secrets in logs or error messages
  • Error classification by type (not generic catch-all)
  • Sampling: 10% production, 100% for errors

Anti-Patterns

Anti-patternFix
Log in catch AND rethrowLog OR throw. Not both.
Generic error messagesInclude code, request_id, details.
No trace context in async flowsPropagate context via context.active().
Silent catch with no logEvery catch logs, handles, or rethrows.
Retry non-retryable errors (400, 401)Check status before retry: isRetryable().
No timeout on external callsAlways set connection (2s) + read (10s) timeouts.
No error classificationClassify by type and handle accordingly.
Plain text logsStructured JSON with consistent fields.
Same retry delay every timeExponential backoff + jitter.
Circuit breaker never testedTest in staging by killing dependencies.
Error budget without alertBurn rate alerts are mandatory.

Error Handling

ErrorCauseFix
OpenTelemetry SDK not exporting tracesExporter endpoint unreachable or OTEL_EXPORTER_OTLP_ENDPOINT not setVerify endpoint with curl $OTEL_EXPORTER_OTLP_ENDPOINT/v1/traces. Set env var before SDK initialization.
ParentBasedSampler import fails from @opentelemetry/sdk-nodeWrong import pathImport from @opentelemetry/sdk-trace-base, not @opentelemetry/sdk-node. Configure via sampler option in NodeSDK constructor.
Traces missing in production but appear in stagingSampling rate too low (TraceIdRatioBased(0.1) drops 90%)Use AlwaysOn for traces with status=ERROR. Never sample errors out.
Structured logs have null trace_id/span_idContext not propagated into async flow or logger not context-awareUse context.active() before async operations. Inject trace_id/span_id into every log line via context propagation.
Circuit breaker stays open indefinitelyresetTimeout is set but half-open state logic never triggersVerify Date.now() - this.lastFailureTime > this.resetTimeout comparison. Add logging when state transitions occur.
Retry exhausts budget on non-retryable errorsisRetryable() returns true for 400/401/403 status codesHard-gate retryable statuses: only [408, 429, 502, 503, 504] and network errors (ETIMEDOUT, ECONNRESET).
Retry with jitter produces thundering herdAll instances retry at the same time due to identical base delayRandomize baseMs per instance or use Math.random() * baseMs as the starting backoff.
Burn rate alert fires for low-traffic servicesDivision by small request count amplifies noiseSet minimum request thresholds: only evaluate SLO when rate(http_requests_total[5m]) > 1.
Error middleware exposes stack traces to usersGeneric INTERNAL_ERROR handler includes err.stack in response bodyNever return stack in API responses. Log it server-side; respond with { error: { code, message, requestId } }.
PII leaks into structured logsFull email, IP, SSN, or credit card numbers in log fieldsMask sensitive fields: email.replace(/(.{3}).*(@.*)/, '$1***$2'). Configure PII redaction in log pipeline.
Timeout wrapped around fetch never firesPromise.race with setTimeout races against a promise that catches and suppresses errorsAlways reject in the timeout handler, not resolve. The timed-out promise must reject, not resolve.

Sources

  • Google SRE Book - Monitoring Distributed Systems (Chapter 6)
  • Google SRE Workbook - Implementing SLOs (Chapter 5)
  • OpenTelemetry documentation (opentelemetry.io)
  • OpenTelemetry semantic conventions
  • Sentry error handling best practices
  • AWS Well-Architected Framework - Reliability Pillar
  • Microsoft Polly - Circuit breaker patterns
  • Hystrix - Netflix circuit breaker patterns

Checklist

  • Skill loads without errors in the AI agent
  • YAML frontmatter is valid (description, compatibility, audience)
  • Workflow section provides clear step-by-step instructions
  • Error handling section covers common failure modes
  • All referenced files (references/, scripts/, assets/) exist
  • Skill triggers correctly for intended use cases
  • No broken links or missing resources
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 318kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 196kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 118kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 117kUpdated Claude CodeCodexVerifiedInstall