Skip to content

supercheck-io/supercheck / .agents/skills/sre-operations

supercheck-sre-operations

Verified

Work on Supercheck AI SRE agents, incidents and investigations, services, connectors, private agents, tool execution, resilience, memory/performance, environment configuration and feature flags, self-hosting with Docker Compose or k3s, CI/CD, or release management.

Supercheck SRE operations

AI SRE architecture

flowchart LR
  SIGNAL[Monitor and telemetry signals] --> INCIDENT[Incident]
  INCIDENT --> ORCH[AI SRE orchestrator]
  ORCH --> AGENT[Specialized agents]
  AGENT --> TOOLS[Scoped evidence tools]
  TOOLS --> BRIEF[Evidence and diagnosis]
  BRIEF --> HUMAN{Authorization gate}
  HUMAN -->|approved| MUTATE[Bounded action]
  • AI SRE source is split between app/src/sre, app/src/lib/sre, API routes, components, and matching worker schema/types.
  • Incidents, services, connectors, private agents, conversations, messages, evidence, tool calls, and usage remain scoped to their owning organization/project.
  • Authorization is enforced server-side for every tool call and persisted resource. Model instructions and UI visibility are not authorization.
  • SRE_ENABLED (default on) alone makes monitor and job alerts open incidents; SRE_ENABLED=false disables all SRE work. SRE_AUTOMATION_ENABLED=true|1 gates only the AI triage step on top of that, and only the app reads it; the worker reads SRE_ENABLED only.
  • Prefer read-only evidence gathering. Mutations require explicit user authority, least-privilege credentials, bounded targets, audit records, and observable outcomes.
  • Kubernetes access uses namespace-scoped service accounts and allowed resources/actions; never grant cluster-admin.
  • Preserve stable assistant/user message identity through streaming, persistence, polling/reconciliation, retries, and reloads.
  • Incident lifecycle (assign, unassign, acknowledge, commander, resolve) is in app/src/actions/sre-incident-lifecycle.ts and /api/sre/incidents/{id}/assign|acknowledge|resolve, all gated by sre_incident:update. Resolve needs a summary (max 2000 characters); assignees must be org owner/admin or project admin/editor. Incident notifications use a fixed body and never include AI text or raw evidence.
  • Alert to incident: the worker queues each sent or failed (not pending) alert on the sre-alert-triage queue (Bull dashboard label "Incident Events"), whose job only opens or resolves the incident and then starts AI triage in the background (the job waits for admission and billing only; a 402 or 409 is a logged skip; a run that never settles is failed by stale-run-recovery.ts); a recovery enqueue is retried in place (250 ms, 1 s) and its failure never suppresses the user's recovery alert; when the organization's AI run cap is full the job is delayed (60 s steps, up to 30 min) instead of dropping triage; app/src/sre/lib/background-alert-triage.ts opens one incident per source (monitor or job), joins later failures to it via sre_incident_alerts, and titles it <target>: <alert type> (app/src/lib/sre/alert-labels.ts). Per check, alertConfig has incidentEnabled (default on), incidentSeverity (auto or sev1-sev4) and autoResolveIncident (default on).
  • Auto-resolve: when a check recovers and autoResolveIncident is on, an unacknowledged auto-opened incident resolves even if recovery alerts are off or no channel is notified; an acknowledged one stays open with a note. A new failure within 30 minutes (INCIDENT_REOPEN_WINDOW_MS in app/src/lib/sre/incident-lifecycle.ts) reopens it. The unacknowledged-incident reminder (app/src/lib/sre/incident-renotify.ts) counts from sre_incidents.reopened_at when set, else created_at.
  • Incident sweep (app/src/sre/lib/incident-sweep.ts, called from the 60 s incident re-notify tick in incident-renotify.ts, no extra queue): behind SRE_ENABLED, under one transaction advisory lock so a second replica skips the tick, at most 25 monitor sources and 25 job sources for recovery and 25 incidents for re-triage per tick, each branch with half of a 60 s budget. (a) Recovery reconcile resolves open, unacknowledged, automatically created incidents (trigger source monitor/job, same org/project) whose source recovered after the incident's last_alert_at (monitor up and last_status_change_at later; job's latest run passed), through handleSourceRecovery with a deterministic ...:sweep discriminator. It skips autoResolveIncident: false and recoveryThreshold above 1 (the worker's signal owns the streak). It recovers a lost sourceRecovery enqueue. (b) Re-triage (the project auto-investigate policy can retry even with SRE_AUTOMATION_ENABLED off; the daily token cap applies either way) calls startAutomaticIncidentTriage (the shared gated tail of the alert job: auto-investigate policy, SRE_AUTOMATION_ENABLED, AI config, billing, org run cap) for an open incident under 2 days old whose only origin = 'auto' run failed with errorCode: stale_run (a dead running run is failed first); the one-automatic-run count is the durable once-only guard. Candidate queries use existing indexes (sre_incidents_active_status_idx, sre_investigation_runs_incident_id_idx, runs_job_status_idx); no migration.
  • Incident emails go to participants (assignee, commander, and the previous assignee on reassign) through notifyIncidentEvent in app/src/lib/sre/incident-notifier.ts: never to the actor, only to users listAssignableUsers still returns, and recovered/reopened at most once per person per 10 minutes. Channels skip recovered and automatic resolved/reopened, because the worker already posted them. Each sre_incident_alerts row snapshots the alert (alert_title, alert_description, alert_severity, alert_fired_at, migration 0046); read it through linkedAlertFields in app/src/lib/sre/incident-alert-snapshot.ts, because the shared sre_alert_events row is refreshed by later failures.
  • Watchers (sre_incident_watchers, app/src/lib/sre/incident-watchers.ts) get incident emails like participants; a read-only viewer may watch. The per-project Default notification policy (notification_policies, app/src/lib/sre/notification-policy.ts, app/src/actions/notification-policy.ts) sets channels, service-owner email on SEV1-2 and one re-notify of an unacknowledged incident (sre_incidents.renotified_at, scheduler in app/src/lib/sre/incident-renotify.ts).
  • The timeline composer can also post to the linked status page. AI drafts the customer status update (app/src/actions/sre-status-update-draft.ts, app/src/lib/sre/status-update-draft.ts) as text only; a person edits and posts it. Ask Copilot on the incident page links to /copilot?incident=<id>.
  • Auto-investigation (app/src/sre/lib/auto-investigate.ts) is opt-in per policy and off by default (auto_investigate_enabled_at, migration 0058): SEV1-2 full investigation, SEV3-4 triage. It takes an advisory lock, holds no DB transaction across the AI call, and retries after a failed run, up to MAX_AUTO_ATTEMPTS (3) per incident. Attempts count only automatic runs (sre_investigation_runs.origin = 'auto', migration 0061; manual runs are uncapped and never block an automatic one). At most MAX_AUTO_RUNNING_PER_ORG (3) automatic runs are in flight per organization, counted and inserted in insertSreRun (app/src/sre/lib/run-insert.ts) under an organization advisory lock.
  • Six display statuses (app/src/lib/sre/incident-status-display.ts: Triggered, Acknowledged, Investigating, Fixing, Monitoring, Resolved) map from the seven stored statuses. The API, MCP and CLI return statusLabel; status filters take stored values.
  • Investigations are evidence-first: they read stored native evidence (monitors, runs, alerts) before any live connector, and live sources are opt-in per run. Token use from investigations, triage, briefs and Copilot counts against the daily org cap (see Environment configuration).
  • Connector and private-agent credentials stay encrypted/server-side and are never placed in prompts, browser payloads, logs, or model-visible errors.

Tool and provider behavior

  • Tool schemas validate inputs and bound output size. Treat provider/tool output as untrusted evidence.
  • Jira, Confluence, and Notion are read-only knowledge connectors (Notion is direct only; Jira and Confluence also run on a Private Agent in worker/src/private-agent/private-agent-knowledge.ts). Page text is redacted, size-capped, fenced as untrusted document evidence, and fetched for top results only. Slack, Teams, Webhook, Kafka, and RabbitMQ are not evidence sources; unsupported records show as Unsupported and must be disableable.
  • Cancellation and timeout propagate through model streams, tools, polling, persistence, and UI state.
  • MCP connectors are Private Agent only, Streamable HTTP only, never stdio. Only tools allowlisted by a user with sre_connector:configure can run (description and schema are pinned at allowlist time); arguments are validated against the saved schema in app/src/sre/tools/mcp-tools.ts and again in worker/src/private-agent/mcp-client.ts. Output is text/JSON only, redacted, and untrusted; server read-only hints never authorize a call. The agent must run the same release as the app.
  • Private Agents report capabilities.sreConnectorTypes (worker SUPPORTED_SRE_CONNECTOR_TYPES), stored in private_agents.metadata; the job router refuses types an agent did not report (private_agent_outdated), and agents that report nothing are treated as supporting only the 10 original types. Keep the worker list equal to PRIVATE_AGENT_CONNECTOR_TYPES (a spec enforces it). Workers fail unknown-type jobs with unsupported_connector_type instead of re-leasing them.
  • A Private Agent job shared through an idempotency key keeps a private_agent_jobs.waiters count (migration 0060): the inserting caller starts at 1, reusers join, and a caller whose bounded wait ends leaves; only the last one out cancels a queued job (app/src/lib/sre/private-agent-job-cancel.ts). A caller that crashes mid-wait never leaves, so the job ages out of the queue. Finished jobs are purged after PRIVATE_AGENT_JOB_RETENTION_DAYS (default 30, newest per connector kept) by telemetry-retention.ts.
  • Retry only transient/idempotent operations; prevent duplicate messages, actions, tool records, usage events, and charges.
  • Observability providers (Jaeger, New Relic, Splunk, Google Cloud, Azure Monitor) live in app/src/lib/sre/connectors/providers/ and are copied to worker/src/private-agent/providers/; edit the app files and run npm run sync:sre-providers in app/. Queries are read-only (reject Splunk write commands, non-SELECT NRQL, KQL control commands) and always time-windowed. New Relic, Google Cloud, and Azure call fixed vendor hosts.
  • Provider fallback (including deterministic evidence-brief fallbacks) must preserve tenant scope, safety policy, configured model selection, usage accounting, and error semantics.
  • Investigation results are structured and validated in app/src/sre/lib/investigation-output.ts (conclusion, confidence, ranked hypotheses; Inconclusive is valid; a hypothesis is Validated only with evidence from this incident). Similar past incidents (5 resolved, 180 days, same project) and user feedback notes are hints, never evidence. Post-mortem drafts are rate limited (5 per user per incident per hour) and fall back to a deterministic template. Their timeline comes from postmortemTimeline in app/src/lib/sre/postmortem.ts, which names events with describeTimelineEvent (app/src/lib/sre/incident-timeline-format.ts, shared with the Activity feed) and writes dates as fixed UTC; unknown sections stay empty, never placeholders. Changes under app/src/sre/** run the sre-eval CI gate.
  • Redact secrets and sensitive telemetry before model invocation and persisted transcripts.

Resilience

  • External calls have explicit connection/operation timeouts and abort handling.
  • Use exponential backoff with jitter for safe transient retries; no circuit-breaker abstraction exists yet, so introduce one deliberately if sustained dependency failure needs containment.
  • Fallbacks degrade explicitly and never bypass authentication, authorization, capacity, billing, or audit controls.
  • Redis connections intentionally survive failover; QueueEvents blocking connections remain isolated from normal command timeout behavior.
  • Distributed schedulers/locks use ownership tokens and bounded leases so one instance cannot release another’s lock.

Memory and performance

  • Bound transcripts, evidence, logs, tool output, arrays, caches, queue payloads, and export/result sizes.
  • Stream large responses/artifacts and paginate growing datasets.
  • Clean up timers, listeners, subscriptions, AbortControllers, child processes, temporary files, and browser contexts on every terminal path.
  • Optimize database access from measured query plans/cardinality; preserve exact React Query cache-key parity between prefetch and hooks.
  • Do not trade tenant/security checks for caching or performance.

Environment configuration

  • Environment access is mixed: the app reads process.env directly, and the worker combines @nestjs/config (ConfigModule.forRoot() plus injected ConfigService) with direct process.env reads in several services. No central schema helper exists today.
  • Keep app, worker, deployment manifests, examples, docs, and health/config diagnostics synchronized. Self-hosting assets live in deploy/docker/ (compose variants, install.sh, init-secrets.sh, setup-k3s.sh, setup-gvisor.sh) and are tested by compose.spec.py and install.spec.py; the user guide is docs/content/docs/app/deployment/.
  • AI SRE gates live in app/src/sre/lib/feature-gates.ts: SRE_ENABLED defaults on, SRE_AUTOMATION_ENABLED and SRE_AGENT_SANDBOX_ENABLED default off. The worker reads its own copy of SRE_ENABLED only (worker/src/monitor/services/sre-alert-triage-queue.service.ts); SRE_AUTOMATION_ENABLED is app-only. Keep the two SRE_ENABLED checks in sync.
  • AI SRE HTTP routes authorize through app/src/app/api/sre/_auth.ts; dynamic [id] routes reject malformed UUIDs before database access.
  • Daily AI token cap per organization: SRE_DAILY_TOKEN_LIMIT_PER_ORG (default 2000000, 0 disables, UTC day, needs Redis, app only) in app/src/lib/sre/sre-daily-token-limit.ts. Investigations, triage, briefs, and Copilot share it; keep it in app/.env.example, the self-hosted docs table, and the AI SRE setup page. Investigation runs stuck for 5 minutes are marked failed by stale-run-recovery.ts.
  • Validate required production values and fail closed when security-critical settings are missing.
  • Never expose secrets through health endpoints, startup logs, client-prefixed variables, build output, or configuration APIs.
  • Use distinct credentials/key namespaces for auth, encryption, object storage, AI providers, billing, email, and infrastructure.
  • BullMQ Redis requires noeviction; production execution requires gVisor/isolation settings.

Release management

flowchart LR
  CODE[Reviewed commit] --> CI[Required CI]
  CI --> TAG[Signed annotated release tag]
  TAG --> IMAGE[Immutable release images]
  IMAGE --> MIGRATE[Compatible migration]
  MIGRATE --> DEPLOY[Bounded deployment]
  DEPLOY --> ACCEPT[Exact-SHA acceptance]
  ACCEPT --> CLEAN[Fixture cleanup]
  • Use semantic project versions and immutable image tags/digests; never deploy main, another branch name, or latest to production.
  • Release tags must be annotated and signed with a key listed in .github/release-allowed-signers; unsigned, lightweight, or unlisted-key tags fail the publish gate. Agents never create, move, or delete tags or dispatch publish workflows (docs/releasing.md).
  • Review database compatibility before rollout and order app/worker/migration changes for version skew.
  • Migration 0043_sre_incident_management.sql (incident assignee, acknowledgement, commander, resolution summary, post-mortem columns) is forward-only and must be applied before the app that reads it is deployed.
  • CI must not use pull_request_target with untrusted contributor code or expose secrets to fork builds.
  • A successful build/deploy workflow is not release acceptance. Verify deployed SHA, health, migrations, browser flows, RBAC/flags, workers/control plane, provider/billing gates, and cleanup.
  • Keep changelog, package versions, deployment defaults, docs, and release evidence synchronized.

Verify

  • Cover tenant/RBAC denial before model/provider/tool execution.
  • Cover cancellation, provider failure, transcript identity, retries, audit, usage/billing exactly-once behavior, and connector/private-agent boundaries.
  • Use disposable non-production fixtures. Never inject failures into production, use customer credentials/data, exhaust limits, or spend quota only to repeat a proven scenario.
writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deploymentStars: 321kUpdated Claude CodeCodexVerifiedInstall
xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved.Stars: 197kUpdated Claude CodeCodexVerifiedInstall
shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy.Stars: 119kUpdated Claude CodeCodexVerifiedInstall
childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release.Stars: 118kUpdated Claude CodeCodexVerifiedInstall