supercheck-io/supercheck / .agents/skills/monitoring-alerts
supercheck-monitoring-alerts
VerifiedWork on Supercheck HTTP request, website/SSL, ping host, port check, or Playwright synthetic monitors, wrong monitor status or uptime (false down, missing regions), check cycles, regional scheduling and aggregation, alert thresholds, state and history, incidents, or email, Slack, webhook, Telegram, Discord, and Microsoft Teams notifications.
Supercheck monitoring and alerts
flowchart LR
SCHED[Distributed monitor scheduler] --> Q[Regional queues]
Q --> CHECK[HTTP, website, ping, port, or synthetic execution]
CHECK --> AGG[Regional aggregation]
AGG --> STATE{Alert state machine}
STATE --> HIST[Alert history]
STATE --> NOTIFY[Notification providers]
Start here
| File | Why |
|---|---|
worker/src/monitor/monitor.service.ts | Check execution per monitor type; SSRF-safe probes via requestPinnedMonitorTarget (common/utils/pinned-monitor-request.ts); cycle arrival recording |
worker/src/monitor/monitor-evaluator.ts | Threshold, transition, and alert logic (evaluateAndSendAlert) |
app/src/lib/monitor-cycles.ts, worker/src/monitor-control/ (cycle-policy.ts, monitor-cycle.service.ts, monitor-cycle.processor.ts) | Durable check cycles: regional quorum, closing, aborting |
app/src/lib/monitor-aggregation-service.ts | Regional rollup and uptime |
worker/src/notification/notification.service.ts, app/src/db/schema/notification.ts | Providers, delivery, alert history |
Monitor definition and execution
- Monitor definitions are tenant-owned configuration; executions and regional results are immutable observations.
- Preserve target, type, request/check config (method, headers/body, auth, timeouts, expected status/keywords, SSL checks), synthetic test reference, frequency, location selection, alert thresholds/flags, enabled state, and notification relationships across schema, APIs, UI, scheduler, queue, and worker. Monitor
retryStrategyis deprecated and ignored. ALLOW_INTERNAL_TARGETS=truealone lets monitors reach intranet hosts (webhook channels additionally needSELF_HOSTED, viaallowInternalWebhookTargets()); metadata and link-local addresses are always blocked for monitors, in the worker (resolvePublicAddresses) and the app (monitor-target-policy.ts).- Check classification (worker
monitor.service.ts,common/utils/network-error.ts): timeouts are detected byisTimeoutError(codeETIMEDOUTincl. insideAggregateError, or message, case-insensitive). DNS failures andMonitorRedirectError(limit, invalid or non-http redirect) aredown; validation and system faults stayerror. Ping countsECONNREFUSEDas up (host answered) and times only the answering attempt. TCPexpectClosedpasses on refusal, and on timeout only when the ping probe finds the host alive (else down; the probe starts with the connect, tries both ports in parallel on the same vetted lookup and shares the configured timeout). Ping counts any ECONNREFUSED in an AggregateError as alive. An oversized 3xx body iserror(redirect unverified). UDP connect failures surface aserror. UDP probes on a connected socket: ICMP refusal = closed, reply = open, silence = open|filtered (up; failsexpectClosed). HTTP pins every vetted address withautoSelectFamily;responseSizeis bytes; a body over the size cap isup/downby status (errorwith no status or a keyword check).isExpectedStatusis strict and case-insensitive (/^[1-5]xx$/,NNN,NNN-NNNascending, 100-599); malformed entries never match and the API rejects them. - Monitor destinations are untrusted. Validate URL shape at configuration time; enforce SSRF-safe DNS-pinned resolution at execution time, re-applied on every redirect and against DNS changes.
- Scheduling is safe across multiple app replicas via BullMQ repeatable monitor jobs keyed by a stable jobId, with a Redis NX lock guarding startup re-scheduling and orphan cleanup (
app/src/lib/monitor-scheduler.ts). Cleanup removes only repeatables whose monitor is deleted, disabled, paused or has a non-nullscheduledJobIdthat differs from its id; it keeps one repeatable per enabled, non-paused monitor at its current cadence (scheduledJobIdnull or matching) and removes extras or stale-cadence ones left by a frequency edit that raced a sync. Init re-reads each monitor row before scheduling it and renews the 120 s lock every 30 s. A periodic repair sweep (repairMonitorSchedules, every 5 min, same Redis lock as init, 100 monitors per sweep) reschedules enabled, non-paused monitors whosescheduledJobIdis null or whose repeatable is missing, so a failed reconcile heals without a restart. A resume racing this is safe: it commits its status first, and init reads active monitors after its removals and re-schedules them. Restarts or overlapping ticks must not duplicate dispatch. PATCH /api/monitors/{id}mergesconfigandalertConfigwithmergePatch(RFC 7396: arrays replace,nulldeletes) inapp/src/lib/monitor-write.ts;PUTreplacesconfig.PATCH .../statustakes onlyup,down,paused,pending, keepsenabledin step withpausedand reconciles the schedule under a per-monitor lock. Turning a paused monitor back on goes throughwithResumeWithinPlan(plan-limit lock, 402/403/503), and the worker never un-pauses a monitor from an in-flight check. SSL expiry alerts are suppressed in maintenance like other alerts.- Website SSL: the worker reads the certificate when due (
shouldPerformSslCheck:sslCheckFrequencyHours, default 24; 6 h within 2x the warning window; 1 h inside it) and saves it withsslLastCheckedAtonmonitors.config.sslCertificatethrough a jsonb merge (recordSslCheck). Both are worker-owned (WORKER_OWNED_CONFIG_KEYS): clients can neither set them on create nor wipe them on PUT. The stored copy carrieshost; a missing certificate, a host that differs from the target, orvalid: falsemakes the check due on every run. Only a certificate verdict (expired, not yet valid, untrusted, hostname mismatch) fails the website check; a probe that read no certificate setssslWarning, is not stamped, and retries next run.checkSslExpirationAlertalerts fromsslCertificate, never fromsslWarningtext; a probe that reads no certificate keeps worker-ownedsslProbeFailure{since, host}(new host restarts the clock; cleared byrecordSslCheckor when SSL Check is off) and alerts once it is 24 h old, only while the site is up. The detail page badge reads the stored certificate, falling back to recent results. - Monitor write paths (create,
PUT/PATCH /api/monitors/{id}, collectionPUT) share helpers inapp/src/lib/monitor-write.ts:frequencyMinutesError(whole minutes 1-1440),timeoutFrequencyError(previous, next)(config.timeoutSecondsmay not exceedfrequencyMinutes * 60; rejects only a save that widens an existing excess, so a legacy invalid row keeps saving),sanitizeMonitorFields(name, target,customMessage; credentials inconfig.authare never rewritten) andthrownStatusResponse. EveryalertConfigwrite (includingnulland a disabled config) validates provider ids against the project and replacesmonitor_notification_settingsrows in the same transaction as the monitor row (syncMonitorProviderRows);POST/DELETE .../notificationsmirror the rows back intoalertConfig.notificationProviders(mirrorProviderRowsToAlertConfig). The collectionPUTonly touchesalertConfigwhen the body has it. After a write that changes the check, routes callreconcileMonitorSchedule(app/src/lib/monitor-schedule-reconcile.ts, per-monitor lock) and answer 503 with a retry message when Redis fails; the row stays saved. - Location selection maps deterministically to worker queues and rejects unknown/unavailable locations deliberately.
Aggregation
- A regional result and aggregate monitor state are separate. Never infer global health from one region.
- Define how missing, late, timed-out, and conflicting regional results affect aggregate status; preserve that rule across dashboards, APIs, and alert evaluation. Cycle status is
open,closed, oraborted(app/src/db/schema/monitor-cycles.ts), and the work is split:decideMonitorCycleinworker/src/monitor-control/cycle-policy.tscomputes the regional quorum decision only.MonitorCycleService.closeCycle(monitor-cycle.service.ts) applies it: monitor status, synthetic missing-region results, and alert evaluation.monitor-cycle.service.tsaborts cycles for paused or disabled monitors. An aborted cycle writes no synthetic failures, so it does not count against uptime.monitor-cycle.processor.tsbacks off and retries closure when a dependency check is temporarily unavailable, instead of failing the cycle.recordCycleArrivalinworker/src/monitor/monitor.service.tstreats a cycle owned by another monitor as absent, and a result for a closed or aborted cycle never re-evaluates it.
- Multi-region status is judged over the regions enqueued for that check (
expectedLocationsin the job data,LocationService.calculateAggregatedStatus's third argument, and the persisted cycle'sexpected_locations), not every configured region. The app skips regions with no live worker heartbeat; those never count as down. The legacy aggregation path names them in the alert reason ("not run: …"); the cycle-closure path (MISSING_CYCLE_CLOSURE_ENABLED) judges only the expected set and does not list them. An enqueued region with no result is never counted as up: the legacy grouped path waits for every expected result before judging status (a job that fails for good writes an error result), while the cycle-closure path closes at its quorum or deadline and counts missing regions as down. The app persists the cycle withpublish_pending = trueand clears it afteraddBulk; a cycle still pending at its deadline with no results and no arrival is aborted, never closed as missing-region downtime (older apps leave it false and keep the old behaviour). The mark is retried 3 times and an error naming the cycle is logged if it still fails; any result or arrival row makes a stale flag irrelevant. Residual case: mark failed on all attempts and every region silent, so a genuine all-down cycle is aborted without an alert (the error log is the only signal). AlertlocationResultscome from the frozen closure (synthetic row for a missing region, rows up to the observed time), not a fresh newest-row query. Jobs withoutexpectedLocations(rolling deploy) fall back to the configured regions. The Redismonitor:aggr:completed:<group>marker lives 2 hours (AGGREGATION_COMPLETED_TTL_SECONDS); the persistedaggregationEvaluationin the DB is the durable duplicate guard. - Use consistent units and boundaries for latency, uptime, thresholds, and time windows.
- Uptime counts CHECKS, not regional rows: rows sharing an
execution_group_idare one cycle, up or down by the monitor's region rule (isCycleUpinmonitor-aggregation-service.ts, same math as the worker'sdecideMonitorCycle; the persistedmonitor_cyclessnapshot when present, else the monitor's currentlocationConfig; a missing region is down). Rows without a group count once each. A cycle still missing a region is left out until its persisteddeadline_at, or 60 minutes after its first row when there is no cycle row (not shown as down while in flight), and a stale aggregate is deleted only when raw rows exist from before its period (raw retention is shorter than aggregate retention). The combined (locationnull) aggregate row and the/statsroute use this; per-location rows and/location-statsstay per region. A cycle belongs to the period of its earliest row. Aggregate rows written before this rule keep regional counts until recomputed (hourly 24h, daily 7 days;runDailyAggregation(days)rebuilds further back within raw retention). Response times are up checks only (> 0 ms) everywhere. 24h p95 is rawpercentile_cont; 30d p95 merges the daily rows' log-bucketresponse_histogram(within ~2.5%). Aggregation visits every monitor with raw rows in the lookback, including paused ones, and/statsreads raw rows for every period without an aggregate row. Unknownlocationfilters answer 400 on/statsand/results. - Store enough immutable evidence to explain why an alert transitioned without retaining secrets or unrestricted response bodies.
Alert state and history
- Model transitions explicitly using the current states and consecutive-failure/recovery policy.
- Persist state transitions with the result that caused them;
alert_historyrecords each delivery attempt (provider, status, outcome). - Deduplicate repeated failing observations while preserving meaningful reminders/escalations configured by the product.
- Recovery closes the active condition once (one recovery alert per outage). Monitor failure alerts are capped at 3 per sequence: the first at the threshold, then reminders when
N * (2^k - 1)failures have passed it (k = alerts already sent, N =max(5, 2 x threshold)), so the gap doubles. Each alert has a window of N checks starting atN * (2^k - 1)after the threshold, but never earlier thanlastAlertCount - threshold + N * 2^(k-1)(k = windows used): a late alert keeps the full scheduled gap. The counter andconfig.aggregatedAlertState.lastAlertCountare persisted after a delivered send (single-location too, viajsonb_set/jsonb_build_object), so an undelivered alert stays due and retries on each check until its window closes, and an unsent alert may use later windows. A delivery consumes its window andalertsSentForFailurestores the windows used (at most 3 alerts).persistAggregatedAlertStatewrites onlyconfig.aggregatedAlertStateviajsonb_set.alert_historyerror and message text has URL credentials and query values redacted. There is no mute feature:monitors.muted_untilis gone from the Drizzle schemas, the API and OpenAPI; the column was dropped by migration 0070 (workers of the previous release still select it, so their checks can fail until they are replaced). - Stale scheduled checks: the scheduler sets
scheduledCheck: truein job data;MonitorDynamicWorkerService.isStaleskips a first-start job older thanmax(3 x scheduling interval, 3 min)(byjob.timestamp; synthetic interval is at least 5 min) only when its cycle is already closed or a real (non-synthetic)monitor_resultsrow checked afterjob.timestampexists for the same monitor and region (any region for a legacy all-region job); otherwise, or if that lookup fails, it runs late, so a backlog does not by itself freeze status. A cycle job whose cycle already closed is skipped at any age (inside the bound quietly: debug log, not counted as a stale skip); this cannot undo a down result or alert the cycle already recorded at its deadline. A skip records no result, downtime or alert. Known limits:checkedAtis worker time andjob.timestampapp time, so a worker clock far ahead can cause a false skip; concurrent stale jobs for one monitor can each run once before the first result lands. The region is dropped from the group's expected set like a no-live-worker region, so the others are still judged:MonitorService.dropStaleRegionshrinks the cycle'sexpected_locationsand writes an arrival hint, or (legacy) adds it to Redismonitor:aggr:dropped:<group>and re-evaluates now. Manual runs (no flag), retries and deferrals (attemptsStarted > 1) never skip; a delayed drop is resumed by the_regionDroppedflag. The worker has no metrics endpoint, so skips are logged individually (info) and as one aggregated warning per minute; a burst means clock skew or queue saturation. - Job failure/timeout alerts are attributed to a run:
alert_history.run_idplus the partial unique index(job_id, run_id, provider)(migration 0055). The worker takes a per-job advisory lock, inserts apendingclaim, then marks itsentorfailed; a failed provider is retried for the same run, and reminders stay capped at 3. - Monitor alerts are claimed per provider before sending (migrations 0068 columns, 0069 index):
alert_historyrows keyed(monitor_id, outage_key, type, alert_ordinal, provider)under a partial unique index (outage_keyandalert_ordinalset; old and unkeyed rows stay outside it).outage_keyidentifies one failure sequence and the recovery that ends it, minted where the counters are committed: multi-location inaggregatedAlertState.outageKey(getNextAggregatedAlertState, inside the cycle-close/prepareGroupEvaluationtransaction; dropped for uncounted cycles), single-location in the result rowdetails.outageKey(nextOutageKey, carried bysaveMonitorResultToDband synthetic cycle rows).alert_ordinalis the reminder window index (0-2) for failures and 0 for the recovery.MonitorAlertService.claimProvidersupsertspendingrows in one statement (ids sorted), taking over afailedrow withclaim_attemptsbelow 5 or apendingone older than 2 minutes (sent_atis the claim clock until the row is finished), and only the claimed providers are sent to.deliveredmeans no provider is still owed the alert, so one failed channel keeps the alert due and is retried alone; a lost counter write cannot resend because the next in-window check finds every rowsent(alreadySent) and only catches the counters up. A claim that cannot be made falls back to sending unclaimed. When every provider has used its attempts (gaveUp) the evaluator treats the window as terminal: counters advance like a delivery but it is not reported as sent. Each provider's row is finished as soon as its own send settles (sendNotificationToMultipleProvidersonSettled), with up to 5 in-process write attempts; an outcome that still cannot be written is retried before this process's next claim, and for a failure alert the incident is opened from the known delivery (enqueueSourceFailure), since a row leftpendingis skipped by triage. A stalependingtakeover cannot tell a crash from a lost write, so a database error while recording can still repeat an alert after 2 minutes (documented as a rare duplicate). Without an outage key (state or rows written by an older worker, incident-only monitors) the alert is sent and recorded unclaimed. Rollout: migrations 0068, 0069 and 0070 run on app start (0070 dropsmonitors.muted_until, so roll workers promptly after the app) and must run before the new worker (0069 builds the claim index CONCURRENTLY outside a transaction, soalert_historystays writable; until it exists the claim fails and the alert is sent unclaimed); workers of the previous release write rows without keys and dropoutageKeywhen they rewrite the alert state, so mid-rollout an outage can alert twice. A recovery held by another evaluator's live claim is retried (deliveryFailed) so a dead holder cannot lose it. Known limits: a provider that keeps failing past its window gets the next window's alert while the others get a repeat, and two simultaneous first checks of one single-location monitor can mint two keys. Retention (AlertHistoryCleanupStrategy) keeps keyed rows of monitors currentlydownorerror. - SSL renewed notice: delivering an
ssl_expiringwarning storesconfig.sslAlertState{id, host, validTo?, warnedAt}(worker-owned; kept for the same host). A later certificate read that is valid withdaysRemainingabove the warning window, or a latervalidTo, sends onessl_renewednotice claimed undersslAlertState.id, then clears the state; none without a stored warning, held back in maintenance, retried on the next read if undelivered, and given up (state cleared) once every channel used its 5 attempts. A warning that was an unreadable certificate has novalidTo, so its notice reads "readable again". The state is also dropped when SSL alerts, alerts or SSL Check are turned off: at write time bydropStaleSslAlertStateinupdateMonitorHandler(deletion is the only client-side change to this worker-owned key), and by the worker when it next evaluates; with SSL Check offcheckSslExpirationAlertreturns before any SSL alert, so an in-flight read cannot warn after checking was disabled. A new outage never reuses the key of a recovered one:getNextAggregatedAlertStatereuses a key only whileconsecutiveFailureCount > 0.ssl_renewednever goes to incident triage (worker skips it,background-alert-triageandisActionableAlertForIncidentignore it) and webhooks getresolvewith thessl:dedup key. - Alerts open incidents: a sent or failed (not pending)
alert_historyrow is queued forapp/src/sre/lib/background-alert-triage.ts, which opens or joins one incident per monitor or job.alertConfigcarriesincidentEnabled(default on),incidentSeverity(autoorsev1-sev4) andautoResolveIncident(default on); a recovery resolves the open auto-opened incident when it is on, even if recovery alerts are off. OnlySRE_ENABLEDis needed;SRE_AUTOMATION_ENABLEDgates AI triage. A monitor with alerts off butincidentEnabled: trueis incident-only (isIncidentOnly,tracksAlertStatein the evaluator;incidentsEnabledinapp/src/lib/incident-settings.ts), andchannelsOptionalthere lets an enabled alert have no notification channel when it still opens incidents.sre_incidents.alert_countandlast_alert_at(migration 0063) count the opening alert plus repeats. Keep these fields in sync across the app schema, worker, CLI schema, MCP tools and OpenAPI. - Alert acknowledgement is not implemented; incident acknowledgement is (see the SRE operations skill). Manual status changes require permission, tenant scope, audit events, and race-safe updates.
Notification providers
- Provider definitions (email, Slack, webhook, Telegram, Discord, Teams; webhook presets for PagerDuty, Jira Service Management, Splunk On-Call, Better Stack, and incident.io) are scoped to their owning tenant and secrets remain encrypted/redacted.
- Render messages from bounded, escaped data. Do not leak headers, variables, tokens, raw response bodies, or internal errors.
- Delivery uses bounded timeouts with provider-appropriate retry behavior (Slack, Telegram, Discord, and Teams retry; webhook and email are single-attempt today).
- Notification text is sanitized centrally: monitor targets and URLs inside messages have userinfo and sensitive query values redacted, Slack text escapes
&,<,>, webhook delivery errors go throughsafeSendError, an email with no deliverable recipient is a failure, and a synthetic monitor whose variables or secrets cannot be resolved reportserrorwithout running the test.sanitizeResponseBodytruncates before redacting and uses bounded regexes. - Webhook destinations use SSRF protection (allowlisted hosts, DNS pinning). Notification delivery is strictly outbound; no inbound callback route exists.
- Record delivery outcome and safe provider diagnostics without blocking the monitor’s terminal persistence indefinitely.
Verify
- Unit-test threshold boundaries, state transitions, deduplication, recovery, and partial/missing region behavior.
- Integration-test scheduler ownership, queue routing, worker output, and provider retry/idempotency.
- Test cross-tenant provider/monitor access and secret redaction.
- Use disposable provider endpoints and clean them up; live delivery acceptance remains separate from automated tests.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 322k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 198k | Updated | Claude CodeCodex | Verified | Install |