Skip to content

ZenUml/conf-app / .claude/skills/error-radar

error-radar

Verified

Daily detector for new and unusual conf-app error signatures in Mixpanel, posted to the

Error radar

scripts/radar.py looks at one UTC day (default: yesterday) and compares every error signature with the 30 days before it. It is deterministic and stateless: the baseline is recomputed from Mixpanel on every run. The only state is the last day posted (--state), which the Action uses to choose the days to post.

Two jobs use it every day:

JobWhen (Sydney)Does
GitHub Actions error-radar.ymlscheduled 11:17 AEDT, often starts 2-3 h latePosts one message per UTC day not yet posted to #github-notifications by webhook. Logs only a tenant-free summary.
Claude scheduled task "Error radar triage"~15:08Follows Daily triage below: replies in the thread of that post with a verdict per alert. Posts the radar message itself if the Action's post is missing.

GitHub starts scheduled runs late (2-3 h in 2026-10), and a run that started before midnight UTC used to report the day before yesterday: 10-25 was never posted and 10-23 was posted four times. So the Action keeps the last day posted in the Actions cache (radar-state/last-day, a date and nothing else) and posts every day after it up to yesterday, at most 7. With no cache it posts yesterday alone. A manual run with day posts that day and never moves the saved date backwards. The triage runs late enough to come after a delayed post.

Signature

A signature is event | error_name | http_status | confluence_error_class | message (scripts/signature.js, the same code runs inside the JQL query and under node --test). The message is error_message, falling back to event_label for legacy trackEvent events, with ids, numbers, uuids, hashes, URLs, Java package prefixes and HTML bodies replaced by placeholders. A 1xx-5xx code right after status, code or HTTP is kept ({"status":403}), because it is the failure kind, not an id.

Events in scope: names ending _failed, _error, _missing, _timeout, _exception, orphan_observed, plus legacy events sent with category error. Excluded: other products (product set), internal sites, category warning alone (it marks probes such as duplication_detect), and create_attachment<type> (a second copy of every attachment_upload_failed).

Counts are weighted by 1 / sample_rate, so a 10%-sampled event reads at its true volume. Distinct users are not weighted.

Alert kinds

KindFires when (constants at the top of radar.py)
newnot seen in the previous 30 days; ≥3 events received, ≥1 real tenant, and ≥5 events or ≥2 tenants
spike> 2.5× the median of the comparable days (previous 20 weekdays, or 8 weekend days) and ≥ 50 above it
concentrated≥ 50 events, one tenant > 50% of them, and that share up ≥ 25 points on the 30-day baseline
spread≥ 8 tenants, ≥ 25 events, and > 2.5× the usual tenant count

Spike, concentrated and spread also need ≥10 events actually received, so a handful of sampled arrivals cannot alert on their ×10 extrapolation. One alert per signature lists every kind that fired. A kind that already fired in the previous 3 days is suppressed unless the count has at least doubled since, so a standing problem alerts when it starts and each time it doubles.

Backtest 2026-09-20..10-19: 1.7 alerts/day, 0 on 13 of 30 days, at most 9 (2026-10-12, the day the October sampling release shipped). Re-run over 2026-09-20..10-27 after keeping status codes and adding the 25-event spread floor: 88 alerts (91 before), 0 on 14 of 38 days, at most 8. The floor removed four spread alerts of 10-13 events; keeping status codes split one report_macro_metrics alert into its 400 and 500 forms. It caught the single-tenant HTML-instead-of-JSON upload ramp on 10-09, a second tenant's missing-content burst on 10-10, the missing-content spread on 10-12, and the first tenant's next jump on 10-19. Re-run the backtest after changing a constant:

S=.claude/skills/error-radar/scripts
python3 $S/radar.py --backtest 2026-09-20 2026-10-19 --save-data /tmp/radar.json   # queries Mixpanel once
python3 $S/radar.py --input /tmp/radar.json --backtest 2026-09-20 2026-10-19        # offline re-runs
python3 -m unittest discover -s $S -p 'test_*.py' && node --test $S/signature_test.js

Running it

python3 .claude/skills/error-radar/scripts/radar.py                     # yesterday, Slack text to stdout
python3 .claude/skills/error-radar/scripts/radar.py --day 2026-10-19 --json
python3 .claude/skills/error-radar/scripts/radar.py --post              # needs SLACK_ERROR_RADAR_WEBHOOK
python3 .claude/skills/error-radar/scripts/radar.py --state /tmp/last-day   # days after the saved one (--post saves)

Auth: MIXPANEL_SA_USER + MIXPANEL_SA_SECRET (service account, sent with project_id), else MIXPANEL_API_SECRET, else .env.mixpanel. Two JQL queries per run, about 15 s.

The GitHub Action needs three values: MIXPANEL_SA_USER, MIXPANEL_SA_SECRET, SLACK_ERROR_RADAR_WEBHOOK (an incoming webhook for #github-notifications). MIXPANEL_SA_USER may be a repository secret or variable. MIXPANEL_SA_SECRET and SLACK_ERROR_RADAR_WEBHOOK must be secrets: GitHub prints variables verbatim in the step env block and this repo's logs are public, so a webhook saved as a variable is published (it was, once, on 2026-10-24). The "Check configuration" step fails first, naming anything missing and refusing a sensitive value found among the variables; it only ever sees true/false for those two.

Privacy

The output names real tenants. It goes to #github-notifications (a public channel inside the team's Slack workspace) and nowhere else: not into the repo, a GitHub issue or PR, a workflow log, or an Actions artifact. The repository is public, and so are its logs. --quiet is the only mode used in CI logs.

Daily triage (the Claude scheduled task follows this)

  1. Run python3 .claude/skills/error-radar/scripts/radar.py --json for yesterday (UTC). No alerts → stop; post nothing.
  2. Find today's "Error radar — " message in #github-notifications (channel id C0687BD6Y9W). If it is missing (the Action failed or has no webhook), post the text from radar.py (without --json) first, and say in its thread that the Action's post was missing.
  3. For each alert, up to 6, most severe first:
    • Where it is emitted: grep the event name in src/ and functions/. Name the file:line and the branch that produces this signature.
    • When it started and on which build: Mixpanel MCP (Get-Business-Context first; filter is_internal_client_domain equals "false") — the signature's events by day for 14 days, and by app_version. Line the onset up with release tags (git tag --sort=-creatordate | head -20) and git log of the emitting file.
    • Known? Search open PRs and issues on ZenUml/conf-app for the event name or message.
    • Verdict, one of: regression from release X · tenant-specific (one site, few users) · known issue (link) · tracking artifact (sampling/rename/new event) · unclear. Then the one next step.
  4. Reply once in the thread: one short paragraph per alert, verdict first. State evidence (numbers, file:line, tag) and say plainly when something was not checked. Before stating a sample rate, an event's meaning or a rename, read src/utils/analytics/eventSampling.ts and the mixpanel skill. Both have tripped this analysis before.
  5. Read-only: never open PRs or issues, push, or change Mixpanel. Tenant names stay in Slack.
slack-gif-creatoranthropics/skillsKnowledge and utilities for creating animated GIFs optimized for Slack. Provides constraints, validation tools, and animation concepts. Use when users request animated GIFs for Slack like "make me a GIF of X doing Y for Slack."Stars: 196kUpdated Claude CodeCodexVerifiedInstall
ponytailDietrichGebert/ponytailLazy senior dev mode for any coding task: the smallest change that fully solves it, and a reply a busy human understands in one read.Stars: 193kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailHarvest every shortcut: comment into one debt ledger, so deferrals get tracked instead of forgotten. One-shot report.Stars: 193kUpdated Claude CodeCodexVerifiedInstall
ponytail-debtDietrichGebert/ponytailList every `shortcut:` comment (and older `ponytail:` ones) as a debt ledger. One-shot report, changes nothing. Use for "ponytail debt", "what did ponytail defer", "list the shortcuts", /ponytail-debt.Stars: 193kUpdated Claude CodeCodexVerifiedInstall