Skip to content

jackson2w/claude-code-skills / forced-outage-drill

forced-outage-drill

Flagged

Simulate a dependency outage on a live host (upstream DNS, an API endpoint, a backup target, a peer) to prove a watcher, failover or alert actually works, without paging anyone and without the drill passing for the wrong reason. Covers proving the block bites (nft counters plus a control) before trusting any result, why container egress needs a forward-hook chain, cache and stale-answer layers that hide the outage, a systemd-run dead-man revert, testing a new watcher against a throwaway instance on a spare port with stubbed alerting, and the drill sequence: baseline, failure surfaces, failover, recovery, cleanup. Trigger phrases include "failover drill", "simulate an outage", "force a failover", "test the watcher", "blackhole the upstream", "nft drop for a test", "dead-man revert", "does the alert actually fire", "drill without paging", "stub telegram", "prove the failover works".

Forced-outage drills that mean something

A drill is only evidence if the outage really happened. The usual failure isn't a crash; it's a green result because the simulated fault never reached the thing under test. On 2026-10-23 a single DNS session hit this three times. All three looked like "the system copes fine":

  • an nft rule keyed on meta skuid 0 dropped nothing the test needed, and lookups kept answering;
  • unbound-control flush <name> didn't empty the cache: with serve-expired it only marks entries expired, and they keep being served;
  • the drill's container chain was named fwd, a reserved word in nft, so it was never created; the output-only rule left container egress untouched.

1. Prove the block bites before reading any result

  • Put a counter on every drop rule and print the counters after the fault window. Zero packets means no outage happened, whatever the probes said.
  • Pair it with a control on the same host that must still work (another port or address on the same path, e.g. :443 to the same IP while :853 is blocked). A dead host and a working block look identical without one.
  • Quick reachability probe, no extra tools: timeout 4 bash -c '</dev/tcp/9.9.9.9/853' && echo REACHED || echo BLOCKED.
  • Expect the failure to surface first. Before the watcher reacts, an uncached request must actually fail. If it succeeds, something (cache, stale answers, a fallback) is masking the fault, and the drill is testing that layer, not the watcher.

2. Block at the layer the traffic actually takes

  • Host processes: an output-hook chain.
  • Docker containers on a bridge: egress traverses the host's forward path, so an output drop alone does nothing. Add a forward-hook chain too:
    nft add table inet drill
    nft add chain inet drill out '{ type filter hook output priority 0; }'
    nft add chain inet drill fw  '{ type filter hook forward priority 0; }'
    nft add rule inet drill out ip daddr '{ 9.9.9.9, 149.112.112.112 }' tcp dport 853 counter drop
    nft add rule inet drill fw  ip daddr '{ 9.9.9.9, 149.112.112.112 }' tcp dport 853 counter drop
    
    Don't name a chain fwd (syntax error, unexpected fwd). Keep the drill in its own table so cleanup is one nft delete table inet drill, and it never touches Docker's own tables.
  • Avoid clever matches (skuid, cgroup) unless you've confirmed with counters that they match. A plain destination plus port match is easier to trust.

3. Defeat the layers that hide the outage

Caches, stale answers and connection reuse make a probe succeed through a dead dependency:

  • Probe something no cache can hold: a never-seen random name or path per run (e.g. probe-$(date +%s%N)-$RANDOM.cloudflare.com), accepting a real negative answer (NXDOMAIN or 404) as success. For DNS, pick a zone with NSEC "black lies" (cloudflare.com) so aggressive-NSEC can't synthesize the answer locally.
  • "Flush" commands aren't always deletes. Check whether they only expire entries.
  • Long-lived TLS/TCP connections can keep working briefly after a new-connection block; a stateless drop of all packets to the destination avoids that.

4. Always arm a dead-man revert

Anything that can cut the host off, or leave production degraded if your session dies, gets a timed undo that outlasts the drill, armed before the fault:

systemd-run --quiet --unit=drill-deadman --on-active=240 /bin/sh -c \
  'nft delete table inet drill; <restore production state>; systemctl start <paused-timer>'

Cancel it on a clean finish (systemctl stop drill-deadman.timer; systemctl reset-failed drill-deadman.service) and confirm it's gone. Size the deadline to outlast the whole run: a revert that fires mid-drill voids the result.

5. Never page a human for a designed failure

  • Pause the real watcher's timer for the drill, and run the real watcher by hand with its alert function stubbed to a log file (send_telegram(){ echo "$1" >> /run/drill/tg.log; }). Restart the timer at the end, and in the dead-man.
  • Point the run's state at a scratch dir, so the production watcher's transition state isn't disturbed.
  • Verify the real alert path separately without sending: e.g. Telegram getMe with the token read from the env file via curl -K <(printf 'url = ...'), never in argv.
  • Design watchers with env overrides for exactly this (PDF_LIB, PDF_STATE_DIR, PDF_FTL_BIN, ports, probe host). Production sets none of them.

6. Test a new or changed watcher before it ships

Run the old and new watcher side by side against a throwaway instance of the dependency on a spare port (a second Unbound on :5399 with the production config, its own control socket and pidfile, chroot: ""). Feed both the same fault. The result worth keeping is the comparison: "the old one stayed healthy, the new one failed over in 4 s". Stub the side-effect binaries too (a fake FTL that records the upstream it was set to), and confirm production state is unchanged at the end.

A throwaway instance started cold can hide a reload-path bug: on 2026-10-23 the test Unbound was fine, while production, changed by HUP, failed every TLS handshake (HUP doesn't load tls-cert-bundle). If production will be changed by reload, test a reload too.

7. The sequence

  1. Baseline: watcher healthy, real path verified (e.g. proto.on.quad9.net → "dot").
  2. Fault: apply the block; confirm an uncached request genuinely fails before any reaction.
  3. Failover: time it, then confirm real traffic is served on the fallback and the protections stay intact (ad blocking, DNSSEC).
  4. Counters: print them. Non-zero, or the run is void.
  5. Recovery: lift the block; time the revert and the recovery alert; re-verify the real path. One empty probe right after a service restart is normal. Re-check before calling it a failure.
  6. Cleanup: drill table gone, dead-man cancelled, real timer active, production state back to the steady-state value, the scratch dir removed.

Related: pihole-dot-upstream-failover (the DNS watcher these lessons came from), ssh-exposure-audit (dead-man around SSH/firewall changes), unifi-zone-firewall-testing (packet-testing gateway policies with controls).

finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the workStars: 319kUpdated Claude CodeCodexVerifiedInstall
systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixesStars: 319kUpdated Claude CodeCodexVerifiedInstall
test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation codeStars: 319kUpdated Claude CodeCodexVerifiedInstall
skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.Stars: 197kUpdated Claude CodeCodexVerifiedInstall