jackson2w/claude-code-skills / forced-outage-drill
forced-outage-drill
FlaggedSimulate a dependency outage on a live host (upstream DNS, an API endpoint, a backup target, a peer) to prove a watcher, failover or alert actually works, without paging anyone and without the drill passing for the wrong reason. Covers proving the block bites (nft counters plus a control) before trusting any result, why container egress needs a forward-hook chain, cache and stale-answer layers that hide the outage, a systemd-run dead-man revert, testing a new watcher against a throwaway instance on a spare port with stubbed alerting, and the drill sequence: baseline, failure surfaces, failover, recovery, cleanup. Trigger phrases include "failover drill", "simulate an outage", "force a failover", "test the watcher", "blackhole the upstream", "nft drop for a test", "dead-man revert", "does the alert actually fire", "drill without paging", "stub telegram", "prove the failover works".
Forced-outage drills that mean something
A drill is only evidence if the outage really happened. The usual failure isn't a crash; it's a green result because the simulated fault never reached the thing under test. On 2026-10-23 a single DNS session hit this three times. All three looked like "the system copes fine":
- an nft rule keyed on
meta skuid 0dropped nothing the test needed, and lookups kept answering; unbound-control flush <name>didn't empty the cache: withserve-expiredit only marks entries expired, and they keep being served;- the drill's container chain was named
fwd, a reserved word in nft, so it was never created; the output-only rule left container egress untouched.
1. Prove the block bites before reading any result
- Put a
counteron every drop rule and print the counters after the fault window. Zero packets means no outage happened, whatever the probes said. - Pair it with a control on the same host that must still work (another port or address on
the same path, e.g.
:443to the same IP while:853is blocked). A dead host and a working block look identical without one. - Quick reachability probe, no extra tools:
timeout 4 bash -c '</dev/tcp/9.9.9.9/853' && echo REACHED || echo BLOCKED. - Expect the failure to surface first. Before the watcher reacts, an uncached request must actually fail. If it succeeds, something (cache, stale answers, a fallback) is masking the fault, and the drill is testing that layer, not the watcher.
2. Block at the layer the traffic actually takes
- Host processes: an
output-hook chain. - Docker containers on a bridge: egress traverses the host's forward path, so an
outputdrop alone does nothing. Add aforward-hook chain too:
Don't name a chainnft add table inet drill nft add chain inet drill out '{ type filter hook output priority 0; }' nft add chain inet drill fw '{ type filter hook forward priority 0; }' nft add rule inet drill out ip daddr '{ 9.9.9.9, 149.112.112.112 }' tcp dport 853 counter drop nft add rule inet drill fw ip daddr '{ 9.9.9.9, 149.112.112.112 }' tcp dport 853 counter dropfwd(syntax error, unexpected fwd). Keep the drill in its own table so cleanup is onenft delete table inet drill, and it never touches Docker's own tables. - Avoid clever matches (
skuid, cgroup) unless you've confirmed with counters that they match. A plain destination plus port match is easier to trust.
3. Defeat the layers that hide the outage
Caches, stale answers and connection reuse make a probe succeed through a dead dependency:
- Probe something no cache can hold: a never-seen random name or path per run (e.g.
probe-$(date +%s%N)-$RANDOM.cloudflare.com), accepting a real negative answer (NXDOMAIN or 404) as success. For DNS, pick a zone with NSEC "black lies" (cloudflare.com) so aggressive-NSEC can't synthesize the answer locally. - "Flush" commands aren't always deletes. Check whether they only expire entries.
- Long-lived TLS/TCP connections can keep working briefly after a new-connection block; a stateless drop of all packets to the destination avoids that.
4. Always arm a dead-man revert
Anything that can cut the host off, or leave production degraded if your session dies, gets a timed undo that outlasts the drill, armed before the fault:
systemd-run --quiet --unit=drill-deadman --on-active=240 /bin/sh -c \
'nft delete table inet drill; <restore production state>; systemctl start <paused-timer>'
Cancel it on a clean finish (systemctl stop drill-deadman.timer; systemctl reset-failed drill-deadman.service) and confirm it's gone. Size the deadline to outlast the whole run: a
revert that fires mid-drill voids the result.
5. Never page a human for a designed failure
- Pause the real watcher's timer for the drill, and run the real watcher by hand with its alert
function stubbed to a log file (
send_telegram(){ echo "$1" >> /run/drill/tg.log; }). Restart the timer at the end, and in the dead-man. - Point the run's state at a scratch dir, so the production watcher's transition state isn't disturbed.
- Verify the real alert path separately without sending: e.g. Telegram
getMewith the token read from the env file viacurl -K <(printf 'url = ...'), never in argv. - Design watchers with env overrides for exactly this (
PDF_LIB,PDF_STATE_DIR,PDF_FTL_BIN, ports, probe host). Production sets none of them.
6. Test a new or changed watcher before it ships
Run the old and new watcher side by side against a throwaway instance of the dependency on a
spare port (a second Unbound on :5399 with the production config, its own control socket and
pidfile, chroot: ""). Feed both the same fault. The result worth keeping is the comparison:
"the old one stayed healthy, the new one failed over in 4 s". Stub the side-effect binaries too
(a fake FTL that records the upstream it was set to), and confirm production state is unchanged
at the end.
A throwaway instance started cold can hide a reload-path bug: on 2026-10-23 the test Unbound
was fine, while production, changed by HUP, failed every TLS handshake (HUP doesn't load
tls-cert-bundle). If production will be changed by reload, test a reload too.
7. The sequence
- Baseline: watcher healthy, real path verified (e.g.
proto.on.quad9.net→"dot"). - Fault: apply the block; confirm an uncached request genuinely fails before any reaction.
- Failover: time it, then confirm real traffic is served on the fallback and the protections stay intact (ad blocking, DNSSEC).
- Counters: print them. Non-zero, or the run is void.
- Recovery: lift the block; time the revert and the recovery alert; re-verify the real path. One empty probe right after a service restart is normal. Re-check before calling it a failure.
- Cleanup: drill table gone, dead-man cancelled, real timer active, production state back to the steady-state value, the scratch dir removed.
Related: pihole-dot-upstream-failover (the DNS watcher these lessons came from),
ssh-exposure-audit (dead-man around SSH/firewall changes), unifi-zone-firewall-testing
(packet-testing gateway policies with controls).
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| finishing-a-development-branchobra/superpowersUse when implementation is complete, all tests pass, and you need to decide how to integrate the work | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| systematic-debuggingobra/superpowersUse when encountering any bug, test failure, or unexpected behavior, before proposing fixes | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| test-driven-developmentobra/superpowersUse when implementing any feature or bugfix, before writing implementation code | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| skill-creatoranthropics/skillsCreate new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |