jackson2w/claude-code-skills / pihole-dot-upstream-failover
pihole-dot-upstream-failover
FlaggedThis skill should be used when adding DNS-over-TLS (DoT) as a Pi-hole upstream resolver, when evaluating cloudflared for DoT/DoH proxying, when a "no fallback DNS" or "DNS failures should surface" constraint needs to coexist with wanting resilience against a recursive resolver (e.g. Unbound) going down, when pihole-FTL --config dns.upstreams rejects an unquoted value as invalid JSON, or when designing an alerted (not silent) failover watcher for infrastructure that must not silently mask its own failures. Trigger phrases include "cloudflared proxy-dns", "cloudflared DoT deprecated", "pihole DNS over TLS", "stubby DoT", "pihole-FTL --config dns.upstreams", "dns.upstreams invalid JSON", "Unbound fallback resolver", "no fallback DNS constraint", "alerted failover watcher", "silent fallback DNS", "pihole upstream failover", "Quad9", "Unbound forward-zone DoT", "forward-tls-upstream", "switch Unbound from recursive to forwarding", "tls-cert-bundle certificate verify failed", "unbound reload vs restart", "serve-expired hides an upstream outage", "FTLCONF_dns_upstreams read-only", "docker Pi-hole upstream".
Pi-hole DoT upstream resolvers: tool choice, config gotchas, and failover design
cloudflared proxy-dns is the wrong tool — don't reach for it
cloudflared proxy-dns was deprecated by Cloudflare in November 2025; Cloudflare stopped
shipping it in new cloudflared releases as of February 2026 (confirmed via Pi-hole's own docs,
which now carry an explicit deprecation warning on the guide). Existing installs kept working for
roughly 12 months post-deprecation, but it's not safe to stand up fresh.
Independent of the deprecation: cloudflared proxy-dns's --upstream flag only ever supported
DoH endpoints (https://1.1.1.1/dns-query), never DoT (tls://). If the goal is specifically
DoT (not DoH), this tool was never the right choice, at any point in its history.
Use stubby instead — a purpose-built DoT stub resolver, packaged directly in Debian's own
apt repo (confirmed: stubby 1.6.0-3.2 on Debian 13/trixie, no third-party repo or GPG key
needed). It's the standard community pattern for "forward only to a specific DoT upstream."
stubby config gotchas
- Packaged default listens on
127.0.0.1:53(and::1:53) — will silently coexist with a wildcard0.0.0.0:53bind from something else (e.g. pihole-FTL) on Linux, since a more-specific bind doesn't conflict with a wildcard one. This is a coincidence of bind ordering, not something to rely on — always pin an explicit non-conflicting port. - Port syntax on
listen_addressesisIP@port, e.g.127.0.0.1@5054— notIP#port(that's Pi-hole/Unbound's own convention, easy to cross-contaminate by habit) and not a separate config key. - Packaged default
upstream_recursive_serversis a demo/test list (Sinodun, getdnsapi.net) withtls_pubkey_pinsetentries — replace it entirely for a real deployment. For Cloudflare:
Skipupstream_recursive_servers: - address_data: 1.1.1.1 tls_auth_name: "cloudflare-dns.com" - address_data: 1.0.0.1 tls_auth_name: "cloudflare-dns.com"tls_pubkey_pinset(hostname validation viatls_auth_nameis sufficient) — pinning is brittle across the upstream's own cert rotations and will silently break resolution when they rotate. - A broken/invalid
stubby.ymlproduces a real crash-loop (Could not parse config file: ...), and an Ansible run that only checks for task success (ok/changed, no failures) will look completely clean while the service is actually down. Always verify withsystemctl status/journalctl -u stubbyafter templating a new config, not just the playbook's exit code. See theansible_managedgotcha in the global CLAUDE.md's Ansible section for one concrete way this happens (a template header silently rendering invalid content while Ansible reports success).
pihole-FTL --config dns.upstreams has asymmetric quoting
Setting the value requires quoted JSON strings:
pihole-FTL --config dns.upstreams '[ "127.0.0.1#5335" ]'
An unquoted value ('[ 127.0.0.1#5335 ]') is rejected outright: not valid JSON.
But a bare read of the same key prints the value unquoted:
$ pihole-FTL --config dns.upstreams
[ 127.0.0.1#5335 ]
If you're writing an idempotency check (only write when the live value differs from the target,
to avoid touching FTL on every poll), the read-comparison string and the write-argument string
are not the same string — compare against the unquoted read form, but pass the quoted form
when actually setting. Verify this round-trip live with a throwaway set/read before wiring it
into a script; don't assume the two forms match (this is the same quoting idiom used for
dns.hosts — see the pihole-local-dns-records skill — but that key's values are themselves
"IP HOSTNAME" strings, so its round-trip masked the same underlying asymmetry differently).
Every dns.upstreams change restarts pihole-FTL
Changing dns.upstreams doesn't hot-reload. FTL logs Restarting FTL due to change of dns.upstreams, then does a full internal restart, which means ~0.6s with no DNS at all, plus a
flushed cache. In FTL v6.7 the key is defined with FLAG_RESTART_FTL (src/config/config.c), a
property of the key itself, so the CLI (pihole-FTL --config), the API and a direct TOML edit all
behave the same. There's no supported no-restart path.
What follows for a failover design:
- A failover and its recovery each cost one short total outage. Measured in the homelab's forced test on 2026-09-12: ~7.7s of failed uncached lookups before the switch, then ~0.6s dark at each restart.
- Never write the key unless the value actually changes (the idempotency check above).
- Don't "fix" it by keeping two upstreams configured permanently. That's exactly the standing silent fallback the constraint forbids.
- Make the probe resistant to false positives, since every false failover is a real outage.
Designing resilience without violating a "no silent fallback" constraint
If the environment has a standing rule like "DNS failures should surface, not be silently patched with a secondary resolver" (or any analogous "no silent fallback" infra constraint), an always-live second upstream (Pi-hole picks whichever responds) violates it even if the intent was just resilience — a failing primary resolver gets silently absorbed by the fallback with no signal that anything is wrong.
Pattern that satisfies both goals (resilience and "failures surface"): an actively health-checked, alerted, actuated failover, not a standing blend:
- Steady state: primary resolver only, single upstream, unchanged from a no-fallback baseline.
- A watcher (systemd timer, ~15s interval, with a
flockso runs can't overlap) runs a real query against the primary, not justsystemctl is-active, which misses a wedged-but-running resolver. Three traps, all found in the homelab's own watcher on 2026-09-12:- Require an actual answer, not a zero exit code.
digexits 0 on SERVFAIL, so an exit-code-only probe reads the broken-root-hints case as healthy. Pipe+shortoutput throughgrep -q .. - Probe TCP as well as UDP (
dig +tcp). FTL talks to its upstream over TCP too, and a UDP-only probe was blind to a real 2026-07-22 failure where TCP connections were reset while UDP still answered. - Two attempts per protocol within a run (2s each), so one dropped packet doesn't fail over. A failover restarts FTL (below), so false positives cost real outages. Keep the worst-case probe time inside the interval; the homelab measured 8.1s against a blackholed host.
- If the primary forwards to an upstream (the homelab's Unbound → Quad9 DoT since
2026-10-23), a fixed-name probe only tests the cache. With
serve-expireda stale answer survives for a day, so a dead upstream reads healthy while every uncached name SERVFAILs (reproduced with a test Unbound behind an nft drop). Add a probe for a never-seen random name (pdf-probe-<ns>-<rand>.cloudflare.com) that must return NOERROR/NXDOMAIN.unbound-control flush <name>is NOT a fix: it marks entries expired, which serve-expired then serves. Pick a zone with NSEC "black lies" (cloudflare.com) soaggressive-nseccan't synthesize the answer locally.
- Require an actual answer, not a zero exit code.
- On a genuine health-check failure: clean cutover (replace the upstream entirely, don't blend) to the secondary, plus an immediate alert (Telegram, PagerDuty, whatever's already wired up) — the human is told the instant the fallback engages, not after the fact via a dashboard someone has to remember to check.
- On recovery: revert, and alert again.
- State for detecting transitions (so you only alert on the edge, not every 30s tick) lives
on tmpfs (e.g.
/run/<name>/state), not disk — a reboot always re-verifies live rather than trusting stale state across a restart. Default a missing state file to "healthy" so a normal boot doesn't fire a spurious recovery alert, while a real ongoing outage at boot still alerts correctly (the health check itself is live, not the default assumption). - Route the alert directly from the watcher (a raw API call), not through a monitoring stack dependency (Prometheus/Grafana alerting) — the failover mechanism and its alert shouldn't depend on the monitoring stack being up, especially for infrastructure the monitoring stack itself might depend on (e.g. DNS).
- If this is managed by the same Ansible/config-management tooling that also asserts a baseline state, make sure the baseline-assertion task doesn't fight a legitimate in-progress failover — only correct the config if it's in neither of the two recognized managed states, not unconditionally reset to the steady-state default.
Verify this kind of design with a live test, not just a config review: actually stop the
primary, confirm an uncached query genuinely fails in the gap before the watcher reacts (proves
the failure really surfaces), confirm the cutover and alert both fire, confirm real traffic gets
served during the outage (check the actual query/access log, not just synthetic test queries),
confirm the secondary path is doing what it claims (e.g. packet-capture the failover traffic and
check for a real TLS record header if it's supposed to be encrypted — 17 03 03 at the start of
the payload, not just "port 853 is being used"), then restore the primary and confirm a clean
revert. If the underlying host can reboot, do that too and confirm both services and the
watcher's own state come back correctly rather than trusting a stale assumption.
For a steady-state "is anything still in plaintext" check, capture egress for port 53 and not net <LAN> and not net 100.64.0.0/10 for ~30 s. The control is a second capture of dst port 853 on
the same interface, which must show packets to the provider, or an empty first capture proves
nothing. On 2026-10-25 both Pi-holes gave 0 plain packets against ~40 DoT packets to Quad9.
Diagnosing a "DNSSEC validation looks broken" false alarm
A health check that queries a deliberately-broken-signature test domain (dnssec-failed.org,
sigfail.verteiltesysteme.net) and expects SERVFAIL can false-alarm for a reason that has
nothing to do with Unbound's own validator: those domains are rarely cached anywhere (Unbound's
val-bogus-ttl default is 60s), so almost every check is a cold query, and their own external
authoritative nameservers occasionally take 20-60s+ to respond — a single bare dig with
default retry behavior (~15s total) has no margin for that. Confirmed live 2026-08-30 on the
homelab's Pi-hole: a weekly sweep's pihole_dnssec_validation check FAILed, but the validator
was genuinely correct and fast the whole time.
How to tell "really broken" from "cold query to a slow external test domain" without guessing:
- Bypass Pi-hole-FTL and query Unbound directly on its own port, not through the resolver
chain:
dig +tries=1 +timeout=20 @127.0.0.1 -p 5335 dnssec-failed.org(adjust the port to whatever the localunbound.conf.d/*.confsets —5335here). This isolates "is Unbound itself slow/wrong" from "is something upstream of Unbound (FTL, the LAN path) the problem." - Force a genuinely cold query rather than trusting an ambient cache state:
unbound-control flush_zone dnssec-failed.orgright before the test query. - Raise verbosity to watch the validator's own reasoning in real time:
unbound-control verbosity 3, thenjournalctl -u unbound -f(or--since) while the query runs. Look forDid not match a DS to a DNSKEY, thus bogus/Could not establish a chain of trust— if that line appears (even if the client-sidedigtimed out before ever seeing a reply), the validator concluded correctly and quickly; the delay is downstream of validation logic, in getting a reply back to the client at all. Set verbosity back to 1 when done — verbosity 3 is chatty and not meant to run steady-state. - Cross-check with a working control query in the same breath:
sigok.verteiltesysteme.net(should resolveNOERROR) and a real-world uncached domain (should resolve in well under a second) — if those are fine, general connectivity/root-server reachability isn't the cause, narrowing it further to that one test domain's own external reachability at that moment. - Retry a few times, spaced a minute+ apart. This kind of slowness is usually transient and self-resolving within minutes (external nameserver hiccup, not a local config problem) — confirmed by 4 consecutive fast/correct results minutes after reproducing the slow case live.
If a health-check script needs to encode this distinction itself (not just a one-off manual
diagnosis), don't just lengthen a single dig's timeout — retry a few times with a moderate
per-try timeout, and treat "never got any response" as a warn (worth a glance) rather than a
fail (confirmed broken), reserving fail for an actual non-SERVFAIL answer. See the
homelab's weekly-housekeeping-checks.sh pihole_dnssec_validation check (fixed
2026-08-30, homelab-ansible commit bd8bfc7) for a working example of this shape.
Unbound DoT forwarding gotchas (2026-10-23)
- A HUP reload does not load
tls-cert-bundle. Adding aforward-tls-upstreamzone plus the bundle bysystemctl reload unboundleft every handshake failingcertificate verify failed / unable to get local issuer certificateuntil a full restart. Restart on any TLS-related config change. A fresh test instance on another port won't show it, because it starts cold. - Check the network allows :853 from that host before switching. A gateway "block DoT except
the Pi-hole" rule made the iMac replica SERVFAIL everything.
timeout 4 bash -c '</dev/tcp/9.9.9.9/853'from the host first, with:443as a control. - Never set
forward-first: yesunder a no-silent-fallback constraint: it falls back to recursion when the forwarders fail. - Verify with
dig proto.on.quad9.net TXT(returns"dot"over DoT),dnssec-failed.org→ SERVFAIL, and Google'so-o.myaddr.l.google.com TXT(noedns0-client-subnetline = no ECS). - Quad9-blocked domains that are DNSSEC-signed come back SERVFAIL through a validating Unbound (Quad9's NXDOMAIN has no proof), not NXDOMAIN. Still blocked; just don't read it as an outage.
- Containerized Pi-hole (docker): an
FTLCONF_dns_upstreamsenv var makesdns.upstreamsread-only, so a watcher can't flip it — unpin it. But removing the var does NOT keep the old value: FTL falls back to the image default (8.8.8.8, 8.8.4.4, plaintext) on recreate. Set the value withdocker exec pihole pihole-FTL --config dns.upstreams '[ "127.0.0.1#5335" ]'in the same breath (better: have Ansible set it before the recreate is even possible to observe). The in-container FTL restart a change triggers is fine — the container doesn't restart. - Simulating an upstream outage for a containerized Unbound: its egress goes through the host's
FORWARD path, so an nft
outputdrop alone doesn't block it — add aforward-hook chain too. And don't name the chainfwd: it's a reserved word in nft ("syntax error, unexpected fwd").
Forwarding mode: a leak-test burst trips the provider's per-IP block (2026-10-25)
A DNS leak test (dnsleaktest.org) sends ~1,000+ random, mostly failing names in seconds. Quad9
blocks the whole public IP for ~30-60 s when it sees that, which was proven from a second LAN host
that sent none of the burst. Unbound makes it about 10x worse by default: on every SERVFAIL it
re-asks each forwarder up to outbound-msg-retry (5) times, and the provider gives the same
answer each time. The forwarders get benched (tA 3, rto 120000 in dump_infra), every
uncached name SERVFAILs in 0 ms, and the watcher fails over. Set, on every Unbound:
outbound-msg-retry: 0 # 1 absorbed single-resolver bursts; 0 needed when a client hits both Pi-holes at once
infra-host-ttl: 60 # backstop: a bench clears in ~1 min, not 15+ (the iMac was stuck 18 min)
infra-keep-probing: yes # host-down case only; does not fix this on its own
Even on 0, a two-resolver leak test (~2,300 names in 16 s) still trips about a 10 s block: what
Quad9 reacts to is the failing-random-name pattern, not raw rate (the house's normal peaks reach
118/s without any block). unbound-control set_option rejects 0, so set it in the config.
Every leak test still trips the block and fails both watchers over for ~1 min (one alert pair
per Pi-hole, recovery unaided). Whether the report shows Cloudflare is a race. The test sends
waves over ~15 s, and the watcher's 15 s timer flips the upstream partway through. Any wave after
that goes to Cloudflare. Four tests on 10/25: Cloudflare 145 (retry 1, primary flipped 2 s before
the last wave), 0 (retry 0, burst ended 5 s before the flip), 158 (retry 0, a final wave of 112
landed 2 s after the flip). retry 0 did not change the outcome; the clean run was timing. Expect
Quad9 plus sometimes Cloudflare, never the ISP, and treat either as a pass, not an incident. Don't add a "ignore failovers under a minute" debounce: it
would also hide the first minute of a real outage.
Ruled out: infra-cache-min-rtt (no help), a Pi-hole per-client rate limit (real devices burst to
700-1,365/min), and an Unbound rate cap (normal peaks are above a leak test's per-resolver rate). Testing: a lab Unbound on another port isolates Unbound state, not the
provider's per-IP block, so each burst still briefly affects the house. Pause BOTH watchers (pihole's and
imac's pihole-dot-failover.timer) before any DNS burst test: the per-IP block covers the shared
public IP, so a burst against either resolver benches both. Resume each only after its own
UDP+TCP+random-name probes pass 3x in a row, and check both Pi-holes' change of dns.upstreams
log lines before reporting how many pages went out. Use an external probe from another host as the
control. Detail: the reference_unbound_forwarder_benching memory.
Related skills
| Skill | Stars | Updated | Works with | Status | Install |
|---|---|---|---|---|---|
| writing-skillsobra/superpowersUse when creating new skills, editing existing skills, or verifying skills work before deployment | Stars: 319k | Updated | Claude CodeCodex | Verified | Install |
| xlsxanthropics/skillsUse this skill any time a spreadsheet file is the primary input or output. This means any task where the user wants to: open, read, edit, or fix an existing .xlsx, .xlsm, .xltx, .csv, or .tsv file (e.g., adding columns, computing formulas, formatting, charting, cleaning messy data); create a new spreadsheet from scratch or from other data sources; or convert between tabular file formats. Trigger especially when the user references a spreadsheet file by name or path — even casually (like "the xlsx in my downloads") — and wants something done to it or produced from it. Also trigger for cleaning or restructuring messy tabular data files (malformed rows, misplaced headers, junk data) into proper spreadsheets. The deliverable must be a spreadsheet file. Do NOT trigger when the primary deliverable is a Word document, HTML report, standalone Python script, database pipeline, or Google Sheets API integration, even if tabular data is involved. | Stars: 197k | Updated | Claude CodeCodex | Verified | Install |
| shipping-and-launchaddyosmani/agent-skillsPrepares production launches. Use when preparing to deploy to production, or when asking what needs to be in place before shipping. Use when you need a pre-launch checklist, when setting up monitoring, when planning a staged rollout, or when you need a rollback strategy. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |
| childhoodthedotmack/claude-memBabysit a shipped release after merge, publication, or deployment. Verify published packages, production behavior, installed plugins, and late CI or user reports until the observation window is clean; use babysit for PR readiness before release. | Stars: 118k | Updated | Claude CodeCodex | Verified | Install |