Resilience isn't a product you buy — it's a question you ask at every layer: "what happens when this dies?" Redundancy added randomly is expensive theatre; redundancy added where the single points of failure actually are is engineering. This is the SPOF audit — a systematic walk through every layer to find what takes you down, and what's worth fixing.
A single point of failure is any component whose death takes down more than you can accept. Every network has them; the question is whether you chose which ones to keep. Resilience engineering is not "add redundancy everywhere" — that's how you double the cost and the complexity while still missing the one SPOF that actually bites. It's a deliberate audit: walk each layer, name what happens when each thing fails, price the fix against the cost of the outage, and redundant-ify only where the math works.
The method is a layer-by-layer walk — power, links, devices, WAN, protocols, and services — asking the same question at each: if this single thing dies, what goes down, for how long, and is that acceptable? Where it isn't, you already have the tools from this series: FHRP for gateways, LACP for links, dual-WAN for uplinks, redundant cores for devices. This post is the framework that tells you where to spend them.
01
Walk the stack from the wall socket up. At each layer, name the single thing whose failure hurts, and decide if it's acceptable.
| Layer / component | The fix (if the outage is unacceptable) |
|---|---|
| Power (one PSU, one feed) | Dual PSUs on different circuits/UPS; generator for critical sites |
| A single uplink cable | LACP bond across two links (and two line cards) |
| The core switch/router | Redundant pair + FHRP for the gateway |
| The default gateway IP | HSRP/VRRP virtual gateway |
| The single ISP / WAN link | Dual-WAN (second provider, ideally different medium) |
| A routing/VPN path | BFD + a feasible backup path |
| DHCP / DNS / RADIUS server | A second server; anycast or failover pairing |
| The one engineer who knows it | Documentation & backups — the most-ignored SPOF |
02
The same SPOF is worth fixing at one site and ignoring at another. What decides it is the cost of the outage, not the technical satisfaction of the fix.
A dead core switch at a 3-person office is a bad afternoon; at a 200-bed hospital it's a safety incident. Redundancy has real costs — the second device, the added config complexity, the new failure modes redundancy itself introduces (a split-brain FHRP, a misconfigured bond). So you price it: expected outage cost (frequency × duration × business impact) against redundancy cost (hardware + complexity + the risk the redundancy misbehaves). Fix the SPOFs where downtime is expensive or dangerous; consciously accept the ones where a short outage is cheaper than eliminating it. Writing down "we accept X as a SPOF because Y" is a legitimate, senior engineering decision — far better than pretending it doesn't exist.
The cruelest failures are the ones where the redundancy was there and didn't work: the backup ISP whose config drifted, the secondary firewall that was never in sync, the UPS with a dead battery, the failover that had never actually been triggered. Untested redundancy is worse than none, because it buys false confidence. Design the failover, then test it by causing the failure — pull the cable, kill the primary, drain the UPS — on a schedule. The broken-network and field-notes posts exist because failure drills are where you learn what your design actually does, versus what you assumed.
03
Teams that redundant-ify the obvious hardware still get taken down by these. They're cheap to fix and routinely ignored.
| Forgotten SPOF | Why it bites |
|---|---|
| Both "redundant" links in one conduit | One backhoe cuts both — diverse physical paths matter |
| Both uplinks to the same ISP / same POP | The provider's outage is still your outage — different providers |
| Two power supplies on one circuit | Dual PSU, single breaker = single point of failure |
| No config backups / no OOB access | Hardware survives; recovery doesn't. See OOB |
| One person holds all the knowledge | Documentation is redundancy for the human layer |
| Expired certificate / licence | A silent time-bomb SPOF — monitor expiry dates |
Takeaways
NOCTIS audits and designs for resilience across Crete — a real SPOF review across power, links, devices, and WAN, with redundancy where it counts and failover we prove by pulling the cable.
Book a Discovery Call →