Availability & Distributed Systems Resilience — AWS Whitepaper Review
I read an AWS whitepaper on availability, primarily on distributed systems resilience and I decided to write about a few things that stood out.
I didn't know what to expect, maybe learn some theory or so. What I got was something that reframed how I think about building reliable systems.
Availability
It is simply the percentage of time your workload is up and usable. Not a feeling, not a vibe. A number you should be able to prove mathematically.
A = uptime / (uptime + downtime)
Types of Failure
Not all failure is total failure. Availability breaks down across partial failures, affecting a percentage of users, a subset of regions, or a percentile of latency. Your system can be "up" and still quietly failing a chunk of your customers.
F = 1 - A
The probability your system is down is just the inverse of its availability. Simple, but sobering when you actually calculate it.
MTBF, MTTD and MTTR
Three numbers that tell the whole story. How often you fail (MTBF — Mean Time Before Failure). How fast you spot it (MTTD — Mean Time To Detection). How fast you recover (MTTR — Mean Time To Recovery).
A = MTBF / (MTBF + MTTR)
Teams might obsess over MTBF and underinvest in MTTD and MTTR, which is where the real leverage is.
Distributed Systems
Hardware fails predictably. Software doesn't. Most production bugs are "Heisenbugs", transient and condition-specific, which is why retry logic matters more than most engineers think.
When you add dependencies, your maximum theoretical availability becomes:
A = α1 × α2 × ... × αn
Every dependency you add is another number less than one multiplied into your availability. Fewer dependencies means higher availability.
Fault Tolerance and Isolation
Fault tolerance keeps you running when something breaks. Redundancy is how you achieve it. Two subsystems each at 99% availability gives you:
A = 1 - (1 - 0.99)² = 99.99%
But redundancy has diminishing returns. Beyond three spares, the gains become fractions of a second of downtime per year. The cost stops being worth it.
Fault isolation makes sure that when something breaks, it only breaks that thing. Together they are the difference between a blip and an outage.
There is a difference between knowing this stuff and actually building like you believe it.
Worth a read if availability or distributed systems are things you are interested in.