notexOPENOverview
Notex/Systems/Overview: Resiliency
05.01LESSON 01 OF 6 · 2 MIN READ

Overview: Resiliency

Always free to read
LESSON VIDEOOverview: Resiliency

Video coming soon. Start with the reading below.

COMING SOON

“Anything that can go wrong will go wrong.”

– Murphy’s law

In the last part, we discussed the three fundamental scalability patterns: functional decomposition, data partitioning, and replication. They all have one thing in common: they increase the number of moving parts (machines, services, processes, etc.) in our applications. But since every part has a probability of failing, the more moving parts there are, the higher the chance that any of them will fail. Eventually, anything that can go wrong will go wrong [8]; power outages, hardware faults, software crashes, memory leaks — you name it.

Remember when we talked about availability and “nines” in Introduction (lesson 01.01)? Well, to guarantee just two nines, an application can only be unavailable for up to 15 minutes a day. That’s very little time to take any manual action when something goes wrong. And if we strive for three nines, we only have 43 minutes per month available. Clearly, the more nines we want, the faster our systems need to detect, react to, and repair faults as they occur. In this part, we will discuss a variety of resiliency best practices and patterns to achieve that.

To build fault-tolerant applications, we first need to have an idea of what can go wrong. In Common failure causes (lesson 05.02), we will explore some of the most common root causes of failures.

Redundancy (lesson 05.03) describes how to use redundancy, the replication of functionality or state, to increase the availability of a system. As we will learn, redundancy is only helpful when the redundant nodes can’t fail for the same reason at the same time, i.e., when failures are not correlated.

Fault isolation (lesson 05.04) discusses how to isolate correlated failures by partitioning resources and then describes two very powerful forms of partitioning: shuffle sharding and cellular architectures.

Downstream resiliency (lesson 05.05) dives into more tactical resiliency patterns for tolerating failures of downstream dependencies that you can apply to existing systems with few changes, like timeouts and retries.

Upstream resiliency (lesson 05.06) discusses resiliency patterns, like load shedding and rate-limiting, for protecting systems against overload from upstream dependencies.

References

  1. Also known as Murphy’s law
Progress is saved on this device.
KEEP EXPLORING

One note leads to another.

Back to the library →
⌕
Search across the entire library ⌘ K