We have been building computer networks for decades—one might hope that by now we would have figured out how to make them reliable. Unfortunately, we have not yet succeeded. Some systematic studies and plenty of anecdotal evidence show that network problems are surprisingly common, even in controlled environments like a datacenter operated by one company:
-
One study in a medium-sized datacenter found about 12 network faults per month, of which half disconnected a single machine and half disconnected an entire rack.
-
Another study measured the failure rates of components like top-of-rack switches, aggregation switches, and load balancers. It found that adding redundant networking gear doesn’t reduce faults as much as you might expect, since it doesn’t guard against human error (e.g., misconfigured switches), which is a major cause of outages.
-
Interruptions of wide-area fiber links have been blamed on cows, beavers, and sharks (though shark bites have become rarer with better shielding of submarine cables). Humans are also often at fault, via accidental misconfiguration, scavenging, or sabotage.
-
Across cloud regions, round-trip times of up to several minutes have been observed at high percentiles [18, Table 3]. Even within a single datacenter, packet delay of more than a minute can occur during a network topology reconfiguration, triggered by a problem during a software upgrade for a switch. Thus, we have to assume that messages might be delayed arbitrarily.
-
Sometimes communications are partially interrupted, depending on who you’re talking to—for example, A and B can communicate, and B and C can communicate, but A and C cannot. Other surprising faults include a network interface that sometimes drops all inbound packets but sends outbound packets successfully. Just because a network link works in one direction doesn’t guarantee it’s also working in the opposite direction.
-
Even a brief network interruption can have repercussions that last for much longer than the original issue.
Even if network faults are rare in your environment, the fact that faults can occur means that your software needs to be able to handle them. Whenever any communication happens over a network, it may fail—there is no way around it.
Network partitions
The term network partition, or netsplit, is sometimes used when one part of the network is cut off from the rest because of a network fault. However, this is not fundamentally different from other kinds of network interruption. Network partitions are not related to sharding of a storage system, which is sometimes also called partitioning.
If the error handling of network faults is not defined and tested, arbitrarily bad things could happen—for example, the cluster could become deadlocked and permanently unable to serve requests, even when the network recovers, or it could potentially delete all of your data. If software is put in an unanticipated situation, it may do arbitrary unexpected things.
Handling network faults doesn’t necessarily mean tolerating them. If your network is normally fairly reliable, a valid approach may be to simply show an error message to users while your network is experiencing problems. However, you do need to know how your software reacts to network problems and ensure that the system can recover from them. It may make sense to deliberately trigger network problems and test the system’s response (see “Fault injection”).