In the past, older computers such as mainframes were made reliable by ensuring that individual components were redundant—for example, by employing RAID to sustain individual disk failures. As discussed in “Shared-Memory, Shared-Disk, and Shared-Nothing Architectures”, the distributed systems we focus on in these notes are mostly shared-nothing systems: a bunch of machines connected by a network. Instead of having redundancy of components within a single machine, shared-nothing systems use replication across separate machines for redundancy. The network is the only way these machines can communicate. We assume that each machine has its own memory and disk, and one machine cannot access another machine’s memory or disk (except by making requests to a service over the network). Even when storage is shared, such as with object storage, machines communicate with shared storage services over the network.
The internet and most internal networks in datacenters (often Ethernet) are asynchronous packet networks. In this kind of network, one node can send a message (a packet) to another node, but the network gives no guarantees as to when it will arrive or whether it will arrive at all. If you send a request and expect a response, many things could go wrong (some of which are illustrated in Figure 9-1):
-
Your request may have been lost (perhaps someone unplugged a network cable).
-
Your request may be waiting in a queue and will be delivered later (perhaps the network or the recipient is overloaded).
-
The remote node may have failed (perhaps it crashed or it was powered down).
-
The remote node may have temporarily stopped responding (perhaps it is experiencing a long GC pause; see “Process Pauses”), but it will start responding again later.
-
The remote node may have processed your request, but the response has been lost on the network (perhaps a network switch has been misconfigured).
-
The remote node may have processed your request, but the response has been delayed and will be delivered later (perhaps the network or your own machine is overloaded).
Diagram illustrating network communication issues, showing potential disruptions between client, network, and service, with undelivered requests and unresponsive nodes over time.
Figure 9-1. If you send a request and don’t get a response, it’s not possible to distinguish whether (a) the request was lost, (b) the remote node is down, or (c) the response was lost.
The sender can’t even tell whether the packet was delivered. The only option is for the recipient to send a response message, which may in turn be lost or delayed. These issues are indistinguishable in an asynchronous network; the only information you have is that you haven’t received a response yet. If you send a request to another node and don’t receive a response, it is impossible to tell why.
The usual way of handling this issue is a timeout: after some time, you give up waiting and assume that the response is not going to arrive. However, when a timeout occurs, you still don’t know whether the remote node got your request (and if the request is still queued somewhere, it may still be delivered to the recipient, even if you’ve given up on that possibility).