Many systems need to automatically detect faulty nodes. For example:
-
A load balancer needs to stop sending requests to a node that is dead (i.e., take it out of rotation).
-
In a distributed database with single-leader replication, if the leader fails, one of the followers needs to be promoted to be the new leader (see “Handling Node Outages”).
Unfortunately, the uncertainty about the network makes it difficult to tell whether a node is working. In specific circumstances, you might get feedback to explicitly tell you that something is not working:
-
If you can reach the machine on which the node should be running, but no process is listening on the destination port (e.g., because the process crashed), the operating system will helpfully close or refuse TCP connections by sending an RST or FIN packet in reply.
-
If a node process crashed (or was killed by an administrator) but the node’s operating system is still running, a script can notify other nodes about the crash so that another node can take over quickly without having to wait for a timeout to expire. For example, HBase does this.
-
If you have access to the management interface of the network switches in your datacenter, you can query them to detect link failures at a hardware level (e.g., if the remote machine is powered down). This option is ruled out if you’re connecting via the internet, if you’re in a shared datacenter with no access to the switches themselves, or if you can’t reach the management interface because of a network problem.
-
If a router is sure that the IP address you’re trying to connect to is unreachable, it may reply to you with an ICMP Destination Unreachable packet. However, the router doesn’t have a magic failure detection capability either; it is subject to the same limitations as other participants in the network.
Rapid feedback about a remote node being down is useful, but you can’t count on it. If something has gone wrong, you may get an error response at some level of the stack, but in general you have to assume that you will get no response at all. You can retry a few times, wait for a timeout to elapse, and eventually declare the node dead if you don’t hear back within the timeout. Since the node could actually be alive, you need to strike a balance between false positives and false negatives: too short a timeout causes alive nodes to be incorrectly suspected to be dead, and too long a timeout causes unnecessary delays waiting for dead nodes.