Hardware and Software Faults

When we think of causes of system failure, hardware faults quickly come to mind:

• Approximately 2%–5% of magnetic hard drives fail per year; in a storage cluster with 10,000 disks, we should therefore expect on average one disk failure per day. Recent data suggests that disks are getting more reliable, but failure rates remain significant.

• Approximately 0.5%–1% of solid state drives (SSDs) fail per year. Small numbers of bit errors are corrected automatically, but uncorrectable errors occur approximately once per year per drive, even in drives that are fairly new (i.e., that have experienced little wear). This error rate is higher than that of magnetic hard drives.

• Other hardware components (such as power supplies, RAID controllers, and memory modules) also fail, although less frequently than hard drives.

• Approximately 1 in 1,000 machines has a CPU core that occasionally computes the wrong result, likely because of manufacturing defects. In some cases an erroneous computation leads to a crash, but in other cases it leads to a program simply returning the wrong result.

• Data in RAM can be corrupted, either because of random events such as cosmic rays or because of permanent physical defects. Even when memory with error-correcting codes (ECC) is used, more than 1% of machines encounter an uncorrectable error in a given year, which typically leads to a crash of the machine and the affected memory module needing to be replaced. Furthermore, certain pathological memory access patterns can flip bits with high probability.

• An entire datacenter might become unavailable (e.g., because of a power outage or network misconfiguration) or even be permanently destroyed (e.g., by fire, flood, or earthquake). A solar storm, which induces large electrical currents in long-distance wires when the sun ejects a large mass of charged particles, could damage power grids and undersea network cables. Although such large-scale failures are rare, their impact can be catastrophic if a service cannot tolerate the loss of a datacenter.

These events are rare enough that you often don’t need to worry about them when working on a small system, as long as you can easily replace hardware that becomes faulty. However, in a large-scale system, hardware faults happen often enough that they become part of normal system operation.

Tolerating hardware faults through redundancy

Our first response to unreliable hardware is usually to add redundancy to the individual hardware components in order to reduce the failure rate of the system. Disks may be set up in a RAID configuration (spreading data across multiple disks in the same machine so that a failed disk does not cause data loss), servers may have dual power supplies and hot-swappable CPUs, and datacenters may have batteries and diesel generators for backup power. Such redundancy can often keep a machine running uninterrupted for years.

Redundancy is most effective when component faults are independent—that is, when the occurrence of one fault does not change the likelihood that another fault will occur. However, experience has shown significant correlations between component failures. Unavailability of an entire server rack or an entire datacenter still happens more often than we would like.

Hardware redundancy increases the uptime of a single machine; however, as discussed in “Distributed Versus Single-Node Systems”, using a distributed system has advantages, such as being able to tolerate a complete outage of one datacenter. For this reason, cloud systems tend to focus less on the reliability of individual machines and instead aim to make services highly available by tolerating faulty nodes at the software level. Cloud providers use availability zones to identify which resources are physically co-located; resources in the same place are more likely to fail at the same time than geographically separated resources.

The fault-tolerance techniques we discuss in these notes are designed to tolerate the loss of entire machines, racks, or availability zones. They generally work by allowing a machine in one datacenter to take over when a machine in another datacenter fails or becomes unreachable. We will discuss such techniques for fault tolerance in the related discussion, 10, and various other points in these notes.

Systems that can tolerate the loss of entire machines also have operational advantages. A single-server system requires planned downtime if you need to reboot the machine (to apply operating system security patches, for example), whereas a multi-node fault-tolerant system can be patched by restarting one node at a time, without affecting the service for users. This is called a rolling upgrade, and we will discuss it further in the related discussion.

Software faults

Although hardware failures can be weakly correlated, they are still mostly independent—for example, if one disk fails, other disks in the same machine will likely be fine, at least for a while. On the other hand, software faults are often very highly correlated, because it is common for many nodes to run the same software and thus have the same bugs. Such faults are harder to anticipate, and they tend to cause many more system failures than uncorrelated hardware faults. Examples include the following:

• A software bug that causes every node to fail at the same time in particular circumstances. For instance, on June 30, 2012, a leap second caused many Java applications to hang simultaneously because of a bug in the Linux kernel, bringing down several internet services. And because of a firmware bug, all SSDs of certain models suddenly fail after precisely 32,768 hours of operation (less than four years), rendering the data on them unrecoverable.

• A runaway process that uses up a shared, limited resource, such as CPU time, memory, disk space, network bandwidth, or threads. For instance, a process that consumes too much memory while processing a large request may be killed by the operating system, or a bug in a client library could cause a much higher request volume than anticipated.

• A service that the system depends on slows down, becomes unresponsive, or starts returning corrupted responses.

• An interaction between different systems results in emergent behavior that does not occur when each system is tested in isolation.

• Cascading failures, where a problem in one component causes another component to become overloaded and slow down, which in turn brings down another component.

The bugs that cause these kinds of software faults often lie dormant for a long time until they are triggered by an unusual set of circumstances. In those circumstances, it is revealed that the software is making some kind of assumption about its environment—and while that assumption is usually true, it eventually stops being true for some reason.

The problem of systematic faults in software has no quick solution. Lots of small things can help: carefully thinking about assumptions and interactions in the system; thorough testing; ensuring process isolation; allowing processes to crash and restart; avoiding feedback loops such as retry storms (see “When an Overloaded System Won’t Recover”); measuring, monitoring, and analyzing system behavior in production.