If a system is close to overload, with throughput pushed close to the limit, it can sometimes enter a vicious cycle where it becomes less efficient and hence even more overloaded. For example, if a long queue of requests is waiting to be handled, response times may increase so much that clients time out and resend their requests. This causes the rate of requests to increase even further, making the problem worse—a retry storm. Even when the load is reduced again, such a system may remain in an overloaded state until it is rebooted or otherwise reset. This phenomenon is called a metastable failure, and it can cause serious outages in production systems.
To avoid retries overloading a service, you can increase and randomize the time between successive retries on the client side (exponential backoff) and temporarily stop sending requests to a service that has returned errors or timed out recently (by using a circuit breaker or token bucket algorithm). The server can also detect when it is approaching overload and start proactively rejecting requests (load shedding), or send back responses asking clients to slow down (backpressure). The choice of queueing and load balancing algorithms can also make a difference.
In terms of performance metrics, the response time is usually what users care about the most, whereas the throughput determines the required computing resources (e.g., how many servers you need) and hence the cost of serving a particular workload. If throughput is likely to increase beyond the current hardware’s capability, the capacity needs to be expanded; a system is said to be scalable if its maximum throughput can be significantly increased by adding computing resources.
In this section we will focus primarily on response times, and we will return to throughput and scalability in “Scalability”.