“Latency” and “response time” are sometimes used interchangeably, but in these notes we will use these and a few related terms in a specific way (illustrated in Figure 2-4):
-
The response time is what the client sees; it includes all delays incurred anywhere in the system.
-
The service time is the duration for which the service is actively processing the client’s request.
-
Queueing delays can occur at several points in the flow—for example, after a request is received, it might need to wait until a CPU is available before it can be processed, or a response packet might need to be buffered before it is sent over the network if other tasks on the same machine are sending a lot of data via the outbound network interface.
-
Latency is a catchall term for time during which a request is not being actively processed—that is, during which it is latent. In particular, network latency or network delay refers to the time that a request and response spend traveling through the network.
Diagram illustrating the components of response time, showing user and service interaction with network latency, queueing delay, and service time.
Figure 2-4. Response time, service time, network latency, and queueing delay
In Figure 2-4, time flows from left to right; each communicating node is shown as a horizontal line, and a request or response message is shown as a thick diagonal arrow from one node to another. You will encounter this style of diagram frequently over the course of these notes.
The response time can vary significantly from one request to the next, even if you keep making the same request over and over again. Many factors can add random delays—for example, a context switch to a background process, the loss of a network packet and TCP retransmission, a garbage collection pause, a page fault forcing a read from disk, or mechanical vibrations in the server rack. We will discuss this topic in more detail in “Timeouts and Unbounded Delays”.
Queueing delays often account for a large part of the variability in response times. As a server can process only a small number of things in parallel (limited, for example, by its number of CPU cores), it takes only a small number of slow requests to hold up the processing of subsequent requests—an effect known as head-of-line blocking. Even if those subsequent requests have fast service times, the client will see a slow overall response time due to the time waiting for the prior request to complete. The queueing delay is not part of the service time, and for this reason it is important to measure response times on the client side.