Network packets have a maximum size (generally a few kilobytes), but many applications need to send messages (requests, responses) that are too big to fit in one packet. These applications most often use TCP, the Transmission Control Protocol, to establish a connection that breaks large data streams into individual packets and puts them back together again on the receiving side.
Note
Most of what we say about TCP applies also to its more recent alternative, QUIC, as well as the Stream Control Transmission Protocol (SCTP) used in WebRTC, the BitTorrent uTP protocol, and other transport protocols. For a comparison to UDP, see “TCP Versus UDP”.
TCP is often described as providing “reliable” delivery, in the sense that it detects and retransmits dropped packets, it detects reordered packets and puts them back in the correct order, and it detects packet corruption by using a simple checksum. It also figures out how fast it can send data so that it is transferred as quickly as possible, but without overloading the network or the receiving node; this is known as congestion control, flow control, or backpressure.
When you “send” data by writing it to a socket, it doesn’t get sent immediately; it’s placed in a buffer managed by your operating system. When the congestion control algorithm decides that it has capacity to send a packet, it takes the next packet’s worth of data from that buffer and passes it to the network interface. The packet passes through several switches and routers, and eventually the receiving node’s operating system places the packet’s data in a receive buffer and sends an acknowledgment packet back to the sender. Only then does the receiving operating system notify the application that more data has arrived.
So, if TCP provides “reliability,” does that mean we no longer need to worry about networks being unreliable? Unfortunately not. TCP decides that a packet must have been lost if no acknowledgment arrives within a certain timeout, but it can’t tell whether it was the outbound packet or the acknowledgment that was lost. Although it can resend the packet, it can’t guarantee that the new packet will get through (if the network cable is unplugged, for example, TCP can’t plug it back in for you). Eventually, after a configurable timeout, it gives up and signals an error to the application. TCP’s deduplication and retransmission capabilities apply to only a single connection, so if the application reconnects and retransmits, data could be duplicated.
If a TCP connection is closed with an error—perhaps because the remote node crashed or the network was interrupted—you unfortunately have no way of knowing how much data was actually processed by the remote node. Even if you receive an acknowledgment that a packet was delivered, this means only that the operating system kernel on the remote node received it; the application may have crashed before it handled that data. If you want to be sure that a request was successful, you need a positive response from the application itself.
Nevertheless, TCP is very useful because it provides a convenient way of sending and receiving messages that are too big to fit in one packet. Once a TCP connection is established, you can also use it to send multiple requests and responses. This is usually done by first sending a header that indicates the length of the following message in bytes, followed by the actual message. HTTP and many RPC protocols (see “Dataflow Through Services: REST and RPC”) work like this.