Batch Processing in Distributed Systems

The machine that runs our Unix tools example has a number of components that work together to process the log data:

• Storage devices that are accessed through the operating system’s filesystem interface

• A scheduler that determines when processes get to run and how to allocate CPU resources to them

• A series of Unix programs whose standard input and standard output (stdin and stdout) are connected together by pipes

These same components exist in distributed data processing frameworks. In fact, you can think of these frameworks as distributed operating systems; they have filesystems, job schedulers, and programs that send data to one another through the filesystem or other communication channels.