Say you have a web server that appends a line to a log file every time it serves a request. For example, using the NGINX default access log format, one line of the log might…
- 12.01Batch Processing with Unix ToolsSay you have a web server that appends a line to a log file every time it serves a request. For example, using the NGINX default access log format, one line of the log might…
- 12.02Batch Processing in Distributed SystemsThe machine that runs our Unix tools example has a number of components that work together to process the log data:
- 12.03Distributed FilesystemsThe filesystem provided by your operating system is composed of several layers:
- 12.04Object StoresObject storage services such as Amazon S3, Google Cloud Storage, Azure Blob Storage, and OpenStack Swift have become a popular alternative to distributed filesystems for batch…
- 12.05Distributed Job OrchestrationOur operating system analogy also applies to job orchestration. When you execute a Unix batch job, something needs to actually run the awk, sort, uniq, and head processes. Data…
- 12.06Batch Processing ModelsWe have seen how batch jobs are scheduled in a distributed environment. Let’s now turn our attention to how batch processing frameworks process data. The two most common models…
- 12.07MapReduceThe pattern of data processing in MapReduce is very similar to the web server log analysis example in “Simple Log Analysis”:
- 12.08Dataflow EnginesTo fix some of MapReduce’s problems, several new execution engines for distributed batch computations were developed, the most well-known of which are Spark and Flink. They are…
- 12.09Shuffling DataAs we’ve seen, both the Unix tools example at the beginning of the chapter and MapReduce are based on sorting. Batch processors need to be able to sort datasets petabytes in…
- 12.10Joins and GroupingLet’s look at how sorted data simplifies distributed joins and aggregations. We’ll continue with MapReduce for illustration purposes, though these concepts apply to most batch…
- 12.11Query LanguagesOver the years, execution engines for distributed batch processing have matured. The infrastructure is now robust enough to store and process many petabytes of data on clusters…
- 12.12DataFramesData scientists and statisticians are generally used to working with the DataFrame data model found in R and Pandas (see “DataFrames, Matrices, and Arrays”). A DataFrame is…
- 12.13Batch Use CasesNow that we’ve seen how batch processing works, let’s see how it is applied to a range of applications. Batch jobs are excellent for processing large datasets in bulk, but they…
- 12.14Recap: Batch ProcessingIn this topic we explored the design and implementation of batch processing systems. We began with the classic Unix toolchain (awk, sort, uniq, etc.) to illustrate fundamental…