So far in this topic we have talked about where streams come from (user activity events, sensors, and writes to databases) and how streams are transported (through direct messaging, via message brokers, and in event logs).
What remains is to discuss what you can do with the stream once you have it—namely, you can process it. Broadly, you have three options:
-
You can take the data in the events and write it to a database, cache, search index, or similar storage system, from where it can then be queried by other clients. As shown in Figure 12-5, this is a good way of keeping a database in sync with changes happening in other parts of the system—especially if the stream consumer is the only client writing to the database. Writing to a storage system is the streaming equivalent of what we discussed in “Batch Use Cases”.
-
You can push the events to users in some way—for example, by sending email alerts or push notifications, or by streaming the events to a real-time dashboard where they are visualized. In this case, a human is the ultimate consumer of the stream.
-
You can process one or more input streams to produce one or more output streams. Streams may go through a pipeline consisting of several such processing stages before they eventually end up at an output (option 1 or 2).
In the rest of this topic, we will discuss option 3: processing streams to produce other derived streams. A piece of code that processes streams like this is known as an operator or a job. It is closely related to the Unix processes and MapReduce jobs we discussed in the related discussion, and the pattern of dataflow is similar: a stream processor consumes input streams in a read-only fashion and writes its output to a different location in an append-only fashion.
The patterns for sharding and parallelization in stream processors are also very similar to those in MapReduce and the dataflow engines we saw in the related discussion, so we won’t repeat those topics here. Basic mapping operations such as transforming and filtering records also work the same.
The one crucial difference from batch jobs is that a stream never ends. This difference has many implications. As discussed at the start of this topic, sorting does not make sense with an unbounded dataset, so sort-merge joins (see “Joins and Grouping”) cannot be used. Fault-tolerance mechanisms must also change. With a batch job that has been running for a few minutes, a failed task can simply be restarted from the beginning, but with a stream job that has been running for several years, restarting from the beginning after a crash may not be a viable option.