In the batch processing world, the inputs and outputs of a job are files (perhaps on a distributed filesystem). What does the streaming equivalent look like?
- 13.01Transmitting Event StreamsIn the batch processing world, the inputs and outputs of a job are files (perhaps on a distributed filesystem). What does the streaming equivalent look like?
- 13.02Messaging SystemsA common approach for notifying consumers about new events is to use a messaging system: a producer sends a message containing the event, which is then pushed to consumers. We…
- 13.03Log-Based Message BrokersSending a packet over a network or making a request to a network service is normally a transient operation that leaves no permanent trace. Although it is possible to record…
- 13.04Databases and StreamsWe have drawn some comparisons between message brokers and databases. Even though they have traditionally been considered separate categories of tools, we saw that log-based…
- 13.05Keeping Systems in SyncAs we have seen throughout these notes, no single system can satisfy all data storage, querying, and processing needs. In practice, most nontrivial applications need to combine…
- 13.06Change Data CaptureThe problem with most databases’ replication logs is that they have long been considered to be an internal implementation detail of the database, not a public API. Clients are…
- 13.07State, Streams, and ImmutabilityWe saw in the related discussion that batch processing benefits from the immutability of its input files, so you can run experimental processing jobs on existing input files…
- 13.08Processing StreamsSo far in this topic we have talked about where streams come from (user activity events, sensors, and writes to databases) and how streams are transported (through direct…
- 13.09Uses of Stream ProcessingStream processing has long been used for monitoring purposes, where an organization wants to be alerted if certain things happen. For example:
- 13.10Reasoning About TimeStream processors often need to deal with time, especially when running analytics tasks, which frequently use time windows such as “the average over the last five minutes.” The…
- 13.11Stream JoinsIn “Joins and Grouping” we discussed how batch jobs can join datasets by key and how such joins form an important part of data pipelines. Since stream processing generalizes…
- 13.12Fault ToleranceIn the final section of this topic, let’s consider how stream processors can tolerate faults. We saw in the related discussion that batch processing frameworks can tolerate…
- 13.13Recap: Stream ProcessingIn this topic we discussed event streams, the purposes they serve, and how to process them. In some ways, stream processing is very much like the batch processing we discussed…