In this topic we discussed new approaches to designing data systems based on ideas from stream processing. We started with the observation that no one single tool can efficiently serve all possible use cases, so applications must compose several pieces of software to accomplish their goals. We discussed how to solve this data integration problem by using batch processing and event streams to let data changes flow among systems.
In this approach certain systems are designated as systems of record, and other data is derived from them through transformations. In this way we can maintain indexes, materialized views, machine learning models, statistical summaries, and more. Making these derivations and transformations asynchronous and loosely coupled helps prevent a problem in one area from spreading to unrelated areas, increasing the robustness and fault tolerance of the system as a whole.
Expressing dataflows as transformations from one dataset to another also helps evolve applications. If you want to change one of the processing steps—for example, to alter the structure of an index or cache—you can just rerun the new transformation code on the whole input dataset to rederive the output. Similarly, if something goes wrong, you can fix the code and reprocess the data to recover.
These processes are quite similar to what databases already do internally, so we recast the idea of dataflow applications as unbundling the components of a database and building an application by composing these loosely coupled components.
Derived state can be updated by observing changes in the underlying data. That state can also be observed by downstream consumers. We can even take this dataflow all the way through to the end-user device displaying the data, and thus build UIs that dynamically update to reflect data changes and continue to work offline.
Next, we discussed how to ensure that all this processing remains correct in the presence of faults. We saw that strong integrity guarantees can be implemented scalably with asynchronous event processing, by using end-to-end request identifiers to make operations idempotent, and by checking constraints asynchronously. Clients can either wait until the check has passed, or go ahead without waiting but risk having to apologize about a constraint violation. This approach is much more scalable and robust than the traditional approach of using distributed transactions and fits with how many business processes work in practice.
By structuring applications around dataflow and checking constraints asynchronously, we can avoid most coordination and create systems that maintain integrity but still perform well, even in geographically distributed scenarios and in the presence of faults. To conclude, we talked a little about using audits to verify the integrity of data and detect corruption and observed that the techniques used by blockchains also have a similarity to event-based systems.