A recurring theme in these notes has been that for any given problem, there are several solutions, each with its own pros, cons, and trade-offs. For example, when discussing…
- 14.01Data IntegrationA recurring theme in these notes has been that for any given problem, there are several solutions, each with its own pros, cons, and trade-offs. For example, when discussing…
- 14.02Combining Specialized Tools by Deriving DataTo give one example, it is common to need to integrate an OLTP database with a full-text search index in order to handle queries for arbitrary keywords. Although some databases…
- 14.03Batch and Stream ProcessingThe goal of data integration is to make sure that data ends up in the right form in all the right places. Doing so requires consuming inputs, transforming, joining, filtering,…
- 14.04Unbundling DatabasesAt an abstract level, databases, batch/stream processors, and operating systems all perform the same functions: they store some data, and they allow you to process and query…
- 14.05Composing Data Storage TechnologiesOver the course of these notes we have discussed various features provided by databases and how they work, including these:
- 14.06Designing Applications Around DataflowThe general idea of updating derived data when its underlying data changes is nothing new. For example, spreadsheets have powerful dataflow programming capabilities: you can…
- 14.07Observing Derived StateAt an abstract level, the dataflow systems discussed in the previous section give you a process for creating derived datasets (such as search indexes, materialized views, and…
- 14.08Aiming for CorrectnessWith stateless services that only read data, it’s not a big deal if something goes wrong; you can fix the bug and restart the service, and everything returns to normal.…
- 14.09The End-to-End Argument for DatabasesJust because an application uses a data system that provides comparatively strong safety properties, such as serializable transactions, that does not mean the application is…
- 14.10Enforcing ConstraintsLet’s think about correctness in the context of the ideas around unbundling databases. We saw that end-to-end duplicate suppression can be achieved with a request ID that is…
- 14.11Timeliness and IntegrityA convenient property of many transactional systems is that as soon as one transaction commits, its writes are immediately visible to other transactions. This property is…
- 14.12Trust, but VerifyAll of our discussion of correctness, integrity, and fault tolerance has been under the assumption that certain things might go wrong, but other things won’t. We call such…
- 14.13Recap: A Philosophy of Streaming SystemsIn this topic we discussed new approaches to designing data systems based on ideas from stream processing. We started with the observation that no one single tool can…