The general idea of updating derived data when its underlying data changes is nothing new. For example, spreadsheets have powerful dataflow programming capabilities: you can put a formula in one cell (e.g., the sum of cells in another column), and whenever any input to the formula changes, the result of the formula is automatically recalculated. This is exactly what we want at a data system level. When a record in a database changes, we want any index for that record to be automatically updated and any cached views or aggregations that depend on the record to be automatically refreshed. We should not have to worry about the technical details of how this refresh happens and should be able to simply trust that it works correctly.
Thus, most data systems still have something to learn from the features that VisiCalc already had in 1979. The difference from spreadsheets is that today’s data systems need to be fault-tolerant, scalable, and capable of storing data durably. They also need to be able to integrate disparate technologies written by different groups of people over time and make use of existing libraries and services. It is unrealistic to expect all software to be developed using one particular language, framework, or tool.
In this section we will expand on these ideas and explore some ways of building applications around unbundled databases and dataflow.
Application code as a derivation function
When one dataset is derived from another, it goes through some kind of transformation function. For example:
-
A secondary index is a kind of derived dataset with a straightforward transformation function—for each row or document in the base table, it picks out the values in the columns or fields being indexed and sorts by those values (assuming an SSTable or B-tree index, which are sorted by key).
-
A full-text search index is created by applying various natural language processing functions such as language detection, word segmentation, stemming or lemmatization, spelling correction, and synonym identification, followed by building a data structure for efficient lookups (such as an inverted index).
-
In an ML system, we can consider the model as being derived from the training data by applying various feature extraction and statistical analysis functions. When the model is applied to new input data, its output is derived from that input and its learned parameters (and hence, indirectly, from the training data).
-
A cache often contains an aggregation of data in the form in which it is going to be displayed in a UI. Populating the cache thus requires knowledge of what fields are referenced in the UI; changes in the UI may require updating the definition of how the cache is populated and rebuilding the cache.
The derivation function for a secondary index is so commonly required that it is built into many databases as a core feature, and you can invoke it by merely running CREATE INDEX. For full-text indexing, basic linguistic features for common languages may be built into a database, but the more sophisticated features often require domain-specific tuning. In machine learning, feature engineering is notoriously application-specific and often has to incorporate detailed knowledge about user interaction with and the deployment of an application.
When the function that creates a derived dataset is not a standard cookie-cutter function like the one to create a secondary index, custom code is required to handle the application-specific aspects. This custom code is where many databases struggle. Although relational databases commonly support triggers, stored procedures, and user-defined functions, which can be used to execute application code within the database, they have been somewhat of an afterthought in database design.
Separation of application code and state
In theory, databases could be deployment environments for arbitrary application code, like an operating system. However, in practice they have turned out to be poorly suited for this purpose. They do not fit well with the requirements of modern application development, such as dependency and package management, version control, rolling upgrades, evolvability, monitoring, metrics, calls to network services, and integration with external systems.
On the other hand, deployment and cluster management tools such as Kubernetes, Docker, Mesos, YARN, and others are designed specifically for the purpose of running application code. By focusing on doing one thing well, they are able to do it much better than a database that provides execution of user-defined functions as one of its many features.
Most web applications today are deployed as stateless services, in which any user request can be routed to any application server, and the server forgets everything about the request after it has sent the response. With this style of deployment, servers can be added and removed at will, which is convenient—but the state has to go somewhere (typically, a database). The trend has been to keep stateless application logic separate from state management (databases): not putting application logic in the database and not putting persistent state in the application. As people in the functional programming community like to joke, “We believe in the separation of Church and state”.
Note
Explaining a joke usually ruins it, but here is an explanation anyway so that nobody feels left out. Church is a reference to the mathematician Alonzo Church, who created the lambda calculus, an early form of computation that is the basis for most functional programming languages. The lambda calculus has no mutable state (i.e., no variables that can be overwritten), so one could say that mutable state is separate from Church’s work.
In this typical web application model, the database acts as a kind of mutable shared variable that can be accessed synchronously over the network. The application can read and update the variable, and the database takes care of making it durable, providing some concurrency control and fault tolerance.
However, in most programming languages you cannot subscribe to changes in a mutable variable—you can only read it periodically. Unlike in a spreadsheet, readers of the variable don’t get notified if the value of the variable changes. (You can implement such notifications in your own code—this is known as the observer pattern—but most languages do not have this pattern as a built-in feature.)
Databases have inherited this passive approach to mutable data. If you want to find out whether the content of the database has changed, often your only option is to poll (i.e., to repeat your query periodically). Subscribing to changes is only just beginning to emerge as a feature.
Dataflow: Interplay between state changes and application code
Thinking about applications in terms of dataflow implies renegotiating the relationship between application code and state management. Instead of treating a database as a passive variable that is manipulated by the application, we think much more about the interplay and collaboration between state, state changes, and code that processes them. Application code responds to state changes in one place by triggering state changes in another place.
We have already seen this idea in CDC, in the actor model, in triggers, and in incremental view maintenance. Unbundling the database means applying it to the creation of derived datasets outside of the primary database: caches, full-text search indexes, machine learning, or analytical systems. We can use stream processing and messaging systems for this purpose.
Maintaining derived data requires the following properties, which log-based message brokers can provide:
-
When maintaining derived data, the order of state changes is often important (if several views are derived from an event log, they need to process the events in the same order so that they remain consistent with one another).
-
Fault tolerance is essential—losing just a single message causes the derived dataset to go permanently out of sync with its data source. Both message delivery and derived state updates must be reliable.
Stable message ordering and fault-tolerant message processing are quite stringent demands, but they are much less expensive and more operationally robust than distributed transactions. Modern stream processors can provide these ordering and reliability guarantees at scale, and they allow application code to be run as stream operators.
This application code can do the arbitrary processing that built-in derivation functions in databases generally don’t provide. Like Unix tools chained by pipes, stream operators can be composed to build large systems around dataflow. Each operator takes streams of state changes as input and produces other streams of state changes as output.
Stream processors and services
The currently dominant style of application development breaks functionality into a set of services that communicate via synchronous network requests such as REST APIs. The advantage of such a service-oriented architecture over a single monolithic application is primarily organizational scalability through loose coupling. Different teams can work on different services, which reduces coordination effort among teams (as long as the services can be deployed and updated independently).
Composing stream operators into dataflow systems has a lot of similar characteristics to the microservices approach. However, the underlying communication mechanism is very different: one-directional, asynchronous message streams rather than synchronous request/response interactions.
Besides the advantages listed in “Event-Driven Architectures”, such as better fault tolerance, dataflow systems can also achieve better performance than traditional REST APIs or RPC. For example, say a customer is purchasing an item that is priced in one currency but paid for in another currency. To perform the currency conversion, you need to know the current exchange rate. This operation could be implemented in two ways:
-
In the microservices approach, the code that processes the purchase would probably query an exchange rate service or database to obtain the current rate for a particular currency.
-
In the dataflow approach, the code that processes purchases would subscribe to a stream of exchange rate updates ahead of time and record the current rate in a local database whenever it changes. When it comes to processing the purchase, the processing code can directly query this local database.
The second approach replaces a synchronous network request to another service with a query to a local database (which may be on the same machine, even in the same process). In the microservices approach, you could avoid the synchronous network request by caching the exchange rate locally in the service that processes the purchase. However, to keep that cache fresh, you would need to periodically poll for updated exchange rates or subscribe to a stream of changes—which is exactly what happens in the dataflow approach.
Not only is the dataflow approach faster, but it is also more robust to the failure of another service. The fastest and most reliable network request is no network request at all! Instead of RPC, we now have a stream join between purchase events and exchange rate update events.
The join is time-dependent: if the purchase events are reprocessed at a later point in time, the exchange rate will have changed. If you want to reconstruct the original output, you will need to obtain the historical exchange rate at the original time of purchase. No matter whether you query a service or subscribe to a stream of exchange rate updates, you will need to handle this time dependence (see “Time dependence of joins”).
Subscribing to a stream of changes, rather than querying the current state when needed, brings us closer to a spreadsheet-like model of computation. When a piece of data changes, any derived data that depends on it can swiftly be updated. Many open questions still remain—for example, around issues like time-dependent joins—but building applications around dataflow ideas is a promising direction to explore.