Over the course of these notes we have discussed various features provided by databases and how they work, including these:
-
Secondary indexes, which allow you to efficiently search for records based on the value of a field
-
Materialized views, which are a kind of precomputed cache of query results
-
Replication logs, which keep copies of the data on other nodes up-to-date
-
Full-text search indexes, which allow keyword search in text and which are built into some relational databases
In the related discussion, similar themes emerged; we talked about building full-text search indexes, about materialized view maintenance, and about replicating changes from a database to derived data systems by using CDC.
It seems that there are parallels between the features that are built into databases and the derived data systems that people are building with batch and stream processors.
Creating an index
Think about what happens when you run CREATE INDEX to create a new index in a relational database. The database has to scan over a consistent snapshot of a table, pick out all the field values being indexed, sort them, and write out the index. Then it must process the backlog of writes that have been made since the consistent snapshot was taken (assuming the table was not locked while creating the index, so writes could continue). Once that is done, the database must continue to keep the index up-to-date whenever a transaction writes to the table.
This process is remarkably similar to setting up a new follower replica (see “Setting Up New Followers”), and also very similar to bootstrapping CDC in a streaming system (see “Initial snapshot”).
Whenever you run CREATE INDEX, the database essentially reprocesses the existing dataset and derives the index as a new view onto the existing data. The existing data may be a snapshot of the state rather than a log of all changes that ever happened, but the two are closely related.
The meta-database of everything
In this light, the dataflow across an entire organization starts looking like one huge database. Whenever a batch, stream, or ETL process transports data from one place and form to another place and form, it is acting like the database subsystem that keeps indexes or materialized views up-to-date.
Viewed like this, batch and stream processors are like elaborate implementations of triggers, stored procedures, and materialized view maintenance algorithms. The derived data systems they maintain are like different index types. For example, a relational database may support B-tree indexes, hash indexes, spatial indexes, and other types of indexes. In the emerging architecture of derived data systems, instead of implementing those facilities as features of a single integrated database product, they are provided by various pieces of software, running on different machines, administered by different teams.
Where will these developments take us in the future? If we start from the premise that no single data model or storage format is suitable for all access patterns, there are two avenues by which different storage and processing tools can nevertheless be composed into a cohesive system:
Federated databases (unifying reads)
It is possible to provide a unified query interface to a wide variety of underlying storage engines and processing methods—an approach known as a federated database or polystore. For example, PostgreSQL’s foreign data wrapper feature fits this pattern, as do federated query engines such as Trino, Hoptimator, and Xorq. Applications that need a specialized data model or query interface can still access the underlying storage engines directly, while users who want to combine data from disparate places can do so easily through the federated interface.
A federated query interface follows the relational tradition of a single integrated system with a high-level query language and elegant semantics, but a complicated implementation.
Unbundled databases (unifying writes)
While federation addresses read-only querying across several systems, it does not have a good answer to synchronizing writes across those systems. We said that within a single database, creating a consistent index is a built-in feature. When we compose several storage systems, we similarly need to ensure that all data changes end up in all the right places, even in the face of faults. Making it easier to reliably plug together storage systems (e.g., through CDC and event logs) is like unbundling a database’s index maintenance features in a way that can synchronize writes across disparate technologies.
The unbundled approach follows the Unix tradition of small tools that do one thing well, that communicate through a uniform low-level API (pipes), and that can be composed using a higher-level language (the shell).
Making unbundling work
Federation and unbundling are two sides of the same coin: composing a reliable, scalable, and maintainable system out of diverse components. Federated read-only querying requires mapping one data model into another, which takes some thought but is ultimately quite a manageable problem. Keeping the writes to several storage systems in sync is the harder engineering problem, so we will focus on that here.
The traditional approach to synchronizing writes requires distributed transactions across heterogeneous storage systems, which are problematic, as discussed previously. Transactions within a single storage or stream processing system are feasible, but when data crosses the boundary between different technologies, an asynchronous event log with idempotent writes is a much more robust and practicable approach.
For example, distributed transactions are used within some stream processors to achieve exactly-once semantics, and this can work quite well. However, when a transaction would need to involve systems written by different groups of people (e.g., when data is written from a stream processor to a distributed key-value store or search index), the lack of a standardized transaction protocol makes integration much harder. An ordered log of events with idempotent consumers is a much simpler abstraction and thus is much more feasible to implement across heterogeneous systems.
The big advantage of log-based integration is loose coupling between the various components, which manifests itself in two ways:
-
At a system level, asynchronous event streams make the system as a whole more robust to outages or performance degradation of individual components. If a consumer runs slow or fails, the event log can buffer messages, allowing the producer and any other consumers to continue running unaffected. The faulty consumer can catch up when it is fixed, so it doesn’t miss any data, and the fault is contained. By contrast, the synchronous interaction of distributed transactions tends to escalate local faults into large-scale failures.
-
At a human level, unbundling data systems allows software components and services to be developed, improved, and maintained independently from one another by different teams. Specialization allows each team to focus on doing one thing well, with well-defined interfaces to other teams’ systems. Event logs provide an interface that is powerful enough to capture fairly strong consistency properties (because of durability and ordering of events), but also general enough to be applicable to almost any kind of data.
Unbundled versus integrated systems
If unbundling does indeed become the way of the future, it will not replace databases in their current form. They will still be needed as much as ever, for maintaining state in stream processors and to serve queries for the output of batch and stream processors. Specialized query engines will also continue to be important for particular workloads—for example, query engines in data warehouses are optimized for exploratory analytical queries and handle this kind of workload very well.
The complexity of running several pieces of infrastructure can be a problem. Each piece of software has a learning curve and its own configuration issues and operational quirks, so it is worth deploying as few moving parts as possible. A single integrated software product may also be able to achieve better and more predictable performance on the kinds of workloads for which it is designed, compared to a system consisting of several tools that you have composed with application code. Building for scale that you don’t need is wasted effort and may lock you into an inflexible design. In effect, it is a form of premature optimization.
The goal of unbundling is not to compete with individual databases on performance for particular workloads; the goal is to allow you to combine several databases in order to achieve good performance for a much wider range of workloads than is possible with a single piece of software. It’s about breadth, not depth.
Thus, if a single technology does everything you need, you’re most likely best off simply using that product rather than trying to reimplement it yourself from lower-level components. The advantages of unbundling and composition come into the picture only when no single piece of software satisfies all your requirements.
The tools for composing data systems are getting better. Debezium can extract change streams from many databases, Kafka’s protocol is becoming a de facto standard for event streams, and incremental view maintenance engines (see “Incremental View Maintenance”) make it possible to precompute and update caches of complex queries.