In all the data models we have discussed so far, the data is queried in the same form as it is written—be it JSON documents, rows in tables, or vertices and edges in a graph. However, in complex applications it can sometimes be difficult to find a single data representation that is able to satisfy all the ways that the data needs to be queried and presented. In such situations, it can be beneficial to write data in one form and then derive from it representations that are optimized for different types of reads.
We previously saw this idea in “Systems of Record and Derived Data”, and ETL (see “Data Warehousing”) is one example of such a derivation process. Now we will take the idea further. If we are going to derive one data representation from another anyway, we can choose different representations that are optimized for writing and reading, respectively. How would you model your data if you wanted to optimize it for only writing, and if efficient queries were of no concern?
Perhaps the simplest, fastest, and most expressive way of writing data is an event log: every time you want to write some data, you encode it as a self-contained string (perhaps as JSON), including a timestamp, and then append it to a sequence of events. Events in this log are immutable; you never change or delete them, but only ever append more events to the log (which may supersede earlier events). An event can contain arbitrary properties.
Figure 3-8 shows an example that could be taken from a conference management system. A conference can be a complex business domain: not only can individual attendees register and pay by card, but companies can also order seats in bulk, pay by invoice, and then later assign the seats to individual people. A certain number of seats may be reserved for speakers, sponsors, volunteer helpers, and so on. Reservations may also be canceled, and the conference organizer might change the capacity of the event by moving it to a different room. With all this going on, simply calculating the number of available seats becomes a challenging query.
Diagram illustrating an event log for a conference management system, showing how events like registration and booking generate multiple materialized views, such as customer booking confirmations and conference organizer dashboards.
Figure 3-8. Using a log of immutable events as the source of truth and deriving materialized views from it
In Figure 3-8, every change to the state of the conference (such as the organizer opening registrations, or attendees making and canceling registrations) is first stored as an event. Whenever an event is appended to the log, several materialized views (also known as projections or read models) are also updated to reflect the effect of that event. In the conference example, there might be one materialized view that collects all information related to the status of each booking, another that computes charts for the conference organizer’s dashboard, and a third that generates files for the printer that produces the attendees’ badges.
The idea of using events as the source of truth and expressing every state change as an event is known as event sourcing. The principle of maintaining separate read-optimized representations and deriving them from the write-optimized representation is called command query responsibility segregation (CQRS). These terms originated in the DDD community, although similar ideas have been around for a long time—e.g., in state machine replication (see “Using shared logs”).
When a request from a user comes in, it is called a command, and it first needs to be validated. Once the command has been executed and has been determined to be valid (e.g., there were enough available seats for a requested reservation), it becomes a fact, and the corresponding event is added to the log. Consequently, the event log should contain only valid events, and a consumer of the event log that builds a materialized view is not allowed to reject an event.
When modeling your data in an event sourcing style, it is recommended that you name your events in the past tense (e.g., “the seats were booked”), because an event is a record of the fact that something has happened. Even if the user later decides to change or cancel their reservation, the fact remains true that they formerly held a booking, and the change or cancellation is a separate event that is added later.
A similarity between event sourcing and a star schema fact table, discussed in “Stars and Snowflakes: Schemas for Analytics”, is that both are collections of events that happened in the past. However, rows in a fact table all have the same set of columns, whereas in event sourcing there may be many event types, each with different properties. In addition, a fact table is an unordered collection, while in event sourcing the order of events is important: if a booking is first made and then canceled, processing those events in the wrong order would not make sense.
Event sourcing and CQRS have several advantages:
-
For the people developing the system, events better communicate the intent of why something happened. For example, it’s easier to understand the event “the booking was canceled” than “the active column on row 4001 of the bookings table was set to false, three rows associated with that booking were deleted from the seat_assignments table, and a row representing the refund was inserted into the payments table.” Those row modifications may still happen when a materialized view processes the cancellation event, but when they are driven by an event, the reason for the updates becomes much clearer.
-
A key principle of event sourcing is that the materialized views are derived from the event log in a reproducible way. You should always be able to delete the materialized views and recompute them by processing the same events in the same order, using the same code. If there was a bug in the view maintenance code, you can just delete the view and recompute it with the new code. It’s also easier to find the bug because you can rerun the view maintenance code as often as you like and inspect its behavior.
-
You can have multiple materialized views that are optimized for the particular queries that your application requires. These views can be stored in either the same database as the events or a different one, depending on your needs. They can use any data model, and they can be denormalized for fast reads. You can even keep a view only in memory and avoid persisting it, as long as it’s OK to recompute the view from the event log whenever the service restarts.
-
If you decide you want to present the existing information in a new way, building a new materialized view from the existing event log is easy. You can also evolve the system to support new features by adding new types of events or adding new properties to existing event types (any older events remain unmodified). You can also chain new behaviors off existing events (e.g., when a conference attendee cancels, their seat could be offered to the next person on the waiting list).
-
If an event was written in error, you can write a subsequent deletion event to reverse it. Downstream views will incorporate this deletion automatically, thereby correcting the data. On the other hand, in a database where you update and delete data directly, a committed transaction is often difficult to reverse. Event sourcing can therefore reduce the number of irreversible actions in the system, making it easier to change (see “Evolvability: Making Change Easy”).
-
The event log can also serve as an audit log of what has occurred in the system, which is valuable in regulated industries that require such auditability.
-
Event logs can typically handle higher write throughput than databases because of their sequential access patterns. If you have a temporary burst of events, the log can absorb it, and downstream systems that maintain materialized views can catch up at their own pace without becoming overwhelmed.
However, event sourcing and CQRS also have downsides:
-
You need to be careful if external information is involved. For example, say an event contains a price given in one currency, and for one of the views it needs to be converted into another currency. Since the exchange rate may fluctuate, it would be problematic to fetch the exchange rate from an external source when processing the event, since you would get a different result if you recomputed the materialized view on another date. To make the event processing logic deterministic, you must either include the exchange rate in the event itself or have a way of querying the historical exchange rate at the timestamp indicated in the event, ensuring that this query always returns the same result for the same timestamp.
-
The requirement that events are immutable creates problems if events contain personal data from users, since users may exercise their right (e.g., under the GDPR) to request deletion of their data. If the event log is on a per-user basis, you can just delete the whole log for that user, but that doesn’t work if your event log contains events relating to multiple users. You can try storing the personal data outside of the actual event, or encrypting it with a key that you can later choose to delete (a technique known as crypto-shredding), but that also makes it harder to recompute derived state when needed.
-
Reprocessing events requires care if there are externally visible side effects—for example, you probably don’t want to resend confirmation emails every time you rebuild a materialized view.
You can implement event sourcing on top of any database, but some systems are specifically designed to support this pattern, such as EventStoreDB, MartenDB (based on PostgreSQL), and Axon Framework. You can also use message brokers such as Apache Kafka to store the event log, and stream processors can keep the materialized views up-to-date; we will return to these topics in the related discussion.
The only important requirement is that the event storage system must guarantee that all materialized views process the events in exactly the same order as they appear in the log. this is not always easy to achieve in a distributed system.