A convenient property of many transactional systems is that as soon as one transaction commits, its writes are immediately visible to other transactions. This property is formalized as strict serializability (discussed in “Linearizability Versus Serializability”).
This is not the case when unbundling an operation across multiple stages of stream processors. Consumers of a log are asynchronous by design, so a producer does not wait until its message has been processed by consumers. However, it is possible for a client to wait for a message to appear on an output stream, like the user waiting for an outgoing payment or payment declined event in Figure 13-2, which depends on whether there was enough money in the source account.
In this example, the correctness of the source account balance check does not depend on whether the user making the request waits for the outcome. The waiting has the purpose of only synchronously informing the user whether the payment succeeded; this notification is decoupled from the effects of processing the request.
More generally, the term consistency conflates two requirements that are worth considering separately:
Timeliness
Timeliness means ensuring that users observe the system in an up-to-date state. We saw previously that if a user reads from a stale copy of the data, they may observe it in an inconsistent state (see “Problems with Replication Lag”). However, that inconsistency is temporary, and it will eventually be resolved simply by waiting and trying again.
The CAP theorem uses “consistency” in the sense of linearizability, which is a strong way of achieving timeliness. Weaker timeliness properties, like read-after-write consistency, can also be useful.
Integrity
Integrity means absence of corruption—no data loss, and no contradictory or false data. In particular, if a derived dataset is maintained as a view onto underlying data, the derivation must be correct. For example, a database index must correctly reflect the contents of the database—an index with missing records is not very useful.
If integrity is violated, the inconsistency is permanent; waiting and trying again is not going to fix database corruption in most cases. Instead, explicit checking and repair is needed. In the context of ACID transactions, “consistency” is usually understood as some kind of application-specific notion of integrity. Atomicity and durability are important tools for preserving integrity.
In slogan form: violations of timeliness are allowed under eventual consistency, whereas violations of integrity result in perpetual inconsistency.
In most applications, integrity is much more important than timeliness. Violations of timeliness can be annoying and confusing, but violations of integrity can be catastrophic.
For example, on your credit card statement, it is not surprising if a transaction that you made within the last 24 hours does not yet appear. It is normal that these systems have a certain lag. We know that banks reconcile and settle transactions asynchronously, and timeliness is not very important here. However, it would be very bad if the statement balance was not equal to the sum of the transactions plus the previous statement balance (an error in the sums), or if a transaction was charged to you but not paid to the merchant (disappearing money). Such problems would be violations of the integrity of the system.
Correctness of dataflow systems
ACID transactions usually provide both timeliness (e.g., linearizability) and integrity (e.g., atomic commit) guarantees. Thus, if you approach application correctness from the point of view of ACID transactions, the distinction between timeliness and integrity is fairly inconsequential.
On the other hand, an interesting property of the event-based dataflow systems that we have discussed in this topic is that they decouple timeliness and integrity. When processing event streams asynchronously, there is no guarantee of timeliness, unless you explicitly build consumers that wait for a message to arrive before returning. For example, a user could request a payment and then read the state of their account before the stream processor has executed the request; the user will not see the payment they just requested.
However, integrity is in fact central to streaming systems. As we’ve seen, exactly-once or effectively-once semantics is a mechanism for preserving integrity. If an event is lost or takes effect twice, the integrity of a data system could be violated. Thus, fault-tolerant message delivery and duplicate suppression (e.g., idempotent operations) are important for maintaining the integrity of a data system in the face of faults.
As we saw in the previous section, reliable stream processing systems can preserve integrity without requiring distributed transactions and an atomic commit protocol, which means they can potentially achieve comparable correctness with much better performance and operational robustness. We achieved this integrity through a combination of mechanisms:
-
Representing the content of the write operation as a single message, which can easily be written atomically—an approach that fits very well with event sourcing
-
Deriving all other state updates from that single message via deterministic derivation functions, similarly to stored procedures
-
Passing a client-generated request ID through all these levels of processing, enabling end-to-end duplicate suppression and idempotence
-
Making messages immutable and allowing derived data to be reprocessed from time to time, which makes it easier to recover from bugs
Loosely interpreted constraints
As discussed previously, enforcing a uniqueness constraint requires consensus, typically implemented by funneling all events in a particular shard through a single node. This limitation is unavoidable if we want the traditional form of uniqueness constraint, and stream processing cannot get around it.
However, many real applications have a business requirement to allow violations of what you might think of as hard constraints:
-
If customers order more items than you have in your warehouse, you can order in more stock, apologize for the delay, and offer them a discount. This is the same as what you’d have to do if, say, a forklift truck ran over some of the items in your warehouse, leaving you with fewer items in stock than you thought you had. Thus, the apology workflow already needs to be part of your business processes anyway in order to deal with such incidents, and a hard constraint on the number of items in stock might be unnecessary.
-
Similarly, many airlines overbook airplanes with the expectation that some passengers will miss their flight, and many hotels overbook rooms, expecting that some guests will cancel. In these cases, the constraint of “one person per seat” is deliberately violated for business reasons, and compensation processes (refunds, upgrades, providing a complimentary room at a neighboring hotel) are put in place to handle demand that exceeds supply. Even if no overbooking occurred, apology and compensation processes would be needed to deal with events such as flights being canceled because of bad weather or staff going on strike—recovering from such issues is just a normal part of business.
-
If someone withdraws more money than they have in their account, the bank can charge them an overdraft fee and ask them to pay back what they owe. By limiting the total withdrawals per day, the risk to the bank is bounded.
-
In systems that integrate data across organizations, inconsistencies will inevitably arise, and correction mechanisms are necessary to handle them. As noted in “Batch Use Cases”, settlement of payments between banks is an example of this.
In many business contexts, it is therefore acceptable to temporarily violate a constraint and fix it up later by apologizing. This kind of change to correct a mistake is called a compensating transaction. The cost of the apology (in terms of money or reputation) varies, but it is often quite low; you can’t unsend an email, but you can send a follow-up email with a correction. If you accidentally charge a credit card twice, you can refund one of the charges, and the cost to you is just the processing fees and perhaps a customer complaint. Once money has been paid out of an ATM, you can’t directly get it back, although in principle you can send debt collectors to recover the money if the account was overdrawn and the customer won’t return it.
Whether the cost of the apology is acceptable is a business decision. If it is acceptable, the traditional model of checking all constraints before even writing the data is unnecessarily restrictive. It may well be reasonable to go ahead with a write optimistically and check the constraint after the fact. You can still ensure that the validation occurs before taking actions that would be expensive to recover from, but that doesn’t imply you must validate before you even write the data.
These applications do require integrity. You would not want to lose a reservation or have money disappear because of mismatched credits and debits. But they don’t require timeliness on the enforcement of the constraint. If you have sold more items than you have in the warehouse, you can patch up the problem after the fact. Doing so is similar to the conflict resolution approaches we discussed in “Dealing with Conflicting Writes”.
Coordination-avoiding data systems
We have now made two interesting observations:
-
Dataflow systems can maintain integrity guarantees on derived data without atomic commit, linearizability, or synchronous cross-shard coordination.
-
Although strict uniqueness constraints require timeliness and coordination, many applications are fine with loose constraints that may be temporarily violated and fixed up later, as long as integrity is preserved throughout.
Taken together, these observations mean that dataflow systems can provide data management services for many applications without requiring coordination, while still giving strong integrity guarantees. Such coordination-avoiding data systems have a lot of appeal; they can achieve better performance and fault tolerance than systems that need to perform synchronous coordination.
For example, such a system could operate distributed across multiple datacenters in a multi-leader configuration, asynchronously replicating between regions. Any one datacenter can continue operating independently from the others, because no synchronous cross-region coordination is required. Such a system would have weak timeliness guarantees—it could not be linearizable without introducing coordination—but it can still have strong integrity guarantees.
In this context, serializable transactions are still useful as part of maintaining derived state, but they can be run at a small scope where they work well. Heterogeneous distributed transactions such as XA transactions are not required. Synchronous coordination can still be introduced in places where it is needed (for example, to enforce strict constraints before an operation from which recovery is not possible), but there is no need for everything to pay the cost of coordination if only a small part of an application needs it.
Another way of looking at coordination and constraints is that they reduce the number of apologies you have to make for inconsistencies but potentially also reduce the performance and availability of your system, and thus potentially increase the number of apologies you have to make for outages. You cannot reduce the number of apologies to zero, but you can aim to find the best trade-off for your needs—the sweet spot with neither too many inconsistencies nor too many availability problems.