With stateless services that only read data, it’s not a big deal if something goes wrong; you can fix the bug and restart the service, and everything returns to normal. Stateful systems such as databases are not so simple. They are designed to remember things forever (more or less), so if something goes wrong, the effects also potentially last forever—which means they require more careful thought.
We want to build applications that are reliable and correct (i.e., programs whose semantics are well defined and understood, even in the face of various faults). For approximately four decades, the transaction properties of atomicity, isolation, and durability have been the tools of choice for building correct applications. However, those foundations are weaker than they seem: witness, for example, the confusion of weak isolation levels (see “Weak Isolation Levels”).
In some areas, transactions have been abandoned entirely and replaced with models that offer better performance and scalability but messier semantics. Consistency is often talked about but poorly defined. Some people assert that we should “embrace weak consistency” for the sake of better availability, while lacking a clear idea of what that means in practice.
For a topic that is so important, our understanding and our engineering methods are surprisingly flaky. For example, it is very difficult to determine whether it is safe to run a particular application using a particular transaction isolation level or replication configuration. Often, simple solutions appear to work correctly when concurrency is low and there are no faults, but turn out to have many subtle bugs in more demanding circumstances.
For example, Kyle Kingsbury’s Jepsen experiments have highlighted the stark discrepancies between some products’ claimed safety guarantees and their actual behavior in the presence of network problems and crashes. Even if infrastructure products like databases were free from problems, application code would still need to correctly use the features they provide, which is error-prone if the configuration is hard to understand (as is the case with weak isolation levels, quorum configurations, and so on).
If your application can tolerate occasionally corrupting or losing data in unpredictable ways, life is a lot simpler, and you might be able to get away with simply crossing your fingers and hoping for the best. If you need stronger assurances of correctness, serializability and atomic commit are established approaches, but they come at a cost. They typically work in only a single datacenter (ruling out geographically distributed architectures), and they limit the scale and fault-tolerance properties you can achieve.
While the traditional transaction approach is not going away, it is not the last word in making applications correct and resilient to faults. In this section we will explore other ways of thinking about correctness in the context of dataflow architectures.