We saw in “Exactly-once message processing” that an important use case for distributed transactions is to ensure that an operation takes effect exactly once, even if a crash occurs while it is being processed and the processing needs to be retried. If you can atomically commit a transaction across a message broker and a database, you can acknowledge the message to the broker if and only if it was successfully processed and the database writes resulting from the process were committed.
However, you don’t actually need distributed transactions to achieve exactly-once semantics. An alternative approach is as follows, which requires only transactions within the database:
-
Assume every message has a unique ID, and in the database you have a table of message IDs that have been processed. When you start processing a message from the broker, you begin a new transaction on the database and check the message ID. If the same message ID is already present in the database, you know that it has already been processed, so you can acknowledge the message to the broker and drop it.
-
If the message ID is not already in the database, you add it to the table. You then process the message, which may result in additional writes to the database within the same transaction. When you finish processing the message, you commit the transaction on the database.
-
Once the database transaction is successfully committed, you can acknowledge the message to the broker.
-
Once the message has successfully been acknowledged to the broker, you know that it won’t try processing the same message again, so you can delete the message ID from the database (in a separate transaction).
If the message processor crashes before committing the database transaction, the transaction is aborted and the message broker will retry processing. If it crashes after committing but before acknowledging the message to the broker, it will also retry processing, but the retry will see the message ID in the database and drop it. If it crashes after acknowledging the message but before deleting the message ID from the database, you will have an old message ID lying around, which doesn’t do any harm besides taking up a little bit of storage space. If a retry happens before the database transaction is aborted (which could happen if communication between the message processor and the database is interrupted), a uniqueness constraint on the table of message IDs should prevent the same message ID from being inserted by two concurrent transactions.
Thus, achieving exactly-once processing requires only transactions within the database—atomicity across database and message broker is not necessary for this use case. Recording the message ID in the database makes the message processing idempotent, so that message processing can be safely retried without duplicating its side effects. A similar approach is used in stream processing frameworks such as Kafka Streams to achieve exactly-once semantics,.
That said, internal distributed transactions within the database are still useful for the scalability of patterns such as these; for example, they would allow the message IDs to be stored on one shard and the main data updated by the message processing to be stored on other shards, and ensure atomicity of the transaction commit across those shards.