In a database, the process that does the writing encodes the data, and the process that does the reading decodes it. Just one process may be accessing the database, in which case the reader is simply a later version of the same process; in such a scenario, you can think of storing something in the database as sending a message to your future self. Backward compatibility is clearly necessary here, as otherwise your future self won’t be able to decode what you previously wrote.
In general, though, it’s common for several processes to be accessing a database at the same time. Those processes might be different applications or services, or they may simply be multiple instances of the same service (running in parallel for scalability or fault tolerance). Either way, in such an environment, it is likely that some processes accessing the database will be running newer code and some will be running older code—for example, because a new version is currently being deployed in a rolling upgrade, so some instances have been updated while others haven’t yet.
This means that a value in the database may be written by a newer version of the code and subsequently read by an older version of the code that is still running. Thus, forward compatibility is also often required for databases.
Different values written at different times
A database generally allows any value to be updated at any time. Within a single database, you may have some values that were written five milliseconds ago and others that were written five years ago.
When you deploy a new version of your application (of a server-side application, at least), you may entirely replace the old version with the new version within a few minutes. The same is not true of database contents; the five-year-old data will still be there, in the original encoding, unless you have explicitly rewritten it since then. This observation is sometimes summed up as data outlives code.
Although rewriting (migrating) data into a new schema is certainly possible, it’s expensive on a large dataset. Therefore, most databases defer the operation, performing it asynchronously and on a best-effort basis. For example, LSM-tree storage engines (see “Log-Structured Storage”) will rewrite data using the latest format during compaction. Most relational databases also allow simple schema changes, such as adding a new column with a null default value, without rewriting existing data. When an old row is read, the database fills in nulls for any columns that are missing from the encoded data on disk. Schema evolution thus allows the entire database to appear as if it was encoded with a single schema, even though the underlying storage may contain records encoded with various historical versions of the schema.
More complex schema changes—for example, changing a single-valued attribute to be multivalued, or moving some data into a separate table—still require data to be rewritten, often at the application level. Maintaining forward and backward compatibility across such migrations remains a research problem.
Archival storage
Perhaps you take a snapshot of your database from time to time—say, for backup purposes or for loading into a data warehouse (see “Data Warehousing”). In this case, the data dump will typically be encoded using the latest schema, even if the original encoding in the source database contained a mixture of schema versions from different eras. Since you’re copying the data anyway, you might as well encode the copy of the data consistently.
As the data dump is written in one go and is thereafter immutable, formats like Avro object container files are a good fit. This is also a good opportunity to encode the data in an analytics-friendly column-oriented format such as Parquet (see “Column compression”).
In the related discussion we will talk more about using data in archival storage.