Apache Avro is another binary encoding format, with some interesting differences from Protocol Buffers. It was started in 2009 as a subproject of Hadoop, as a result of Protocol Buffers not being a good fit for Hadoop’s use cases.
Avro also uses a schema to specify the structure of the data being encoded. It has two schema languages: one (Avro IDL) intended for human editing, and one (based on JSON) that is more easily machine-readable. As with Protocol Buffers, the schema languages specify only fields and their types and do not support complex validation rules like those in JSON Schema.
Written in Avro IDL, our example schema might look like this:
record Person {
string userName;
union { null, long } favoriteNumber = null;
array<string> interests;
}
The equivalent JSON representation of that schema is as follows:
{
"type": "record",
"name": "Person",
"fields": [
{"name": "userName", "type": "string"},
{"name": "favoriteNumber", "type": ["null", "long"], "default": null},
{"name": "interests", "type": {"type": "array", "items": "string"}}
]
}
Notice that the schema has no tag numbers. If we encode our record (Example 5-2) using this schema, the Avro binary encoding is just 32 bytes long—the most compact of all the encodings we have seen. The breakdown of the encoded byte sequence is shown in Figure 5-4.
If you examine the byte sequence, you can see that nothing identifies fields or their datatypes. The encoding simply consists of values concatenated together. A string is just a length prefix followed by UTF-8 bytes, but nothing in the encoded data tells you that it is a string. It could just as well be an integer or something else entirely. An integer is encoded using a variable-length encoding.
To parse the binary data, you go through the fields in the order that they appear in the schema and use the schema to determine the datatype of each field. This means that the binary data can be decoded correctly only if the code reading the data is using the exact same schema as the code that wrote the data. Any mismatch in the schema between the reader and the writer would mean incorrectly decoded data.
Diagram demonstrating how a byte sequence is encoded and decoded in Avro, showing a breakdown of strings, integers, and array elements according to their variable-length encoding format.
Figure 5-4. Our record encoded using Avro
So, how does Avro support schema evolution?
The writer’s schema and the reader’s schema
When an application wants to encode some data (to write it to a file or database, send it over the network, etc.), the application uses whatever version of the schema it knows about—for example, a schema that is compiled into the application. This is known as the writer’s schema.
To decode some data (read it from a file or database, receive it from the network, etc.), an application uses two schemas: the writer’s schema, which is identical to the one used for encoding, and the reader’s schema, which may be different. This is illustrated in Figure 5-5. The reader’s schema defines the fields of each record that the application code is expecting, and their types.
If the reader’s and writer’s schemas are the same, decoding is easy. If they are different, Avro resolves the differences by comparing the two and translating the data from the writer’s schema into the reader’s schema.
Diagram comparing Protocol Buffers and Avro encoding and decoding processes, highlighting Avro's use of both writer's and reader's schemas for decoding.
Figure 5-5. In Protocol Buffers, encoding and decoding can use different versions of a schema. In Avro, decoding uses two schemas: the writer’s schema must be identical to the one used for encoding, but the reader’s schema can be an older or newer version.
The Avro specification defines exactly how this resolution works. As illustrated in Figure 5-6, it’s no problem if the writer’s schema and the reader’s schema have their fields in a different order, because the schema resolution matches up the fields by field name. If the code reading the data encounters a field that appears in the writer’s schema but not in the reader’s schema, it is ignored. If the code reading the data expects a certain field but the writer’s schema does not contain a field of that name, it is filled in with a default value declared in the reader’s schema.
Diagram showing the resolution of differences between a writer's schema and a reader's schema for a Person record, highlighting matched and unmatched fields.
Figure 5-6. An Avro reader resolving differences between the writer’s schema and the reader’s schema
Schema evolution rules
With Avro, forward compatibility means the writer can use a newer version of the schema than the reader. Conversely, backward compatibility means the writer can use an older version of the schema than the reader.
To maintain compatibility, you may add or remove only a field that has a default value (like the field favoriteNumber in our Avro schema.) For example, say you add a field with a default value, so this new field exists in the new schema but not the old one. When a reader using the new schema reads a record written with the old schema, the default value is filled in for the missing field.
If you were to add a field that has no default value, new readers wouldn’t be able to read data written by old writers, so you would break backward compatibility. If you were to remove a field that has no default value, old readers wouldn’t be able to read data written by new writers, so you would break forward compatibility.
In some programming languages, null is an acceptable default for any variable, but this is not the case in Avro: if you want to allow a field to be null, you have to use a union type. For example, union { null, long, string } field; indicates that field can be a number, a string, or null. You can use null as a default value only if it is the first branch of the union. This is a little more verbose than having everything nullable by default, but it helps prevent bugs by being explicit about what can and cannot be null.
Changing the datatype of a field is possible, provided that Avro can convert the type. Changing the name of a field is also possible but a little tricky. The reader’s schema can contain aliases for field names, so it can match an old writer’s schema field names against the aliases. This means that changing a field name is backward compatible but not forward compatible. Similarly, adding a branch to a union type is backward compatible but not forward compatible.
But what is the writer’s schema?
We’ve glossed over an important question: how does the reader know the schema that was used to encode a particular piece of data? We can’t just include the entire schema with every record, because the schema would likely be much bigger than the encoded data, negating all the space savings from the binary encoding.
The answer depends on the context in which Avro is being used. To give a few examples:
Large file with lots of records
A common use for Avro is storing a large file containing millions of records, all encoded with the same schema. In this case, the writer of that file can just include the schema once at the beginning of the file. Avro specifies a file format (object container files) to do this.
Database with individually written records
In a database, different records may be written at different points in time using different schemas—you cannot assume that all the records will have the same schema. The simplest solution in this case is to include a version number at the beginning of every encoded record and keep a list of schema versions in your database. A reader can fetch a record, extract the version number, and then fetch the writer’s schema corresponding to that version number from the database. It can then decode the rest of the record by using that schema. Confluent’s schema registry for Apache Kafka and LinkedIn’s Espresso work this way, for example.
Sending records over a network connection
When two processes are communicating over a bidirectional network connection, they can negotiate the schema version on connection setup and then use that schema for the lifetime of the connection. The Avro RPC protocol (see “Dataflow Through Services: REST and RPC”) works like this.
A database of schema versions is useful to have in any case, since it acts as documentation and gives you a chance to check schema compatibility. You can use a simple incrementing integer or a hash of the schema as the version number.
Dynamically generated schemas
One advantage of Avro’s approach, compared to Protocol Buffers, is that the schema doesn’t contain any tag numbers. But why is this important? What’s the problem with keeping a couple of numbers in the schema?
The difference is that Avro is friendlier to dynamically generated schemas. For example, say you have a relational database whose contents you want to dump to a file, and you want to use a binary format to avoid the aforementioned problems with textual formats (JSON, CSV, XML). If you use Avro, you can fairly easily generate an Avro schema (in the JSON representation we saw earlier) from the relational schema and encode the database contents using that schema, dumping it all to an Avro object container file. You can generate a record schema for each database table, and each column becomes a field in that record. The column name in the database maps to the field name in Avro.
Now, if the database schema changes (e.g., if a table has one column added and one column removed), you can just generate a new Avro schema from the updated database schema and export data in the new Avro schema. The data export process does not need to pay any attention to the schema change—it can simply do the schema conversion every time it runs. Anyone who reads the new data files will see that the fields of the record have changed, but since the fields are identified by name, the updated writer’s schema can still be matched up with the old reader’s schema.
By contrast, if you were using Protocol Buffers for this purpose, the field tags would likely have to be assigned by hand. Every time the database schema changed, an administrator would have to manually update the mapping from database column names to field tags. (It might be possible to automate this, but the schema generator would have to be very careful to not assign previously used field tags.) This kind of dynamically generated schema simply wasn’t a design goal of Protocol Buffers, whereas it was for Avro.