When moving to standardized encodings that can be written and read by many programming languages, JSON and XML are the obvious contenders: they are widely known and widely supported. CSV is another popular language-independent format, but it supports only tabular data without nesting.
JSON, XML, and CSV are textual formats, and thus they are somewhat human-readable although the syntax is a common topic of debate. Besides the superficial syntactic issues, they also have various other problems:
-
XML is often criticized for being too verbose and unnecessarily complicated.
-
There is a lot of ambiguity around the encoding of numbers. In XML and CSV, you cannot distinguish between a number and a string that happens to consist of digits (except by referring to an external schema). JSON distinguishes strings and numbers, but it doesn’t distinguish integers and floating-point numbers, and it doesn’t specify a precision.
This is a problem when dealing with large numbers—for example, integers greater than 253 cannot be exactly represented in an IEEE 754 double-precision floating-point number, so such numbers become inaccurate when parsed in a language that uses floating-point numbers, such as JavaScript. An example of numbers larger than 253 occurs on X, which uses a 64-bit number to identify each post. The JSON returned by the API includes post IDs twice, once as a JSON number and once as a decimal string, to work around the incorrect parsing of numbers by JavaScript applications.
-
JSON and XML have good support for Unicode character strings (i.e., human-readable text), but they don’t support binary strings (sequences of bytes without a character encoding). Binary strings are a useful feature, so people get around this limitation by encoding the binary data as text using Base64. The schema is then used to indicate that the value should be interpreted as Base64 encoded. This works, but it’s somewhat hacky and increases the data size by about a third.
-
XML Schema and JSON Schema are powerful and thus quite complicated to learn and implement. Since the correct interpretation of data (such as numbers and binary strings) depends on information in the schema, applications that don’t use XML/JSON Schemas may need to hardcode the appropriate encoding/decoding logic instead.
-
CSV does not have any schema, so it is up to the application to define the meaning of each row and column. If an application change adds a new row or column, you have to handle that change manually. CSV is also a quite vague format (what happens if a value contains a comma or a newline character?). Although its escaping rules have been formally specified, not all parsers implement them correctly.
Despite these flaws, JSON, XML, and CSV are good enough for many purposes. They will likely remain popular, especially as data interchange formats (i.e., for sending data from one organization to another). In these situations, as long as people agree on the format, it often doesn’t matter how pretty or efficient it is. The difficulty of getting different organizations to agree on anything outweighs most other concerns.
JSON Schema
JSON Schema has become widely adopted as a way to model data whenever it’s exchanged between systems or written to storage. You’ll find JSON Schemas in web services (see “Web services”) as part of the OpenAPI web service specification, in schema registries such as Confluent’s Schema Registry and Red Hat’s Apicurio Registry, and in databases (e.g., PostgreSQL’s pg_jsonschema validator extension and MongoDB’s $jsonSchema validator syntax).
The JSON Schema specification offers a number of features. Schemas include standard primitive types such as string, number, integer, object, array, boolean, and null. But JSON Schema also offers a separate validation specification that allows developers to overlay constraints on fields. For example, a port field might have a minimum value of 1 and a maximum of 65,535.
JSON Schemas can have either open or closed content models. An open content model permits any field not defined in the schema to exist with any datatype, whereas a closed content model allows only fields that are explicitly defined. The open content model in JSON Schema is enabled when additionalProperties is set to true, which is the default. Thus, JSON Schemas are usually a definition of what isn’t permitted (namely, invalid values on any of the defined fields) rather than what is permitted.
Open content models are powerful, but they can be complex. For example, say you want to define a map from integers (such as IDs) to strings. JSON does not have a map or dictionary type that allows integer keys; JSON objects always use strings as keys. To accommodate your needs, you can constrain this type with JSON Schema so that keys can contain only digits and values can be only strings using patternProperties and additionalProperties, as shown in Example 5-1.
Example 5-1. A JSON Schema with integer keys and string values
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"patternProperties": {
"^[0-9]+$": {
"type": "string"
}
},
"additionalProperties": false
}
In addition to open and closed content models and validators, JSON Schema supports conditional if/else schema logic, named types, references to remote schemas, and much more. All of this makes for a very powerful schema language. Such features also make for unwieldy definitions. It can be challenging to resolve remote schemas, reason about conditional rules, or evolve schemas in a forward- or backward-compatible way. Similar concerns apply to XML Schema.
Binary encodings
JSON is less verbose than XML, but both still use a lot of space compared to binary formats. This observation led to the development of a profusion of binary encodings for JSON (MessagePack, CBOR, BSON, BJSON, UBJSON, BISON, Hessian, and Smile, to name a few) and XML (WBXML and Fast Infoset, for example). These formats have been adopted in various niches, as they are more compact and sometimes faster to parse, but none of them are as widely adopted as the textual versions of JSON and XML.
Some of these formats extend the set of datatypes (e.g., distinguishing integers and floating-point numbers, or adding support for binary strings), but otherwise they keep the JSON/XML data model unchanged. In particular, since they don’t prescribe a schema, they need to include all the object field names within the encoded data. That is, in a binary encoding of the JSON document in Example 5-2, they will need to include the strings userName, favoriteNumber, and interests somewhere.
Example 5-2. A record that we will encode in several binary formats in this topic
{
"userName": "Martin",
"favoriteNumber": 1337,
"interests": ["daydreaming", "hacking"]
}
Let’s look at an example of MessagePack, a binary encoding for JSON. Figure 5-2 shows the byte sequence that you get if you encode the JSON document in Example 5-2 with MessagePack.
A MessagePack encoding diagram showing a record compressed into 66 bytes, detailing the object structure with strings and data entries such as userName, favoriteNumber, and interests.
Figure 5-2. Our record (Example 5-2) encoded using MessagePack
The first few bytes are as follows:
-
The first byte, 0x83, indicates that what follows is an object (most significant four bits = 0x80) with three fields (least significant four bits = 0x03). (In case you’re wondering what happens if an object has more than 15 fields, so that the number of fields doesn’t fit in four bits, it then gets a different type indicator, and the number of fields is encoded in two or four bytes.)
-
The second byte, 0xa8, indicates that what follows is a string (most significant four bits = 0xa0) that is eight bytes long (least significant four bits = 0x08).
-
The next eight bytes are the field name userName in ASCII. Since the length was indicated previously, there’s no need for any marker to tell us where the string ends (or any escaping).
-
The next seven bytes encode the six-letter string value Martin with the prefix 0xa6, and so on.
The binary encoding is 66 bytes long, which is only a little less than the 81 bytes taken by the textual JSON encoding (with whitespace removed). All the binary encodings of JSON are similar in this regard. It’s not clear whether such a small space reduction (and perhaps a speedup in parsing) is worth the loss of human-readability.
In the following sections we will see how we can do much better, encoding the same record in half as many bytes.