Established data warehouse vendors such as Teradata, Vertica, and SAP HANA offer on-premises deployments under commercial licenses as well as cloud-based solutions. But as more customers have moved to the cloud, new cloud-only data warehouses such as Google Cloud’s BigQuery, Amazon Redshift, and Snowflake have also become widely adopted. Unlike traditional data warehouses, cloud data warehouses can take advantage of scalable cloud infrastructure such as object storage and serverless computation platforms.
Cloud data warehouses tend to integrate better with other cloud services. For example, many cloud warehouses support automatic log ingestion and offer easy integration with data processing frameworks such as Google Cloud’s Dataflow or AWS Kinesis. These warehouses are also more elastic because they decouple query computation from the storage layer. Data is persisted in object storage rather than on local disks, which makes it easy to adjust storage capacity and compute resources for queries independently, as we saw in “Cloud Native System Architecture”.
Open source data warehouses such as Apache Hive, Trino, and Apache Spark have also evolved with the cloud. As data storage for analytics has moved to data lakes on object storage, open source warehouses have begun to break apart. The following components, which were previously integrated in a single system such as Hive, are now often implemented as separate components:
Query engine
Query engines such as Trino, Apache DataFusion, and Presto parse SQL queries, optimize them into execution plans, and execute them against the data. Execution usually requires parallel, distributed data processing tasks. Some query engines provide built-in task execution, while others choose to use third-party execution frameworks such as Spark or Flink.
Storage format
The storage format determines how the rows of a table are encoded as bytes in a file, which is then typically stored in object storage or a distributed filesystem. This data can then be accessed not only by the query engine, but also by other applications using the data lake. Examples of such storage formats are Parquet, ORC, Lance, and Nimble; we’ll talk more about them in the next section.
Table format
Files written in Parquet and similar storage formats are typically immutable once written. To support row inserts and deletions, a table format such as Apache Iceberg or Databricks’s Delta format can be used. Table formats specify a file format that defines which files constitute a table along with the table’s schema. Such formats also offer advanced features such as time travel (the ability to query a table as it was at a previous point in time), GC, and even transactions.
Data catalog
Just as a table format defines which files make up a table, a data catalog defines which tables are contained in a database. Catalogs are used to create, rename, and drop tables. Unlike storage and table formats, data catalogs such as Snowflake’s Polaris and Databricks’s Unity Catalog usually run as a standalone service that can be queried using a REST interface. Apache Iceberg also offers a catalog, which can be run inside a client or as a separate process. Query engines use catalog information when reading and writing tables. Traditionally, catalogs and query engines have been integrated, but decoupling them has enabled data discovery and data governance systems (discussed in “Data Systems, Law, and Society”) to access a catalog’s metadata as well.