If you are working on data systems in an enterprise, you are likely to encounter several different types of people who work with data. The first type are backend engineers who build services that handle requests for reading and updating data; these services often serve external users, either directly or indirectly via other services (see “Microservices and Serverless”). Sometimes services are for internal use by other parts of the organization.
In addition to the teams managing backend services, two other groups of people typically require access to an organization’s data: business analysts, who generate reports about the activities of the organization to help management make better decisions (business intelligence, or BI), and data scientists, who look for novel insights in data or who create user-facing product features that are enabled by data analysis and machine learning (ML)/AI (e.g., “people who bought X also bought Y" recommendations on an ecommerce website, predictive analytics such as risk scoring or spam filtering, and ranking of search results).
Although business analysts and data scientists tend to use different tools and operate in different ways, they have some practices in common. First, both perform analytics, which means they look at the data that the users and backend services have generated. Second, they generally do not modify this data (except perhaps for fixing mistakes), although they might create derived datasets in which the original data has been processed in some way.
This has led to a split between two types of systems—a distinction that we will use throughout these notes:
-
Operational systems consist of the backend services and data infrastructure where data is created—for example, by serving external users. Here, the application code both reads and modifies the data in its databases, based on the actions performed by the users.
-
Analytical systems serve the needs of business analysts and data scientists. They contain a read-only copy of the data from the operational systems, and they are optimized for the types of data processing that are needed for analytics.
As we shall see in the next section, operational and analytical systems are often kept separate, for good reasons. As these systems have matured, two new specialized roles have emerged: data engineers and analytics engineers. Data engineers are the people who know how to integrate the operational and analytical systems and who take responsibility for the organization’s data infrastructure more widely. Analytics engineers model and transform data to make it more useful for the business analysts and data scientists in an organization.
Many engineers specialize in either the operational or the analytical side. However, these notes covers both operational and analytical data systems, since both play an important role in the lifecycle of data within an organization. We will explore in depth the data infrastructure that is used to deliver services to both internal and external users so that you can work better with your colleagues on the other side of this divide.