In recent years, cloud storage services such as Google Drive, Dropbox, Microsoft OneDrive, and Apple iCloud have become very popular. In this chapter, you are asked to design…
- 15.01Design Google DriveIn recent years, cloud storage services such as Google Drive, Dropbox, Microsoft OneDrive, and Apple iCloud have become very popular. In this chapter, you are asked to design…
- 15.02Step 1 - Understand the problem and establish design scopeDesigning a Google drive is a big project, so it is important to ask questions to narrow down the scope. Candidate: What are the most important features? Interviewer: Upload…
- 15.03Back of the envelope estimationAssume the application has 50 million signed up users and 10 million DAU.Users get 10 GB free space.Assume users upload 2 files per day. The average file size is 500 KB.1:1…
- 15.04Step 2 - Propose high-level design and get buy-inInstead of showing the high-level design diagram from the beginning, we will use a slightly different approach. We will start with something simple: build everything in a…
- 15.05APIsWhat do the APIs look like? We primary need 3 APIs: upload a file, download a file, and get file revisions. 1. Upload a file to Google Drive Two types of uploads are supported:…
- 15.06Move away from single serverAs more files are uploaded, eventually you get the space full alert as shown in Figure 15-4. Only 10 MB of storage space is left! This is an emergency as users cannot upload…
- 15.07Sync conflictsFor a large storage system like Google Drive, sync conflicts happen from time to time. When two users modify the same file or folder at the same time, a conflict happens. How…
- 15.08High-level designFigure 15-10 illustrates the proposed high-level design. Let us examine each component of the system. User: A user uses the application either through a browser or mobile app.…
- 15.09Step 3 - Design deep diveIn this section, we will take a close look at the following: block servers, metadata database, upload flow, download flow, notification service, save storage space and failure…
- 15.10Block serversFor large files that are updated regularly, sending the whole file on each update consumes a lot of bandwidth. Two optimizations are proposed to minimize the amount of network…
- 15.11High consistency requirementOur system requires strong consistency by default. It is unacceptable for a file to be shown differently by different clients at the same time. The system needs to provide…
- 15.12Metadata databaseFigure 15-13 shows the database schema design. Please note this is a highly simplified version as it only includes the most important tables and interesting fields. User: The…
- 15.13Upload flowLet us discuss what happens when a client uploads a file. To better understand the flow, we draw the sequence diagram as shown in Figure 15-14. In Figure 15-14, two requests…
- 15.14Download flowDownload flow is triggered when a file is added or edited elsewhere. How does a client know if a file is added or edited by another client? There are two ways a client can…
- 15.15Notification serviceTo maintain file consistency, any mutation of a file performed locally needs to be informed to other clients to reduce conflicts. Notification service is built to serve this…
- 15.16Save storage spaceTo support file version history and ensure reliability, multiple versions of the same file are stored across multiple data centers. Storage space can be filled up quickly with…
- 15.17Failure HandlingFailures can occur in a large-scale system and we must adopt design strategies to address these failures. Your interviewer might be interested in hearing about how you handle…
- 15.18Step 4 - Wrap upIn this chapter, we proposed a system design to support Google Drive. The combination of strong consistency, low network bandwidth and fast sync make the design interesting.…
References
Step 4 - Wrap up
- Google Drive: https://www.google.com/drive/
- Upload file data: https://developers.google.com/drive/api/v2/manage-uploads
- Amazon S3: https://aws.amazon.com/s3
- Differential Synchronization https://neil.fraser.name/writing/sync/
- Differential Synchronization youtube talk https://www.youtube.com/watch? v=S2Hp_1jqpY8
- How We’ve Scaled Dropbox: https://youtu.be/PE4gwstWhmc
- Tridgell, A., & Mackerras, P. (1996). The rsync algorithm.
- Librsync. (n.d.). Retrieved April 18, 2015, from https://github.com/librsync/librsync
- ACID: https://en.wikipedia.org/wiki/ACID
- Dropbox security white paper: https://www.dropbox.com/static/business/resources/Security_Whitepaper.pdf
- Amazon S3 Glacier: https://aws.amazon.com/glacier/faqs/