notexOPENOverview
Notex/Systems/Design Google Drive

Design Google Drive

Always free to read

In recent years, cloud storage services such as Google Drive, Dropbox, Microsoft OneDrive, and Apple iCloud have become very popular. In this chapter, you are asked to design…

15

DESIGN CASE STUDIES

Design Google Drive

18 LESSONS · 23 MIN
  1. 15.01Design Google DriveIn recent years, cloud storage services such as Google Drive, Dropbox, Microsoft OneDrive, and Apple iCloud have become very popular. In this chapter, you are asked to design…READING1 MIN
  2. 15.02Step 1 - Understand the problem and establish design scopeDesigning a Google drive is a big project, so it is important to ask questions to narrow down the scope. Candidate: What are the most important features? Interviewer: Upload…READING2 MIN
  3. 15.03Back of the envelope estimationAssume the application has 50 million signed up users and 10 million DAU.Users get 10 GB free space.Assume users upload 2 files per day. The average file size is 500 KB.1:1…READING1 MIN
  4. 15.04Step 2 - Propose high-level design and get buy-inInstead of showing the high-level design diagram from the beginning, we will use a slightly different approach. We will start with something simple: build everything in a…READING1 MIN
  5. 15.05APIsWhat do the APIs look like? We primary need 3 APIs: upload a file, download a file, and get file revisions. 1. Upload a file to Google Drive Two types of uploads are supported:…READING1 MIN
  6. 15.06Move away from single serverAs more files are uploaded, eventually you get the space full alert as shown in Figure 15-4. Only 10 MB of storage space is left! This is an emergency as users cannot upload…READING2 MIN
  7. 15.07Sync conflictsFor a large storage system like Google Drive, sync conflicts happen from time to time. When two users modify the same file or folder at the same time, a conflict happens. How…READING1 MIN
  8. 15.08High-level designFigure 15-10 illustrates the proposed high-level design. Let us examine each component of the system. User: A user uses the application either through a browser or mobile app.…READING2 MIN
  9. 15.09Step 3 - Design deep diveIn this section, we will take a close look at the following: block servers, metadata database, upload flow, download flow, notification service, save storage space and failure…READING1 MIN
  10. 15.10Block serversFor large files that are updated regularly, sending the whole file on each update consumes a lot of bandwidth. Two optimizations are proposed to minimize the amount of network…READING1 MIN
  11. 15.11High consistency requirementOur system requires strong consistency by default. It is unacceptable for a file to be shown differently by different clients at the same time. The system needs to provide…READING1 MIN
  12. 15.12Metadata databaseFigure 15-13 shows the database schema design. Please note this is a highly simplified version as it only includes the most important tables and interesting fields. User: The…READING1 MIN
  13. 15.13Upload flowLet us discuss what happens when a client uploads a file. To better understand the flow, we draw the sequence diagram as shown in Figure 15-14. In Figure 15-14, two requests…READING1 MIN
  14. 15.14Download flowDownload flow is triggered when a file is added or edited elsewhere. How does a client know if a file is added or edited by another client? There are two ways a client can…READING1 MIN
  15. 15.15Notification serviceTo maintain file consistency, any mutation of a file performed locally needs to be informed to other clients to reduce conflicts. Notification service is built to serve this…READING1 MIN
  16. 15.16Save storage spaceTo support file version history and ensure reliability, multiple versions of the same file are stored across multiple data centers. Storage space can be filled up quickly with…READING1 MIN
  17. 15.17Failure HandlingFailures can occur in a large-scale system and we must adopt design strategies to address these failures. Your interviewer might be interested in hearing about how you handle…READING2 MIN
  18. 15.18Step 4 - Wrap upIn this chapter, we proposed a system design to support Google Drive. The combination of strong consistency, low network bandwidth and fast sync make the design interesting.…READING2 MIN

References

Step 4 - Wrap up
  1. Google Drive: https://www.google.com/drive/
  2. Upload file data: https://developers.google.com/drive/api/v2/manage-uploads
  3. Amazon S3: https://aws.amazon.com/s3
  4. Differential Synchronization https://neil.fraser.name/writing/sync/
  5. Differential Synchronization youtube talk https://www.youtube.com/watch? v=S2Hp_1jqpY8
  6. How We’ve Scaled Dropbox: https://youtu.be/PE4gwstWhmc
  7. Tridgell, A., & Mackerras, P. (1996). The rsync algorithm.
  8. Librsync. (n.d.). Retrieved April 18, 2015, from https://github.com/librsync/librsync
  9. ACID: https://en.wikipedia.org/wiki/ACID
  10. Dropbox security white paper: https://www.dropbox.com/static/business/resources/Security_Whitepaper.pdf
  11. Amazon S3 Glacier: https://aws.amazon.com/glacier/faqs/
KEEP EXPLORING

One note leads to another.

Back to the library →
⌕
Search across the entire library ⌘ K