notexOPENOverview
Notex/Systems/Failure Handling
15.17LESSON 17 OF 18 · 2 MIN READ

Failure Handling

Always free to read
SECTION VIDEOFailure Handling

Video coming soon. Start with the reading below.

COMING SOON

Failures can occur in a large-scale system and we must adopt design strategies to address these failures. Your interviewer might be interested in hearing about how you handle the following system failures:

  • Load balancer failure: If a load balancer fails, the secondary would become active and pick up the traffic. Load balancers usually monitor each other using a heartbeat, a periodic signal sent between load balancers. A load balancer is considered as failed if it has not sent a heartbeat for some time.
  • Block server failure: If a block server fails, other servers pick up unfinished or pending jobs.
  • Cloud storage failure: S3 buckets are replicated multiple times in different regions. If files are not available in one region, they can be fetched from different regions.
  • API server failure: It is a stateless service. If an API server fails, the traffic is redirected to other API servers by a load balancer.
  • Metadata cache failure: Metadata cache servers are replicated multiple times. If one node goes down, you can still access other nodes to fetch data. We will bring up a new cache server to replace the failed one.
  • Metadata DB failure.
  • Master down: If the master is down, promote one of the slaves to act as a new master and bring up a new slave node.
  • Slave down: If a slave is down, you can use another slave for read operations and

bring another database server to replace the failed one. • Notification service failure: Every online user keeps a long poll connection with the notification server. Thus, each notification server is connected with many users. According to the Dropbox talk in 2012 [6], over 1 million connections are open per machine. If a server goes down, all the long poll connections are lost so clients must reconnect to a different server. Even though one server can keep many open connections, it cannot reconnect all the lost connections at once. Reconnecting with all the lost clients is a relatively slow process. • Offline backup queue failure: Queues are replicated multiple times. If one queue fails, consumers of the queue may need to re-subscribe to the backup queue.

Progress is saved on this device.
KEEP EXPLORING

One note leads to another.

Back to the library →
⌕
Search across the entire library ⌘ K