notexOPENOverview
Notex/Systems/Error handling
14.12LESSON 12 OF 13 · 1 MIN READ

Error handling

Always free to read
SECTION VIDEOError handling

Video coming soon. Start with the reading below.

COMING SOON

For a large-scale system, system errors are unavoidable. To build a highly fault-tolerant system, we must handle errors gracefully and recover from them fast. Two types of errors exist:

  • Recoverable error. For recoverable errors such as video segment fails to transcode, the general idea is to retry the operation a few times. If the task continues to fail and the system believes it is not recoverable, it returns a proper error code to the client.
  • Non-recoverable error. For non-recoverable errors such as malformed video format, the system stops the running tasks associated with the video and returns the proper error code to the client.

Typical errors for each system component are covered by the following playbook:

  • Upload error: retry a few times.
  • Split video error: if older versions of clients cannot split videos by GOP alignment, the entire video is passed to the server. The job of splitting videos is done on the server-side.
  • Transcoding error: retry.
  • Preprocessor error: regenerate DAG diagram.
  • DAG scheduler error: reschedule a task.
  • Resource manager queue down: use a replica.
  • Task worker down: retry the task on a new worker.
  • API server down: API servers are stateless so requests will be directed to a different API server.
  • Metadata cache server down: data is replicated multiple times. If one node goes down, you can still access other nodes to fetch data. We can bring up a new cache server to replace the dead one.
  • Metadata DB server down:
  • Master is down. If the master is down, promote one of the slaves to act as the new master.
  • Slave is down. If a slave goes down, you can use another slave for reads and bring up another database server to replace the dead one.
Progress is saved on this device.
KEEP EXPLORING

One note leads to another.

Back to the library →
⌕
Search across the entire library ⌘ K