▷LESSON VIDEOOverview: ResiliencyVideo coming soon. Start with the reading below.COMING SOON
- 05.01Overview: Resiliency▷LESSON VIDEOOverview: ResiliencyVideo coming soon. Start with the reading below.COMING SOON
- 05.02Common failure causes▷LESSON VIDEOCommon failure causesVideo coming soon. Start with the reading below.COMING SOON
- 05.03Redundancy▷LESSON VIDEORedundancyVideo coming soon. Start with the reading below.COMING SOON
- 05.04Fault isolation▷LESSON VIDEOFault isolationVideo coming soon. Start with the reading below.COMING SOON
- 05.05Downstream resiliency▷LESSON VIDEODownstream resiliencyVideo coming soon. Start with the reading below.COMING SOON
- 05.06Upstream resiliency▷LESSON VIDEOUpstream resiliencyVideo coming soon. Start with the reading below.COMING SOON
References
Overview: Resiliency
- Also known as Murphy’s law
Common failure causes
- “A Conceptual Framework for System Fault Tolerance,” https://resources.sei.cmu.edu/asset_files/TechnicalReport/1992_005_001_16112.pdf
- “Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems,” https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf
- This is the reason the Go language puts so much emphasis on error handling.
- “A List of Post-mortems: Config Errors,” https://github.com/danluu/post-mortems#config-errors
- “Early Detection of Configuration Errors to Reduce Failure Damage,” https://www.usenix.org/system/files/conference/osdi16/osdi16-xu.pdf
- “It’s always DNS,” https://twitter.com/ahidalgosre/status/1315345619926609920?lang=en-GB
- “Foursquare Goes Dark Too. Unintentionally.,” https://techcrunch.com/2010/03/27/foursquare-offline
- “Stop using .IO Domain Names for Production Traffic,” https://hackernoon.com/stop-using-io-domain-names-for-production-traffic-b6aa17eeac20
- “Microsoft Teams goes down after Microsoft forgot to renew a certificate,” https://www.theverge.com/2020/2/3/21120248/microsoft-teams-down-outage-certificate-issue-status
- “Fallacies of distributed computing,” https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
- “Gray Failure: The Achilles’ Heel of Cloud-Scale Systems,” https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf
- “Metastable Failures in Distributed Systems,” https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf
- “Risk matrix,” https://en.wikipedia.org/wiki/Risk_matrix
Redundancy
- “When Redundancy Actually Helps,” https://brooker.co.za/blog/2019/06/20/redundancy.html
- “Active-Active for Multi-Regional Resiliency,” https://netflixtechblog.com/active-active-for-multi-regional-resiliency-c47719f6685b
- “The CJEU judgment in the Schrems II case,” https://www.europarl.europa.eu/RegData/etudes/ATAG/2020/652073/EPRS_ATA(2020)652073_EN.pdf
Fault isolation
- We discussed partitioning in Partitioning (lesson 04.04) from a scalability point of view.
- “Shuffle Sharding: Massive and Magical Fault Isolation,” https://aws.amazon.com/blogs/architecture/shuffle-sharding-massive-and-magical-fault-isolation/
- Partitioning by user is just an example; we could partition just as well by physical location, workload, or any other dimension that makes sense for the application.
- “New Relic case: Huge scale, small clusters: Using Cells to scale in the Cloud,” https://www.youtube.com/watch?v=eMikCXiBlOA
Downstream resiliency
- “Web APIs: XMLHttpRequest.timeout,” https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/timeout
- “Add a timeout option, to prevent hanging,” https://github.com/whatwg/fetch/issues/951
- “Web APIs: AbortController,” https://developer.mozilla.org/en-US/docs/Web/API/AbortController
- “Requests Quickstart: Timeouts,” https://requests.readthedocs.io/en/master/user/quickstart/#timeouts
- “net/http: make default configs have better timeouts,” https://github.com/golang/go/issues/24138
- “HttpClient.Timeout Property,” https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient.timeout?view=net-6.0#remarks
- “Timeouts, retries, and backoff with jitter,” https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
- “Exponential Backoff And Jitter,” https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
Upstream resiliency
- We have already met a mechanism to protect against upstream pressure when we discussed health checks in the context of load balancers in Network load balancing (lesson 04.06).
- “Using load shedding to avoid overload,” https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload
- “Autoscaling,” https://en.wikipedia.org/wiki/Autoscaling
- This is another application of the concept of static stability, first introduced in Control planes and data planes (lesson 04.10).
- “Some opinionated thoughts on SQL databases,” https://blog.nelhage.com/post/some-opinionated-sql-takes/
- “Reliability, constant work, and a good cup of coffee,” https://aws.amazon.com/builders-library/reliability-and-constant-work/