notexOPENOverview
Notex/Systems/Resiliency

Resiliency

Always free to read

▷LESSON VIDEOOverview: ResiliencyVideo coming soon. Start with the reading below.COMING SOON

References

Overview: Resiliency
  1. Also known as Murphy’s law
Common failure causes
  1. “A Conceptual Framework for System Fault Tolerance,” https://resources.sei.cmu.edu/asset_files/TechnicalReport/1992_005_001_16112.pdf
  2. “Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems,” https://www.usenix.org/system/files/conference/osdi14/osdi14-paper-yuan.pdf
  3. This is the reason the Go language puts so much emphasis on error handling.
  4. “A List of Post-mortems: Config Errors,” https://github.com/danluu/post-mortems#config-errors
  5. “Early Detection of Configuration Errors to Reduce Failure Damage,” https://www.usenix.org/system/files/conference/osdi16/osdi16-xu.pdf
  6. “It’s always DNS,” https://twitter.com/ahidalgosre/status/1315345619926609920?lang=en-GB
  7. “Foursquare Goes Dark Too. Unintentionally.,” https://techcrunch.com/2010/03/27/foursquare-offline
  8. “Stop using .IO Domain Names for Production Traffic,” https://hackernoon.com/stop-using-io-domain-names-for-production-traffic-b6aa17eeac20
  9. “Microsoft Teams goes down after Microsoft forgot to renew a certificate,” https://www.theverge.com/2020/2/3/21120248/microsoft-teams-down-outage-certificate-issue-status
  10. “Fallacies of distributed computing,” https://en.wikipedia.org/wiki/Fallacies_of_distributed_computing
  11. “Gray Failure: The Achilles’ Heel of Cloud-Scale Systems,” https://www.microsoft.com/en-us/research/wp-content/uploads/2017/06/paper-1.pdf
  12. “Metastable Failures in Distributed Systems,” https://sigops.org/s/conferences/hotos/2021/papers/hotos21-s11-bronson.pdf
  13. “Risk matrix,” https://en.wikipedia.org/wiki/Risk_matrix
Redundancy
  1. “When Redundancy Actually Helps,” https://brooker.co.za/blog/2019/06/20/redundancy.html
  2. “Active-Active for Multi-Regional Resiliency,” https://netflixtechblog.com/active-active-for-multi-regional-resiliency-c47719f6685b
  3. “The CJEU judgment in the Schrems II case,” https://www.europarl.europa.eu/RegData/etudes/ATAG/2020/652073/EPRS_ATA(2020)652073_EN.pdf
Fault isolation
  1. We discussed partitioning in Partitioning (lesson 04.04) from a scalability point of view.
  2. “Shuffle Sharding: Massive and Magical Fault Isolation,” https://aws.amazon.com/blogs/architecture/shuffle-sharding-massive-and-magical-fault-isolation/
  3. Partitioning by user is just an example; we could partition just as well by physical location, workload, or any other dimension that makes sense for the application.
  4. “New Relic case: Huge scale, small clusters: Using Cells to scale in the Cloud,” https://www.youtube.com/watch?v=eMikCXiBlOA
Downstream resiliency
  1. “Web APIs: XMLHttpRequest.timeout,” https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/timeout
  2. “Add a timeout option, to prevent hanging,” https://github.com/whatwg/fetch/issues/951
  3. “Web APIs: AbortController,” https://developer.mozilla.org/en-US/docs/Web/API/AbortController
  4. “Requests Quickstart: Timeouts,” https://requests.readthedocs.io/en/master/user/quickstart/#timeouts
  5. “net/http: make default configs have better timeouts,” https://github.com/golang/go/issues/24138
  6. “HttpClient.Timeout Property,” https://docs.microsoft.com/en-us/dotnet/api/system.net.http.httpclient.timeout?view=net-6.0#remarks
  7. “Timeouts, retries, and backoff with jitter,” https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/
  8. “Exponential Backoff And Jitter,” https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
Upstream resiliency
  1. We have already met a mechanism to protect against upstream pressure when we discussed health checks in the context of load balancers in Network load balancing (lesson 04.06).
  2. “Using load shedding to avoid overload,” https://aws.amazon.com/builders-library/using-load-shedding-to-avoid-overload
  3. “Autoscaling,” https://en.wikipedia.org/wiki/Autoscaling
  4. This is another application of the concept of static stability, first introduced in Control planes and data planes (lesson 04.10).
  5. “Some opinionated thoughts on SQL databases,” https://blog.nelhage.com/post/some-opinionated-sql-takes/
  6. “Reliability, constant work, and a good cup of coffee,” https://aws.amazon.com/builders-library/reliability-and-constant-work/
KEEP EXPLORING

One note leads to another.

Back to the library →
⌕
Search across the entire library ⌘ K