Video coming soon. Start with the reading below.
Rate limiting can be implemented using different algorithms, each with its pros and cons. The interactions between an interviewer and a candidate help to clarify the type of rate limiters we are trying to build.
Candidate: What kind of rate limiter are we going to design? Is it a client-side rate limiter or server-side API rate limiter? Interviewer: Great question. We focus on the server-side API rate limiter.
Candidate: Does the rate limiter throttle API requests based on IP, the user ID, or other properties? Interviewer: The rate limiter should be flexible enough to support different sets of throttle rules.
Candidate: What is the scale of the system? Is it built for a startup or a big company with a large user base? Interviewer: The system must be able to handle a large number of requests.
Candidate: Will the system work in a distributed environment? Interviewer: Yes.
Candidate: Is the rate limiter a separate service or should it be implemented in application code? Interviewer: It is a design decision up to you.
Candidate: Do we need to inform users who are throttled? Interviewer: Yes.
Requirements Here is a summary of the requirements for the system:
- Accurately limit excessive requests.
- Low latency. The rate limiter should not slow down HTTP response time.
- Use as little memory as possible.
- Distributed rate limiting. The rate limiter can be shared across multiple servers or processes.
- Exception handling. Show clear exceptions to users when their requests are throttled.
- High fault tolerance. If there are any problems with the rate limiter (for example, a cache server goes offline), it does not affect the entire system.