System design interview question · medium · rate limiting · consistency
Token-based rate limits
Design rate limiting and quotas for a public LLM API. Each organization has limits on requests per minute and tokens per minute for each model, plus a monthly spend cap. Limits apply across all API servers in all regions.
Practice it as a 50-minute mock interview
- Requirements & scope (7 min): Think like a product manager, translate design question to technical requirements, narrow the scope and write the non-functional requirements as numbers.
- Estimation (3 min): Size the system: estimate the scale it must handle, find the number that drives the design, and name the first bottleneck.
- API & data model (5 min): Write concrete endpoints and entities, and pick stores from access patterns.
- High-level architecture & Data flow (10 min): Start with the simplest design that works and trace a request end to end.
- Deep dives (20 min): Go deep on the 2–3 hardest parts, with alternatives and a reasoned choice.
- Failures, ops & evolution (5 min): Walk each component: what if it dies, slows down, or is overloaded?