System design interview question · hard · gpu · routing · streaming
LLM inference gateway
Design the API gateway in front of a fleet of GPU model servers. It accepts developer requests for several models, authenticates them, applies limits, routes each request to a suitable model replica, and streams the response back.
Practice it as a 50-minute mock interview
- Requirements & scope (7 min): Think like a product manager, translate design question to technical requirements, narrow the scope and write the non-functional requirements as numbers.
- Estimation (3 min): Size the system: estimate the scale it must handle, find the number that drives the design, and name the first bottleneck.
- API & data model (5 min): Write concrete endpoints and entities, and pick stores from access patterns.
- High-level architecture & Data flow (10 min): Start with the simplest design that works and trace a request end to end.
- Deep dives (20 min): Go deep on the 2–3 hardest parts, with alternatives and a reasoned choice.
- Failures, ops & evolution (5 min): Walk each component: what if it dies, slows down, or is overloaded?