Design an API Gateway (LLD)
Routing, middleware chains (auth/rate-limit/logging), and pluggable filters behind one entry point.
Problem Statement
Design the low-level object model for an API Gateway that receives client requests, matches them to backend services, applies a configurable filter chain, transforms requests and responses, and returns a single consistent gateway response.
The interview is about the LLD view: classes, ownership, extension seams, and how cross-cutting concerns such as authentication, rate limiting, logging, and transformation are added without rewriting route matching or backend dispatch.
Business context
API gateways sit at the front door of microservice platforms at companies like Amazon, Netflix, and Atlassian. They concentrate concerns that would otherwise be duplicated in every service: identity checks, quotas, request normalization, audit logging, response shaping, and routing. Interviewers use this problem to see whether you can keep routing policy separate from middleware policy and avoid a giant controller full of conditionals.
Functional Requirements
Accept a Request with method, path, headers, body, and client identity.
Match the request against ordered Route rules and pick the target backend service.
Resolve backend service names through a ServiceRegistry instead of hard-coding service objects.
Run a filter chain before dispatch so authentication, rate limiting, logging, and transformation are composable.
Allow request and response transformation as pluggable strategies in the chain.
Return consistent Response objects for success, unauthorized access, throttling, no route, and backend errors.
Allow new filters to be inserted without changing route matching or backend service classes.
Non-Functional Requirements
Extensibility of cross-cutting concerns
Adding a new filter such as metrics, feature flags, or request signing should mean adding one class and registering it in order, not editing ApiGateway.dispatch.
Predictable ordering
Filters run in a deterministic order because auth before rate limit before logging can produce different semantics than the reverse.
Low routing latency
The base implementation uses an ordered in-memory route list for clarity; production can replace it with a trie or indexed matcher without changing filters.
Failure isolation
Gateway-generated failures such as 401, 429, and 404 are returned before backend dispatch, so downstream services are protected from bad traffic.
Thread safety at shared state
The registry and rate limiter guard mutable maps. Immutable request and response objects prevent filters from corrupting shared state.
Requirement Clarification
QIs this a system-design gateway with service discovery, retries, and autoscaling?
No. Focus on the LLD view: request model, route matching, filter chain, transformation, and service registry. Retries and discovery adapters are production extensions.
QShould filters be tied to individual routes or shared globally?
Use a global ordered chain for the base design. Route-specific filters can be added later by letting Route provide extra filters around the same dispatch seam.
QWhat route matching rules are required?
Support method plus path-prefix matching through a Route matcher strategy. Exact paths, host routing, and header predicates are extensions.
QDo transformations happen before or after routing?
The transformation filter normalizes the request before dispatch and shapes the response after dispatch. Route matching still receives the normalized request.
QDo we model real HTTP clients and network I/O?
No. A BackendService interface represents a backend. The gateway proxies to that interface; real network clients are adapters behind it.
UML Class Diagram
Sequence Diagram
Entity Identification
Request
Immutable inbound message carrying method, path, headers, body, and client id. Filters can create modified copies instead of mutating shared state.
Response
Immutable outbound message with status, headers, and body. Gateway, filters, and backends all return the same response abstraction.
Filter
Middleware contract. Each filter may reject the request, enrich it, observe it, or call chain.proceed to pass control onward.
FilterChain
Chain cursor that invokes filters one by one and finally calls the terminal gateway dispatch function.
AuthFilter
Checks the authorization header before traffic reaches routing or backend services. It protects every route uniformly.
RateLimitFilter
Maintains per-client fixed windows and short-circuits with 429 when a caller exceeds quota.
LoggingFilter / TransformationFilter
Logging observes both sides of the chain. Transformation applies request and response strategies around downstream processing.
Route
A named routing rule. It owns a matcher strategy and the backend service name to resolve when the rule matches.
ApiGateway
Facade and proxy entry point. It builds the chain for each request, then dispatches matched traffic to a backend from the registry.
ServiceRegistry
Singleton registry from logical service name to BackendService. It decouples routes from concrete backend objects.
Design Patterns Used
FilterChain lets authentication, rate limiting, logging, and transformation decide independently whether to handle, reject, or forward a request. New cross-cutting concerns are additive and do not touch routing.
Route accepts a matcher strategy, and TransformationFilter accepts request and response transformation strategies. Matching and shaping logic can vary without subclassing the gateway.
ApiGateway is the client-facing proxy for backend services. It controls access, adds policies, and forwards only clean requests to BackendService.
ServiceRegistry.getInstance provides one shared in-memory registry for the demo so all routes resolve names against the same source of truth.
Step-by-Step Design
1Start with immutable request and response messages
Filters should not mutate a shared request in place. Request.withHeader and Response.withHeader return copies, making it safe for transformations to enrich messages.
public Request withHeader(String name, String value) { return toBuilder().header(name, value).build(); }2Define middleware as a Chain of Responsibility
The filter contract receives the current request and a chain cursor. A filter can return immediately or call proceed. That one method is the extension seam for auth, quotas, logging, metrics, and transformation.
public interface Filter { Response apply(Request request, FilterChain chain); } public Response proceed(Request request) { if (index == filters.size()) { return terminal.handle(request); } FilterChain nextChain = new FilterChain(filters, terminal, index + 1); return filters.get(index).apply(request, nextChain); }3Keep authentication and rate limiting out of routing
AuthFilter and RateLimitFilter short-circuit before dispatch. ApiGateway.dispatch does not know why a request was rejected because rejection belongs to filters.
4Represent route matching as a strategy
A Route owns a predicate. The starter factory builds method plus prefix routes, but exact path, host-based, or header-based routes can reuse the same gateway.
public boolean matches(Request request) { return matcher.test(request); }5Resolve services by name through a registry
Routes store logical service names. ServiceRegistry maps those names to BackendService implementations so route configuration and service construction remain separate.
6Place transformation around dispatch
TransformationFilter normalizes the request before proceed and shapes the response after proceed returns. It is still just another filter in the same chain.
Request outgoing = requestTransformer.apply(request); Response incoming = chain.proceed(outgoing); return responseTransformer.apply(incoming);
7Make the gateway a proxy, not a policy dump
ApiGateway.handle runs the chain and dispatch only matches routes plus calls the registry. Every cross-cutting decision lives in a filter, keeping routing stable.
Complete Java Implementation
Explanation of Every Class
Request
Immutable request value object with builder-style construction and copy methods. Filters use withHeader and withBody to transform safely.
Response
Immutable response value object used by filters, gateway-generated failures, and backends. The builder keeps status, body, and headers consistent.
Filter / FilterChain
Filter is the middleware contract. FilterChain is the cursor that advances through filters and finally calls terminal dispatch.
AuthFilter
Validates the authorization token and returns 401 without calling proceed when the caller is unknown.
RateLimitFilter
Maintains synchronized per-client fixed windows and returns 429 with Retry-After when a client exceeds quota.
LoggingFilter / TransformationFilter
LoggingFilter observes request and response events. TransformationFilter wraps downstream processing with request and response strategy functions.
Route
Encapsulates matching policy and the logical backend name. The prefix factory is one strategy; callers may pass any predicate.
ApiGateway / ServiceRegistry / BackendService
ApiGateway runs the chain then proxies to the resolved backend. ServiceRegistry is the singleton name-to-service map, and BackendService is the backend abstraction.
Dry Run
Sample input
Routes: GET /users to user-service, POST /orders to order-service. Filters: auth, rate limit 2 per minute, transformation adding X-Gateway-Version, logging. Client c1 sends three authorized GET requests and one unauthorized request.
| Step | Request | Filter outcome | Route chosen | Response |
|---|---|---|---|---|
| 1 | GET /users with valid token | Auth pass, quota 1 of 2, transformed | user-service | 200 with gateway header |
| 2 | GET /users with valid token | Auth pass, quota 2 of 2, transformed | user-service | 200 with gateway header |
| 3 | GET /users with valid token | Rate limit short-circuits | none | 429 Too Many Requests |
| 4 | POST /orders without token | Auth short-circuits | none | 401 Unauthorized |
| 5 | GET /unknown with valid token from c2 | Filters pass | none | 404 No route |
The third request proves the chain can stop before routing. The fourth proves authentication also stops before routing. The fifth proves routing failures are gateway responses, not backend errors.
Complexity Analysis
| Operation | Time | Space | Note |
|---|---|---|---|
| handle request through filters | O(F + R) | O(F) | F filters are visited at most once; R routes are scanned only if all filters proceed. |
| route matching | O(R) | O(1) | The base design scans ordered routes. A trie or indexed matcher can reduce this. |
| service resolution | O(1) | O(1) | Hash-map lookup by service name in the singleton registry. |
| rate limit check | O(1) | O(C) | C active clients have one fixed window each. |
| request or response transformation | O(H + B) | O(H + B) | Copying headers and body-sized transformations dominate the cost. |
The hot path is F + R. Keep filters small and deterministic, then replace the route list with a trie, radix tree, or precompiled matcher when route count grows.
Extensibility
Adding a new filter
Create a class implementing Filter and insert it into the filter list. ApiGateway, Route, and backends remain unchanged.
Changing route matching
Pass a different predicate into Route or replace route storage with a matcher component. The filter chain still ends at the same dispatch method.
Adding a backend
Register a new BackendService under a logical name and add a route pointing to that name. No filter needs to know the backend exists.
Route-specific policies
Let Route optionally expose extra filters and compose them before terminal dispatch. This extends policy without moving conditionals into the gateway.
Dynamic service discovery
Swap the in-memory singleton map for a registry adapter backed by Consul, Kubernetes, or a database while preserving the resolve contract.
Alternative Designs
Giant gateway controller
Put auth, rate limiting, logging, transformation, route matching, and dispatch in one handle method with conditionals.
Tradeoffs
Simple for a toy demo, but every new concern edits the same method, ordering becomes fragile, and unit tests become broad.
Route owns all middleware
Each route carries its own filter list and the gateway selects a route before running filters.
Tradeoffs
Useful for route-specific policy, but global authentication and rate limiting now wait until after route matching and may be duplicated across routes.
Trie-based router component
Move matching from an ordered list to a trie keyed by method and path segments.
Tradeoffs
Improves route matching for large catalogs, but adds complexity and does not replace the filter chain seam.
External registry adapter
Use a ServiceRegistry interface and adapter implementations for static maps, service discovery, or load balancers.
Tradeoffs
More production-ready and testable than a singleton, but the singleton is concise for the interview demo.
Common Mistakes
- ×
Putting authentication, quotas, and logging inside ApiGateway.dispatch, which makes routing change whenever policy changes.
- ×
Letting filters mutate the same Request instance, which makes ordering bugs hard to diagnose.
- ×
Running route matching before global authentication, leaking route existence to unauthorized clients.
- ×
Hard-coding backend objects inside routes instead of resolving by logical service name.
- ×
Using unordered filters, which makes rate limit and authentication semantics unpredictable.
- ×
Catching every backend exception as success or exposing raw exceptions directly to clients.
- ×
Making the service registry a hidden global dependency in every class instead of injecting it into ApiGateway.
Follow-up Interview Questions
QHow do you add metrics without touching routing?
Add a MetricsFilter that records start time, calls proceed, then emits latency and status. Insert it into the filter list near logging.
QHow would you support per-route rate limits?
Either enrich Request with the matched route before quota evaluation or allow Route to contribute a route-specific filter list after global filters.
QHow do you make route matching faster for thousands of routes?
Replace the ordered list with a method plus path trie or compiled route table. Keep Route.matches or a router interface so filters stay unchanged.
QWhere do retries and circuit breakers belong?
They belong behind or around BackendService as proxy filters or backend adapters. They should not be mixed into route selection.
QHow would you make the registry production-ready?
Replace the singleton map with a ServiceRegistry interface backed by service discovery, caching, health checks, and load balancing.
Production Considerations
Observability
Emit structured logs, request ids, per-filter latency, route hit counts, status codes, and backend error rates. Logging as a filter keeps this orthogonal.
Security
Validate tokens, normalize headers, strip spoofed internal headers, and avoid revealing route information to unauthorized clients.
Backpressure
Rate limiting should be distributed in production. Use Redis or token buckets so multiple gateway instances enforce the same quota.
Resilience
Wrap BackendService calls with timeouts, retries for safe operations, circuit breakers, and bulkheads so a failing backend does not pin gateway threads.
Configuration safety
Route and filter order should be versioned, validated, and rolled out gradually. A bad rule at the gateway can impact every client.
What Interviewers Look For
Can the candidate clearly separate filter policy from route dispatch?
Do they identify Chain of Responsibility as the main extensibility seam?
Do request and response transformations avoid mutating shared objects?
Does the route abstraction hide matching strategy and backend naming?
Do they explain why the gateway is a proxy and where a singleton registry is only a demo simplification?
Can they discuss production replacements without destroying the LLD boundaries?
Quiz
0/5 answered
1.Why is **FilterChain** better than putting every policy in **ApiGateway.dispatch**?
2.Which filter should usually run before backend dispatch?
3.What does the **Route** matcher strategy buy you?
4.Why is **ApiGateway** a Proxy in this design?
5.What is the main limitation of the singleton **ServiceRegistry** shown here?
Practice Variants
Add a metrics filter
IntermediateImplement MetricsFilter that measures latency around proceed, tags by route or status, and proves routing code stays unchanged.
Per-route filter chains
AdvancedExtend Route to include optional filters for admin-only routes while keeping global filters in front of all traffic.
Trie router
ExpertReplace ordered route scanning with a method plus path trie. Keep the public ApiGateway.handle and filter contracts unchanged.
Flashcards
Cheat Sheet
Core model: Request, Response, Filter, FilterChain, Route, ApiGateway, ServiceRegistry, BackendService.
Flow: handle → auth filter → rate limit filter → logging or transformation filters → dispatch → route match → registry resolve → backend handle → response transformations.
Patterns: Chain of Responsibility for middleware, Strategy for matching and transformations, Proxy for client-facing forwarding, Singleton for the demo registry.
Key invariant: adding a cross-cutting concern must not modify route matching or backend dispatch.
Short-circuits: auth returns 401, rate limit returns 429, no route returns 404, missing backend returns 502.
Scale path: distributed quotas, trie router, injected service discovery registry, backend timeout and circuit-breaker adapters.
References
- BookEnterprise Integration Patterns — Pipes and Filters — Gregor Hohpe and Bobby Woolf
- DocsRefactoring Guru — Chain of Responsibility
- DocsMicrosoft Azure Architecture Center — Gateway Routing pattern
- DocsNGINX Documentation — Reverse Proxy