API Gateway

Assume 20M daily active users, 250 gateway requests per active user per day, 500 backend microservices, 12,000 configured routes, three active regions, 70 percent authenticated traffic, average request metadata plus small body of 3 KB, average backend response of 8 KB, and 1 KB of structured log data per request.

Use a 10x peak multiplier for diurnal traffic and product launches. Assume a gateway instance can safely sustain 8,000 simple proxy requests per second at 60 percent CPU after reserving headroom for TLS, policy evaluation, and occasional retries.

• Requests: 20M active users times 250 calls per user per day equals 5B gateway requests per day.
• Average QPS: 5B divided by 86,400 seconds is about 57,870 requests per second, rounded to 57,900.
• Peak QPS: a 10x multiplier gives about 579,000 requests per second globally. With three active regions, the steady peak is about 193,000 per region before failover reserves.
• Auth checks: 70 percent authenticated traffic means about 40,500 token or key checks per second on average. Most checks must use local JWT verification, cached public keys, or cached introspection results.
• Rate limits: if the gateway updates both user and tenant buckets, average counter traffic is about 115,800 operations per second. At peak it is about 1.16M operations per second, so the design needs local batching, sharding, or distributed token buckets.
• Config memory: 12,000 routes times 10 KB is about 120 MB of normalized route and policy data. Every data-plane instance can keep this in memory, but rollout and validation should avoid loading broken snapshots.
• Logs: 5B requests times 1 KB is about 5 TB raw logs per day. Compression and sampling help, but security audit events and error logs should remain durable.
• Bandwidth: average response egress is 57,900 times 8 KB, about 463 MB per second. Peak response egress is about 4.6 GB per second, before retries and TLS overhead.
• Fleet: at 8,000 requests per second per instance, 579,000 peak requests need about 73 instances. Add availability-zone, deployment, failover, and noisy-route headroom to plan for 90 to 120 instances globally.

• route_definitions.tenant_id, host_patterns, path_pattern supports route conflict checks during control-plane validation.
• route_definitions.status, version supports snapshot creation and audit diffing.
• consumer_credentials.consumer_id, status supports credential management and revocation workflows.
• config_snapshots.version is unique so gateways can request the next signed bundle by version.
• Data-plane route lookup should use compiled in-memory tries or deterministic finite automata, not database indexes.

A route references one policy chain and one logical upstream service. A consumer credential references a quota plan and scopes. A config snapshot materializes many active route and policy versions into one immutable bundle. Service instance membership comes from service discovery and is intentionally separate from the route database.

For very large multi-tenant platforms, route snapshots can be stored as immutable objects in blob storage with metadata in a key-value database. Rate-limit counters belong in Redis, DynamoDB, Aerospike, or a purpose-built distributed counter store. Access logs, traces, and metrics belong in streaming and observability systems, not the control-plane relational database.

The most important architectural split is between the control plane and the data plane. The data plane is the gateway fleet that handles customer requests, so it must be fast, local, horizontally scalable, and able to continue with the last known good configuration. It should not synchronously call the configuration database, admin API, or service catalog for every request.

The control plane handles humans and automation: route creation, policy authoring, validation, compilation, signing, rollout, rollback, and audit. This plane needs stronger consistency and safety checks. It can use relational transactions, approval workflows, and canary health gates because it is not in the proxy hot path.

A common failure mode is letting control-plane availability determine data-plane availability. The safer design is that gateways subscribe to signed snapshots, verify checksums, load them atomically, and keep the previous version if the new version fails validation or health checks.

Route matching must be deterministic. A good design defines exact host before wildcard host, exact path before prefix path, explicit method before any method, and route priority as the final tie-breaker. The control plane should reject accidental overlaps or require the owner to declare precedence.

Filters should run in a predictable order. Typical request order is normalization, WAF, authentication, authorization, rate limit, request transform, cache lookup, discovery, proxy, response transform, and telemetry. If filters are unordered, teams will create subtle security bugs such as transforming an authorization header before token validation.

Request and response transformation is useful for version translation, header enrichment, mobile payload shaping, and backend-for-frontend aggregation. It must be bounded by payload size, timeout, fanout limits, and schema versioning. Heavy business workflows should live in dedicated backend services, not in gateway scripts.

The gateway should offload common authentication: validate JWT signatures locally, rotate and cache JWKS keys, check API key hashes, support OAuth introspection for opaque tokens, and optionally verify mTLS client certificates. This avoids every microservice implementing the same security plumbing.

Authorization at the gateway should be coarse and route-oriented: required scopes, tenant membership, partner plan, internal versus external caller, and service-to-service identity. Domain-specific checks such as whether a user owns a particular order still belong to the backend service because the gateway should not fetch domain entities or understand business invariants.

Identity propagation must be explicit. Forward signed identity headers only after stripping any client-supplied versions, include tenant and scopes, preserve trace context, and make downstream services trust the gateway only over an authenticated internal network.

Rate limiting protects both the platform and paying customers. Token bucket works well for bursty APIs because it allows short bursts while enforcing average rates. Sliding windows give smoother semantics but can be more expensive. Production systems often combine local per-instance buckets for latency with a shared distributed counter for global quota correctness.

Dimensions matter. Per-IP limits stop anonymous abuse, per-token limits enforce customer plans, per-tenant limits protect noisy neighbors, and per-route limits protect expensive APIs. Return useful 429 responses with reset time and remaining quota where safe.

Caching can reduce backend load, but only for safe responses. Cache keys must include tenant, authorization context, query parameters, content negotiation headers, and any relevant version. Never cache personalized or privileged responses unless the key proves isolation. Cache negative auth or credential lookups briefly to protect identity stores.

The gateway is often the first place to notice backend degradation. Every route needs a request deadline. The gateway should use short connection timeouts, bounded retry attempts, retry budgets, and circuit breakers that open when an upstream is failing. Without this, slow services tie up gateway resources and harm unrelated routes.

Retries are dangerous. Retrying GET may be acceptable, but POST requires idempotency keys or explicit backend support. Retrying after a request reached the service can duplicate side effects. The gateway should avoid retrying on application errors and should never retry so aggressively that it multiplies traffic during an incident.

Backpressure is as important as failure recovery. When queues, connection pools, or worker pools saturate, the gateway should shed load quickly with clear errors rather than allowing latency to grow until clients time out and retry from the outside.

An API Gateway is strategically central, which makes it risky. If every team embeds custom business logic, the gateway becomes a monolith with unclear ownership, slow releases, and broad blast radius. Keep plugins generic, declarative, reviewed, and versioned. Create dedicated backend-for-frontend services when aggregation becomes complex.

Avoid a single point of failure through multi-zone gateway fleets, health-checked load balancers, active-active regional deployment, last-known-good configuration, and independent scaling of control plane, data plane, rate limiter, and telemetry. Roll out gateway binaries and config separately so a route change does not require a full proxy deployment.

Blast-radius controls should exist at multiple levels: per-route circuit breakers, per-tenant quotas, per-cell deployment, per-region failover, and emergency route disablement. The gateway must be powerful enough to protect the platform but constrained enough that one bad plugin or config cannot break every API.

Alternative one is client-side service discovery with no gateway. It removes a hop, but clients must handle auth, routing, retries, and service changes, which is poor for public APIs and mobile apps.

Alternative two is a service mesh only. A mesh is excellent for east-west service-to-service traffic, mTLS, retries, and observability, but it does not fully replace a north-south edge gateway that handles public API products, partners, WAF, quotas, and client-facing transformations.

Alternative three is one backend-for-frontend per client. This is useful for complex aggregation and mobile-specific payloads, but it still benefits from a thinner gateway in front for TLS, authentication, rate limits, and telemetry.

Do not introduce a heavy API Gateway for a small monolith, a few internal services, or a system where managed load balancers and service-level libraries are enough. Also avoid placing deep business workflows in a shared gateway; build a dedicated service when orchestration needs domain data, transactions, or independent releases.