APIs, WebJobs & Kafka
Real systems are many services talking to each other — synchronously over APIs and asynchronously over queues and event streams. Senior interviews probe whether you can make that communication resilient: idempotent, rate-limited, versioned, and correct under retries and failures.
This module covers API design for resilience, background processing with WebJobs, and the parts of Kafka that trip people up in interviews — delivery semantics, ordering, and consumer lag.
client -> API (validate, idempotency key, rate limit)
|
+-- write DB
+-- publish event -> Kafka topic (partitioned)
|
v
consumer group -> WebJob / worker
(at-least-once, commit after work)Senior-level focus
- Make write APIs idempotent so retries cannot double-charge or duplicate.
- Protect services with rate limiting and backpressure, not unbounded queues.
- Version APIs so clients never break on a change.
- Understand Kafka delivery, ordering, and rebalancing well enough to reason about correctness.
- Track consumer lag as the health signal for async processing.
Idempotency, rate limiting & versioning
A resilient API survives retries and abuse. Idempotency means the same request applied twice has the same effect once. Rate limiting protects the service from overload. Versioning lets the API evolve without breaking existing clients.
An idempotency key lets the server detect and dedupe a retried write. A rate limiter (token bucket) sheds load with a clear 429 and Retry-After. Additive, backward-compatible versioning keeps old clients working while new fields ship.
request (key=abc) -> seen key abc? no -> do work, store result under abc -> return yes -> return stored result (no duplicate work)
What matters in interviews
- Idempotency key dedupes retried writes — essential for payments.
- Token-bucket rate limiting sheds load with 429 + Retry-After.
- Version additively and keep changes backward compatible.
- Return clear error contracts so clients can retry safely.
Real-world example
A payment request times out and the client retries with the same idempotency key; the server returns the original result and the customer is charged exactly once.
Likely interview questions
- 1.How do you make a payment POST safe to retry?
- 2.How do you evolve an API without breaking existing clients?
Background processing with WebJobs
Not all work belongs in the request path. WebJobs (and similar workers) run background jobs — sending email, processing uploads, reacting to events — so the API responds fast and heavy work happens asynchronously.
The API enqueues a message and returns immediately; a worker consumes it and does the slow work. The queue absorbs spikes, work is retried on failure, and the user-facing latency stays low. Jobs must be idempotent because messages can be delivered more than once.
What matters in interviews
- Move slow work off the request path — enqueue and return fast.
- The queue absorbs spikes and provides retry on failure.
- Workers must be idempotent — messages can be redelivered.
- Use a dead-letter queue for messages that keep failing.
Real-world example
Uploading a video returns instantly; a WebJob transcodes it in the background and the queue smooths out a burst of uploads without overloading the workers.
Likely interview questions
- 1.Where do you draw the line between synchronous and background work?
Kafka: delivery semantics & ordering
Kafka is a partitioned, durable log. Delivery is usually at-least-once (consumers can see a message twice). Ordering is guaranteed only within a partition, and the partition is chosen by the message key.
Because delivery is at-least-once, consumers must be idempotent or use exactly-once semantics for correctness. To keep related events ordered, give them the same key so they land in the same partition. Commit offsets only after the work succeeds so a crash reprocesses rather than loses.
key=user-1 --> partition 0 : e1, e2, e3 (ordered) key=user-2 --> partition 1 : e1, e2 (ordered) no cross-partition ordering guarantee
What matters in interviews
- At-least-once is the default — consumers must be idempotent.
- Ordering holds only within a partition; key by entity to preserve it.
- Commit offsets after work succeeds so crashes reprocess, not drop.
- Exactly-once is possible but has real throughput cost — justify it.
Real-world example
All events for one account share the account id as the key, so they stay ordered in one partition even though the topic is spread across many.
Likely interview questions
- 1.How does Kafka guarantee ordering, and what are the limits?
- 2.How do you achieve exactly-once processing, and is it worth it?
Consumer lag & rebalancing
Consumer lag is how far behind the latest message a consumer group is. Rising lag means consumers cannot keep up. Rebalancing reassigns partitions when consumers join or leave the group.
Lag is the primary health signal for streaming: alert on it and scale consumers (up to the partition count) to catch up. Frequent rebalances stall processing, so keep sessions healthy and avoid long pauses that make a consumer look dead.
What matters in interviews
- Lag is the health metric for async processing — alert and scale on it.
- Max useful parallelism equals the partition count.
- Rebalances pause consumption — minimise churn and long pauses.
- Slow consumers, not the broker, are usually the bottleneck.
Real-world example
A traffic spike pushes consumer lag up; autoscaling adds consumers up to the partition count and the group drains the backlog within minutes.
Likely interview questions
- 1.Consumer lag is growing in production — how do you diagnose and fix it?
Key Takeaways
- Make write APIs idempotent so retries never double-apply.
- Rate-limit and version APIs so services stay protected and clients never break.
- Push slow work to background workers behind a queue that absorbs spikes.
- Kafka is at-least-once with per-partition ordering — key by entity and commit after work.
- Watch consumer lag as the health signal and scale consumers up to the partition count.