You are asked to design a new backend service that will be called very frequently but has strict latency requirements. How would you choose between synchronous HTTP, asynchronous messaging, and caching layers, and what concrete design would you propose?

Question

VirtualInterview.AI · Accepted Answer

I’d start by clarifying the latency target and consistency needs. Let’s say we need p95 under 50ms at 5–10k RPS.

For a user-facing, request/response flow, I’d keep the core service synchronous HTTP, but aggressively minimize what happens on the critical path. Anything non-essential (logging, analytics, email, heavy writes) goes async via a queue like Kafka or SQS.

To actually hit the latency goal, caching is usually the lever that moves the needle. In a similar scenario, we fronted the service with a CDN for public data and added Redis as a read-through cache. Roughly 80% of reads were served from Redis in under 5ms, and we cut database load by ~70%.

Design-wise, I’d propose: API gateway → stateless service (HTTP) → Redis cache → primary datastore, with background workers consuming messages for slow or retryable work, and strict timeouts/budgets on each hop.

Sample Answer

Keywords