A load balancer that assigns work blindly lets hotspots build: the fleet looks fine on average while one unlucky queue quietly stacks up โ and that queue is exactly where your slowest requests are sitting. Scanning every server for the shortest queue fixes it, at the price of full information on every arrival. The surprising middle ground: peek at just two random servers and join the shorter queue. That single extra look collapses the long queues exponentially and lands almost on the full-scan ideal โ one peek is worth nearly all the information. All three policies race here on the same simulated arrivals.
| Policy | Mean wait | p95 | p99 |
|---|---|---|---|
Random pick 1 blind | 876ms | 2.9s | 4.3s |
Two choices peek at 2, join the shorter | 200ms | 604ms | 883ms |
Shortest scan all 10 | 87ms | 366ms | 589ms |
The mean is the least of it โ the tail is where blind assignment hurts. Random's p99The 99th-percentile wait: the wait the slowest 1% of jobs see. In a fan-out system the tail, not the mean, is what users actually feel. is set by whichever queue happened to run hot, and the average hides it. Checking two flattens that tail exponentially โ the celebrated result of Azar, Broder, Karlin & Upfal (1994), made famous by Mitzenmacher's thesis โ and real systems ship exactly this: NGINX's random-with-two-choices and Envoy's least-request both peek at two servers and take the less busy one.