How many connections do you need?

A connection β€” or thread, or worker β€” is held for the length of a request, so the number busy at once is just throughput Γ— latency (Little's Law). That's the floor. But size the pool to that average and it sits at 100% busy β€” the queue cliff. Add headroom so a burst has somewhere to go.

Your load & target
Serverequests/sec that each hold a connection forms, keeping the pool under% busy.
27connections

At 500 req/s Γ— 40 ms, about 20 are busy on average (Little's Law: Ξ» Γ— W). Sizing to 27 keeps the pool 74% busy at target load, with 7 spare β€” room to absorb a burst up to 1.35Γ— (675 req/s) before it saturates.

20busy on averageL = λ·W, the average number in flight. The hard floor on pool size β€” go below it and requests must wait for a slot even with zero variance.
27recommended pool
1.35Γ—burst headroomHow far load can spike above target before the pool hits 100% and requests start queueing. It's exactly 1 / target-utilization.
Utilization at 500 req/s vs. pool size
100%40%
20 (floor)pool = 2750

At the bare floor (20) the pool is pinned at 100% β€” in the amber zone, where the smallest burst queues. Each connection you add buys headroom, fast at first then with diminishing returns. The recommended pool lands you at your target, clear of the cliff.

What headroom costs
Target busyPool size
50%burst 2Γ—
40
65%burst 1.55Γ—
31
80%burst 1.25Γ—
25
95%burst 1.1Γ—
22

Lower the target utilization and the pool grows β€” but so does the burst it can swallow. Around 70–80% is the usual sweet spot: comfortable headroom without paying for connections that sit idle. Variable latency or spiky arrivals push you lower.

Connection / thread pool sizing Β· pool = ⌈λ·W / utilizationβŒ‰