Burst it, or sustain it

A token bucketThe standard rate-limit algorithm: a bucket holds up to B tokens; every request takes one. Refill drips back in at rate r. Run dry and you wait. Two numbers โ€” depth and refill โ€” set both the sustained cap and the burst headroom independently. decouples two things people often conflate: how fast you can go sustained, and how big a burst you can swallow. The refill rate sets the long-run cap; the depth says how much spike you can absorb before throttling kicks in.

Holdstokens, refilling atper.
Offered traffic:requests per.
sustained cap 10.0 tok/sec ยท burst headroom 50.0 tokens
3.33 secbefore throttling

At 25.0 req/sec you outrun the refill by 15.0 req/sec โ€” the burst lasts roughly 3.33 sec. After that, you flow at the refill rate (10.0 req/sec); the rest is throttled or queued.

10.0sustained rate /sec
50.0max burst
5.00 secidle to fully refill
Btime โ†’tokens

The bucket starts full and drains at ฮป โˆ’ r = 15.0 tokens/sec; it empties at the marker, and from there everything past the refill rate gets throttled or dropped.

Steady-state token-bucket model. The leaky-bucket sibling is equivalent: same algebra, requests waiting in a queue instead of tokens accumulating.