What's a cache really worth?

A cache hit is cheap; a miss pays full price. Your effective latency — or cost — is just the blend of the two, weighted by how often you hit. The catch is Amdahl's Law in disguise: the misses you can't eliminate play the role of the serial fraction, so they cap the win however fast your hits get. A 90% hit rate can't beat 10×, no matter how quick the cache.

The cache
A cachehitA hit is served straight from the cache — the fast, cheap path. This is the latency a cached request returns in.returns inms, amissA miss isn't in the cache, so the request falls through to the slow source — a database, an API, a recomputation — and pays full latency.inms.
The cachehitsThe share of requests served from the cache. The single most important number here — and the one that's hardest to push toward 100%.% of the time.
5.9 ms effective latency · down from 50 ms uncached · 8.5× faster
The win curve
50×
0%20%40%60%80%90%95%
perfect cache (1 ÷ miss rate)this cache
5.9 mseffective latency

At 90% hits a request averages 5.9 ms, down from 50 ms uncached — 8.5× faster, against a floor of 1 ms.

8.5×faster now
10×miss-rate ceilingThe hard cap your miss rate sets: 1 ÷ (1−hit rate). Even a free, instant cache can't beat it — the requests you can't cache still cost full price. This is the 'serial fraction' of Amdahl's Law.
50×cache ceilingThe most this cache could ever give, at a 100% hit rate: miss ÷ hit. Set by how much faster (or cheaper) the hit path is than a miss.
At 90% hits the misses are the ceiling: even an instant cache caps the win at 10×. The 10% you can't cache dominate the average — to win more you have to cut the miss rate, not speed up hits.
As the hit rate climbs
Hit rateEffective latencyFaster
50%
25.5 ms
2×
80%
10.8 ms
4.6×
90%you
5.9 ms
8.5×
95%
3.45 ms
14×
99%
1.49 ms
34×
99.9%
1.05 ms
48×

The early gains are enormous — the first jump off a cold cache buys the most — then they flatten as the curve nears the cache ceiling. And each extra nine of hit rate is exponentially harder to win: more memory, more careful keys, longer TTLs. The misses you can never kill set the floor on effective latency, which is why past some point a faster miss path beats chasing a higher hit rate.

effective = hit × hit-cost + miss × miss-cost · Amdahl's Law for caches · see Amdahl's Law