When a node joins, how much moves?

Partition keys across nodes by hashing. With plain key mod NAssign each key to node (key mod N). Fast and simple โ€” but if N changes, almost every key gets a new home. hashing, changing the node count remaps almost every key โ€” a stampede of data movement. Consistent hashingPlace nodes and keys on a virtual ring by hashing their identifiers. Each key belongs to the nearest clockwise node โ€” so adding or removing one node only disturbs its neighbours on the ring. places nodes on a ring so adding or removing one only disturbs its neighbour's share.

Scale fromnodes tonodes, holdingkeys.

1 node added.

20%of keys move

200k of 1M keys shift to a new home with consistent hashing โ€” the minimum possible. Plain mod-4 hashing would move 80% (800k).

200kconsistent โ€” keys moved
800kmod-N โ€” keys moved
+1node added
100%0%
1 nodetarget = 58
consistent hashingmod-N

The consistent line stays low: it moves exactly |ฮ”N| / max(N, M) โ€” the irreducible minimum. The mod-N line runs near the top because changing the divisor reassigns most keys, except in the occasional tidy case where M is a multiple of N and some keys happen to land on the same node by arithmetic luck.

MoveConsistentMod-N
+1 (4โ†’5)now
20%
80%
โˆ’1 (4โ†’3)
25%
75%
ร—2 (4โ†’8)
50%
50%
รท2 (4โ†’2)
50%
50%

Consistent hashing puts nodes and keys on a ring; a new node only steals the arc between it and its successor, so just ~1/M of keys move; a leaving node hands its arc to the next node. Mod-N ties every key's home to the count, so changing the count remaps almost everything โ€” except when the new count is a tidy multiple, when some keys happen to stay. Virtual nodesAssign each physical node multiple positions on the ring โ€” typically 100โ€“200 virtual nodes per physical node. This evens out the load distribution so no one node holds a much larger arc than the others, which is the main weakness of basic consistent hashing with few nodes. even out the per-node load without changing these moved-key fractions.

consistent hashing ยท moves |ฮ”N|/max(N,M) of the keys