June 3, 2024 4 min read A to Z Series: K Home

Key Partitioning: Put Data in the Right Place

Key partitioning maps each record to a shard or partition using a chosen key.

I look at partition keys as a production load decision, not a schema detail. A good key spreads reads and writes; a bad key creates the hot shard that everyone discovers during traffic.

The key carries the load pattern

When I talk about "The key carries the load pattern", I am really asking what Key Partitioning does when a message is late, duplicated, or missing.

A customer id might spread account reads well. A date key might send every Black Friday order to the same partition. Good partitioning starts with how the system is used, not only with how data is modeled.

Tenant routing example

I use "Tenant routing example" to keep Key Partitioning grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.

An orders API partitions by customer id. Orders for customer 42 go to shard 2, while customer 91 goes to shard 5. Each shard handles a slice of customers.

Partition choice code

When I show "Partition choice code", I want the code in Key Partitioning to reveal the production decision, not just the syntax.

The partition key should route the request to the owner shard:

int shardFor(String customerId, int shardCount) {
  return Math.floorMod(customerId.hashCode(), shardCount);
}

int shard = shardFor(order.customerId(), 16);

The shard function makes the placement rule deterministic. Every service can compute the same shard for a customer id without consulting a central router.

Sequence diagram: route by partition key

route by partition key The partition key decides where the request goes and where the load lands. sd route by partition key Orders API Orders API PartitionRouter PartitionRouter Shard Map Shard Map Shard 2 Shard 2 Send customer id 42. Find that customer 42belongs to shard 2. Send the read or write to the owner shard. Return the customer order result.

The partition key decides where the request goes and where the load lands.

The split decision

Next scaling reads

Written by Arunkumar Ganesan.

My recommendation is to test partition keys against real access patterns before the data volume makes the mistake expensive.

#DistributedSystems #Partitioning #DataArchitecture #Scalability