June 3, 2024 4 min read A to Z Series: K Home

Key Partitioning: Put Data in the Right Place

Key partitioning maps each record to a shard or partition using a chosen key.

I look at partition keys as a production load decision, not a schema detail. A good key spreads reads and writes; a bad key creates the hot shard that everyone discovers during traffic.

The key carries the load pattern

The pattern earns its place only when it explains the messy network case.

A customer id might spread account reads well. A date key might send every Black Friday order to the same partition. Good partitioning starts with how the system is used, not only with how data is modeled.

Tenant routing example

The point here is to move from concept to operation: who uses it, what breaks, and what decision changes.

An orders API partitions by customer id. Orders for customer 42 go to shard 2, while customer 91 goes to shard 5. Each shard handles a slice of customers.

Partition choice code

The useful code path is the one that makes a safety decision obvious.

The partition key should route the request to the owner shard:

int shardFor(String customerId, int shardCount) {
  return Math.floorMod(customerId.hashCode(), shardCount);
}

int shard = shardFor(order.customerId(), 16);

The shard function makes the placement rule deterministic. Every service can compute the same shard for a customer id without consulting a central router.

Sequence diagram: route by partition key

route by partition key The partition key decides where the request goes and where the load lands. sd route by partition key Orders API Orders API PartitionRouter PartitionRouter Shard Map Shard Map Shard 2 Shard 2 Send customer id 42. Find that customer 42belongs to shard 2. Send the read or write to the owner shard. Return the customer order result.

The partition key decides where the request goes and where the load lands.

The split decision

Next scaling reads

Written by Arunkumar Ganesan.

My recommendation is to test partition keys against real access patterns before the data volume makes the mistake expensive.

#DistributedSystems #Partitioning #DataArchitecture #Scalability