July 30, 2024 4 min read A to Z Series: P Home

Partition Tolerance: Keep Rules During a Split

Partition tolerance is the ability to behave deliberately when messages between nodes are delayed, dropped, or cut off.

I do not read partition tolerance as a promise that everything stays perfect. I read it as a reminder to define safe behavior when links break and both sides still want to continue.

A split needs policy, not hope

When I talk about "A split needs policy, not hope", I am really asking what Partition Tolerance does when a message is late, duplicated, or missing.

When communication breaks, services still receive user traffic. The design must decide whether to serve old reads, accept writes, queue work, or refuse certain operations.

Cross zone example

I use "Cross zone example" to keep Partition Tolerance grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.

A wallet service cannot reach enough replicas to confirm a withdrawal. It allows balance reads from cache but blocks the withdrawal until quorum is reachable.

Partition aware code

When I show "Partition aware code", I want the code in Partition Tolerance to reveal the production decision, not just the syntax.

The service keeps low risk reads open and closes high risk writes:

WalletResponse handle(WalletCommand command, boolean quorumReachable) {
  if (!quorumReachable && command.isWithdrawal()) {
    return WalletResponse.rejected("Quorum unavailable");
  }
  return wallet.execute(command);
}

The wallet rejects withdrawals when quorum is unavailable, because accepting them during a partition could spend money the system cannot safely verify.

Sequence diagram: tolerate the split with a safe rule

tolerate the split with a safe rule The service tolerates the partition by keeping safe operations available and unsafe operations closed. sd tolerate the split with a safe rule Customer Customer Wallet API Wallet API Local Replica Local Replica Request current balance. Read the last confirmedbalance. Request a withdrawal. Reject the withdrawalbecause quorum isunavailable.

The service tolerates the partition by keeping safe operations available and unsafe operations closed.

The failure assumption

Related resilience topics

Written by Arunkumar Ganesan.

What I learnt is that partition tolerance is a set of rules for a bad day, not a marketing property of a database.

#DistributedSystems #PartitionTolerance #CAPTheorem #Architecture