July 8, 2024 4 min read A to Z Series: N Home

Network Partitions: Designing for Broken Links

A network partition happens when parts of a distributed system can run but cannot communicate with each other.

I think about network partitions as the moment both sides may still look alive. I recommend writing explicit rules for reads, writes, leadership, and reconciliation before the network proves it can split.

The danger is split truth

When I talk about "The danger is split truth", I am really asking what Network Partitions does when a message is late, duplicated, or missing.

If isolated nodes make independent decisions about the same data, the system may later discover conflicts. A good design names which side can accept writes, which side can serve reads, and how recovery works.

Regional outage example

I use "Regional outage example" to keep Network Partitions grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.

A regional inventory service loses the link to the primary region. It can still show the last known inventory, but it should not promise a scarce item until it can confirm ownership.

Partition response code

When I show "Partition response code", I want the code in Network Partitions to reveal the production decision, not just the syntax.

A partition rule lets safe reads continue while blocking scarce writes:

Response handleInventory(Command command, boolean partitioned) {
  if (partitioned && command.reservesLastItem()) {
    return Response.pending("Reservation waits for primary region");
  }
  return inventory.apply(command);
}

The code allows ordinary inventory work but holds the dangerous last-item reservation during a partition. That protects correctness where overselling would hurt.

Sequence diagram: serve reads and block risky writes

serve reads and block risky writes The system remains useful for safe reads while protecting scarce write decisions. sd serve reads and block risky writes Customer Customer Region B Region B Local Cache Local Cache Ask whether an item isvisible in the catalog. Serve the last knowncatalog read. Try to reserve the lastunit. Reject or mark pendingbecause the primary linkis broken.

The system remains useful for safe reads while protecting scarce write decisions.

The survival choice

Next partition topics

Written by Arunkumar Ganesan.

What I learnt is that a partition is dangerous because the system can keep moving while losing the ability to agree.

#DistributedSystems #NetworkPartitions #Reliability #Architecture