ZooKeeper: Coordination for Distributed Systems

I recommend thinking of ZooKeeper as a careful coordination layer, not a place for high volume business data. I use it for small facts that many processes need to agree on.

Coordination data should be small and important

When I talk about "Coordination data should be small and important", I am really asking what ZooKeeper does when a message is late, duplicated, or missing.

Systems use ZooKeeper to decide who is leader, where configuration lives, and which workers are active. Its value comes from giving processes a shared, ordered view of coordination state.

Invoice leader example

I use "Invoice leader example" to keep ZooKeeper grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.

Picture a finance platform with three invoice reconciliation workers running in different availability zones. Every ten minutes, one worker must compare settled payments with ledger entries and create missing adjustments. If two workers run the same batch, the system can create duplicate adjustments, noisy alerts, and support work that should never exist.

ZooKeeper can own only the coordination part. Each worker creates an ephemeral sequential node under an election path. The worker with the lowest sequence number becomes the leader for that batch. Other workers watch the node immediately before their own. If the leader process dies or loses its session, ZooKeeper removes the ephemeral node and the next worker takes over.

The business job should still be idempotent with a batch id. ZooKeeper decides who may run the batch. The ledger still protects what has already been written.

Code sketch

When I show "Code sketch", I want the code in ZooKeeper to reveal the production decision, not just the syntax.

In Java, most teams avoid writing raw ZooKeeper recipes by hand and use Apache Curator. This is the shape of the reconciliation worker:

import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.leader.LeaderSelector;
import org.apache.curator.framework.recipes.leader.LeaderSelectorListenerAdapter;

final class ReconciliationLeader implements AutoCloseable {
  private final LeaderSelector selector;
  private final LedgerReconciliationJob job;

  ReconciliationLeader(CuratorFramework client, LedgerReconciliationJob job) {
    this.job = job;
    this.selector = new LeaderSelector(
        client,
        "/reconcile/leader",
        new LeaderSelectorListenerAdapter() {
          @Override
          public void takeLeadership(CuratorFramework client) throws Exception {
            String batchId = job.nextBatchId();
            job.reconcileOnce(batchId);
          }
        });
    this.selector.autoRequeue();
  }

  void start() {
    selector.start();
  }

  @Override
  public void close() {
    selector.close();
  }
}

The important line is not the class name. It is the boundary: leadership chooses one worker, while reconcileOnce(batchId) must still be safe to retry.

The coordination boundary

• Keep only ownership and coordination state in ZooKeeper. Put invoices, payments, and ledger rows in the business data store.
• Use ephemeral sequential nodes for leader election so a dead worker does not keep ownership forever.
• Make the business operation idempotent with a batch id because coordination reduces duplicates but should not be the only safety net.

Next coordination reads

• Microservices Are Not the Default Answer
• Use the Ambassador Pattern When Connectivity Should Move Out of the App
• Distributed Computing Jargon: An A to Z Glossary