May 21, 2024 4 min read A to Z Series: J Home

Join Protocols: How Nodes Enter a Cluster

A join protocol is the set of steps a new node follows before it becomes a trusted part of the cluster.

I treat node join as more than a deployment event. Before a node receives traffic, I recommend proving who it is, what data it owns, and whether it has caught up enough to serve safely.

Joining should be staged

When I talk about "Joining should be staged", I am really asking what Join Protocols does when a message is late, duplicated, or missing.

A careful join protocol prevents a half ready node from serving wrong answers. The node first discovers peers, downloads configuration, catches up on data, passes health checks, and only then receives traffic.

Node bootstrap example

I use "Node bootstrap example" to keep Join Protocols grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.

A cache cluster adds Node D. It contacts a seed node, receives the current membership list, warms key ranges, then enters the load balancer target set.

Join validation code

When I show "Join validation code", I want the code in Join Protocols to reveal the production decision, not just the syntax.

A node should warm its assigned data before it enters rotation:

void joinCluster(Node node) {
  Membership membership = seedNode.join(node.id());
  node.warmRanges(membership.assignedRanges(node.id()));
  loadBalancer.markReady(node.id());
}

A node is not marked ready just because it contacted the seed. It joins, warms its assigned ranges, and only then enters the load balancer.

Sequence diagram: safe cluster join

safe cluster join The new node becomes visible to clients only after it can answer correctly. sd safe cluster join Node D Node D Seed Node Seed Node Data Owners Data Owners Cluster Manager Cluster Manager Load Balancer Load Balancer Ask to join the cluster. Return membership andassigned key ranges. Copy or warm the data it will serve. Add Node D after healthchecks pass.

The new node becomes visible to clients only after it can answer correctly.

The admission rule

Related cluster topics

Written by Arunkumar Ganesan.

What I learnt is that unsafe joins create some of the hardest bugs because the node looks healthy while serving the wrong slice of truth.

#DistributedSystems #ClusterManagement #Scalability #Architecture