I treat membership as the foundation under quorum, leader election, shard ownership, and failover. If the system cannot agree who is in the cluster, I do not trust the decisions built on top.
The list must change safely
I use this section to force the failure case into the conversation, not just the clean path.
A membership change is not just a list update. If half the system thinks there are three nodes and the other half thinks there are five, quorum and routing decisions can become dangerous.
Service registry example
I keep the example grounded in production behavior: owner, delay, retry, and recovery.
A database cluster removes a failed replica. The cluster updates membership before recalculating quorum so reads and writes know how many acknowledgements are required.
Membership update code
I use the code here to show the decision boundary, not just syntax.
Membership changes should update routing from one agreed view:
Set<Node> activeMembers(MembershipView view) {
return view.nodes().stream()
.filter(Node::healthy)
.collect(Collectors.toUnmodifiableSet());
}The function builds the usable membership view by filtering out unhealthy nodes. Callers should route work to this view, not to every node ever seen.
Sequence diagram: membership update before routing changes
Routing changes after the cluster agrees on who belongs to the cluster.
The source of truth
- Treat membership as state that needs correctness.
- Make membership changes visible in logs and dashboards.
- Avoid independent local membership decisions for critical writes.