I treat membership as the foundation under quorum, leader election, shard ownership, and failover. If the system cannot agree who is in the cluster, I do not trust the decisions built on top.
The list must change safely
When I talk about "The list must change safely", I am really asking what Membership does when a message is late, duplicated, or missing.
A membership change is not just a list update. If half the system thinks there are three nodes and the other half thinks there are five, quorum and routing decisions can become dangerous.
Service registry example
I use "Service registry example" to keep Membership grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.
A database cluster removes a failed replica. The cluster updates membership before recalculating quorum so reads and writes know how many acknowledgements are required.
Membership update code
When I show "Membership update code", I want the code in Membership to reveal the production decision, not just the syntax.
Membership changes should update routing from one agreed view:
Set<Node> activeMembers(MembershipView view) {
return view.nodes().stream()
.filter(Node::healthy)
.collect(Collectors.toUnmodifiableSet());
}The function builds the usable membership view by filtering out unhealthy nodes. Callers should route work to this view, not to every node ever seen.
Sequence diagram: membership update before routing changes
Routing changes after the cluster agrees on who belongs to the cluster.
The source of truth
- Treat membership as state that needs correctness.
- Make membership changes visible in logs and dashboards.
- Avoid independent local membership decisions for critical writes.