April 8, 2024 4 min read A to Z Series: F Home

Fencing Tokens: Stop the Old Owner

A fencing token is a steadily increasing value that lets a resource reject work from an old owner.

I use fencing tokens when a lock alone is not enough. A paused process can wake up and still believe it owns the resource, so I recommend giving every owner a token that downstream systems can compare.

The resource must enforce ownership

When I talk about "The resource must enforce ownership", I am really asking what Fencing Tokens does when a message is late, duplicated, or missing.

A lock service can issue token 17 to one worker and token 18 to the next worker. The storage layer accepts writes only from the highest valid token it has seen. That way, the old worker cannot corrupt state after it wakes up.

Lease example

I use "Lease example" to keep Fencing Tokens grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.

A job runner writes reports to shared storage. Worker A pauses after receiving token 17. Worker B receives token 18 and starts writing. When Worker A wakes up, storage rejects token 17.

Fenced write code

When I show "Fenced write code", I want the code in Fencing Tokens to reveal the production decision, not just the syntax.

The shared resource rejects stale owners by comparing tokens:

void writeReport(long token, Report report) {
  if (token < highestTokenSeen.get()) {
    throw new StaleOwnerException();
  }
  highestTokenSeen.set(token);
  storage.save(report);
}

The write is allowed only when the token is at least as new as the highest token already seen. That prevents an old leader from writing after a newer owner took over.

Sequence diagram: reject the stale owner

reject the stale owner The important enforcement happens at the shared resource, not only at the lock service. sd reject the stale owner Worker A Worker A Lock Service Lock Service Shared Storage Shared Storage Worker B Worker B Acquire ownership andreceive token 17. Pause before finishing the write. Acquire ownership and receive token 18. Resume with token 17 and get rejected.

The important enforcement happens at the shared resource, not only at the lock service.

The safety check

Related coordination topics

Written by Arunkumar Ganesan.

What I learnt is that the token matters because the dangerous actor is often the old owner, not a stranger.

#DistributedSystems #FencingTokens #Concurrency #Reliability