March 14, 2024 4 min read A to Z Series: D Home

Determinism: Same Inputs, Same Outcome

Determinism means the same ordered inputs produce the same result, no matter which node runs the code.

I reach for determinism when I need replay to mean something. If the same events can rebuild different balances on different nodes, I treat that as a correctness problem, not an operations inconvenience.

Replay should not be a guess

When I talk about "Replay should not be a guess", I am really asking what Determinism does when a message is late, duplicated, or missing.

A service that reads random time, hidden external state, or unordered inputs can be hard to replay. A deterministic service records the important inputs and applies them in a known order. That makes test failures, recovery, and audits much easier.

Replay example

I use "Replay example" to keep Determinism grounded in a real system, because abstract patterns are too easy to agree with and too hard to operate.

An account service rebuilds a balance by replaying deposits and withdrawals from the event log. Every replica should end with the same balance after reading the same event sequence.

Deterministic handler code

When I show "Deterministic handler code", I want the code in Determinism to reveal the production decision, not just the syntax.

Replay stays safe when state is rebuilt from ordered facts:

long replayBalance(List<AccountEvent> events) {
  long balance = 0;
  for (AccountEvent event : events) {
    balance += event.amountDelta();
  }
  return balance;
}

The replay function has no clock, random value, or outside call, so the same events always rebuild the same balance. That is the practical test for deterministic recovery.

Sequence diagram: deterministic replay

deterministic replay The replicas match because the inputs and their order are the same. sd deterministic replay Event Log Event Log Replica A Replica A Replica B Replica B State Store State Store Send event 1, event 2, andevent 3 in order. Send the same events in the same order. Apply the events and produce balance 120. Apply the events andproduce balance 120.

The replicas match because the inputs and their order are the same.

The repeatability decision

Related system design notes

Written by Arunkumar Ganesan.

My recommendation is to make every replay path boring before the incident, because replay is usually needed when people are already under pressure.

#DistributedSystems #Determinism #EventDrivenArchitecture #SoftwareEngineering