I reach for determinism when I need replay to mean something. If the same events can rebuild different balances on different nodes, I treat that as a correctness problem, not an operations inconvenience.
Replay should not be a guess
The idea only becomes real for me when I can explain delay, duplication, and loss.
A service that reads random time, hidden external state, or unordered inputs can be hard to replay. A deterministic service records the important inputs and applies them in a known order. That makes test failures, recovery, and audits much easier.
Replay example
My reality check is whether the idea survives an actual workflow, not just a clean diagram.
An account service rebuilds a balance by replaying deposits and withdrawals from the event log. Every replica should end with the same balance after reading the same event sequence.
Deterministic handler code
The code is useful only if it shows the safety check the system depends on.
Replay stays safe when state is rebuilt from ordered facts:
long replayBalance(List<AccountEvent> events) {
long balance = 0;
for (AccountEvent event : events) {
balance += event.amountDelta();
}
return balance;
}The replay function has no clock, random value, or outside call, so the same events always rebuild the same balance. That is the practical test for deterministic recovery.
Sequence diagram: deterministic replay
The replicas match because the inputs and their order are the same.
The repeatability decision
- Record business inputs rather than depending on hidden process state.
- Avoid unordered reads when rebuilding state.
- Make replay a normal test path, not only an incident tool.