Part 13 · 1 chapters · ~12 min
Testing Distributed Systems
Why ordinary tests miss distributed bugs, Jepsen-style history checking against consistency models, deterministic simulation with seeded replay, TLA+ specifications and model checking, property-based tests with shrinking, and a ladder for spending testing effort where failure costs most.
22
Finding bugs that need bad luck
code
// property-based tests with fast-check: the CRDT merge and an idempotent ledger apply
import fc from 'fast-check';
const counter = fc.dictionary(fc.constantFrom('A', 'B', 'C'), fc.nat(100));
test('G-counter merge is commutative, associative and idempotent', () => {
fc.assert(fc.property(counter, counter, counter, (a, b, c) => {
expect(mergeG(a, b)).toEqual(mergeG(b, a));
expect(mergeG(mergeG(a, b), c)).toEqual(mergeG(a, mergeG(b, c)));
expect(mergeG(a, a)).toEqual(a);
}));
});
const transfer = fc.record({ key: fc.uuid(), amount: fc.integer({ min: 1, max: 10_000 }) });
test('applying a log twice gives the same balances', () => {
fc.assert(fc.property(fc.array(transfer), log => {
const once = applyAll(new Ledger(), log);
const twice = applyAll(applyAll(new Ledger(), log), log); // retries, redeliveries
expect(twice.balances()).toEqual(once.balances());
}));
});code
\* a fragment of TLA+: an invariant the model checker tries to break
TypeOK == leader \in [Terms -> Nodes \cup {None}]
OneLeader == \A t \in Terms : Cardinality({n \in Nodes : role[n] = "leader" /\ term[n] = t}) <= 1
Conserved == Sum(balance) = InitialTotalread these
Kyle Kingsbury's Jepsen analyses (jepsen.io), the FoundationDB testing talk ("Testing Distributed Systems w/ Deterministic Simulation"), and "How Amazon Web Services Uses Formal Methods" (2015). Each is short and changes how you think about "it passed the tests".
TESTING DISTRIBUTED SYSTEMS
from unit tests that cannot see the bugs to Jepsen, deterministic simulation and specifications that are checked
swipe the figure sideways, or tap expand for full screen
1/6
why tests miss
Why ordinary tests miss them: a unit test runs one interleaving, the friendly one. The lost-update, split-brain and double-spend bugs need a message delayed past a timeout, a crash between two writes, or a partition healing at the wrong moment. You have to generate those situations on purpose.