Documentation

Mbolo is three binaries: mbolo-node runs a node, mbolo-cli talks to it, mbolo-sim replays the test simulator.

Install

The script detects your OS and CPU, downloads the archive from assets.zaleo.dev, checks its SHA-256 sum and copies the binaries to ~/.local/bin.

$ curl -fsSL https://mbolo.zaleo.dev/install.sh | sh
$ mbolo-cli --version
mbolo-cli 0.1.0

Two variables change what it does: MBOLO_INSTALL_DIR (where to install) and MBOLO_VERSION. By hand:

$ curl -fsSLO https://assets.zaleo.dev/mbolo/0.1.0/mbolo-0.1.0-linux-x86_64.tar.gz
$ curl -fsSLO https://assets.zaleo.dev/mbolo/0.1.0/SHA256SUMS
$ sha256sum -c --ignore-missing SHA256SUMS
mbolo-0.1.0-linux-x86_64.tar.gz: OK

The Linux archives are static (musl), so they run on any distribution. To uninstall, delete ~/.local/bin/mbolo-*.

Start a cluster

--peers lists every node and its peer address; the entry for a node’s own id is where it listens. --client is the client address, --data the log directory.

$ P=1=127.0.0.1:7001,2=127.0.0.1:7002,3=127.0.0.1:7003
$ for i in 1 2 3; do
    mbolo-node --id $i --peers $P --client 127.0.0.1:800$i --data mbolo/n$i 2>/dev/null &
  done
$ mbolo-cli status
$ pkill mbolo-node        # stop everything; the data stays in mbolo/

For five nodes, add 4=…:7004,5=…:7005 to P, loop to 5 and give the client all five addresses with MBOLO_NODES. Across machines, swap 127.0.0.1 for real addresses: traffic is neither encrypted nor authenticated, so keep it on a private network.

mbolo-cli

The client tries nodes in order, follows redirects to the leader and resends the same request (same sequence number) on failure, so a write is never applied twice.

CommandWhat it does
get KEYLinearizable read. Prints the value, or (nil).
put KEY VALUEWrites the value. Prints OK once a majority has it.
cas KEY FROM TOCompare-and-set. - as FROM means “the key does not exist”.
statusRole, term, leader, commit, applied and last index, snapshot, fsync count and voters, for each node.
transfer NAsks the leader to hand over to node N.
add N, remove NAdds or removes a voter (one change at a time).
--nodes A,B,CThe nodes’ client addresses. Otherwise MBOLO_NODES, otherwise 127.0.0.1:8001-8003.

If a command fails with “may or may not have been applied”, the cluster did not answer in time: the operation may or may not have happened. Read before retrying.

mbolo-node

OptionDefaultWhat it does
--id NThis node’s id, as listed in --peers.
--peers ID=ADDR,…Every node and its peer address.
--client ADDRWhere clients connect.
--data DIRWrite-ahead log (CRC-checked) and snapshots.
--read-modereadindexlog, readindex or lease: how a read proves it is up to date.
--batch on|offonOne fsync for every write waiting.
--pipeline on|offonSeveral AppendEntries in flight per follower.
--snapshot-every N10000Snapshot every N applied entries.
--tick-ms10Tick length. Election after 15 to 30 ticks, heartbeat every 3.
--voters 1,2,3allInitial voters; other nodes wait for an add.
--unsafe-no-fsyncFor benchmark comparison only: loses committed writes on a crash.

Membership changes

Start four nodes with three voters, then add the fourth and remove the first. A second change is refused until the first one commits.

$ P=1=127.0.0.1:7001,2=127.0.0.1:7002,3=127.0.0.1:7003,4=127.0.0.1:7004
$ for i in 1 2 3 4; do
    mbolo-node --id $i --peers $P --client 127.0.0.1:800$i --data mbolo/n$i --voters 1,2,3 2>/dev/null &
  done
$ export MBOLO_NODES=127.0.0.1:8001,127.0.0.1:8002,127.0.0.1:8003,127.0.0.1:8004
$ mbolo-cli add 4
$ mbolo-cli remove 1

mbolo-sim

The simulator runs the same server code as mbolo-node, with time, network and disk simulated from a seed. A seed describes a whole run, so any failure replays exactly.

$ mbolo-sim --seeds 0..200 --threads 4
seeds: 200  failed: 0  conf changes: 0  wall: 1.1s  (174.5 seeds/s)
simulated: 3156082 events, 1544507 messages (133341 dropped), 858 crashes (465 torn), 600 partitions, 1076 elections, 162074 client ops ok, 1 indeterminate
$ mbolo-sim --seeds 0..2000 --bug read-without-quorum --stop-on-failure
FAIL seed 19: t=11999ms: linearizability violation on keys ["k0"] (976 ops)
  replay: mbolo-sim --seed 19 --bug read-without-quorum --trace
OptionWhat it does
--seed N, --seeds A..BOne seed, or a range.
--threads TSeeds in parallel.
--tracePrints role changes, faults and client replies.
--read-modelog, readindex (default) or lease.
--drift FMax clock error per node (0.3 = ±30%).
--membership5 slots, voters {1,2,3}, adds and removes during faults.
--no-crashes, --no-partitionsDrop one family of faults.
--duration-ms MSSimulated length (12,000 by default, faults during the first 8 seconds).
--bug NAMEInjects a known bug, to check that the tests catch it.
--stop-on-failure, --json PATHStop at the first failure; write the summary as JSON.

Testing and numbers

After every simulated step, four Raft invariants are checked: one leader per term, log matching, leader completeness, committed entries never change. At the end every client operation is checked for linearizability (Wing-Gong-Lowe search, per key), replicas must converge and the cluster must answer again.

CampaignResult
Simulation30,000 seeds, 0 failures; 135,860 crashes (77,608 in the middle of a write), 89,031 partitions, clock drift up to ±50%.
Jepsen-style12 runs of 60 s on real processes, 162,028 operations, 58 kill -9, 98 cuts, 7 membership changes: 0 anomalies.
Mutations7 of 8 planted bugs caught. The figure 8 bug never is, in 20,000 seeds: the new leader’s no-op hides it.

Performance

3 nodes on an i7-10510U (4 cores, WSL2, ext4 on a virtual disk), every write fsynced on the leader and a follower. Medians of 3 runs.

MeasureThroughputp50p99
Writes, 64 clients2,295/s26 ms49 ms
Lease reads19,656/s3 ms
ReadIndex reads12,011/s4.9 ms
Reads through the log1,972/s30 ms
Failover after kill -9 of the leader, 3 nodes268 ms636 ms
Failover, 5 nodes228 ms

The published binaries don’t include the Jepsen harness or the benchmarks. The source is private for now; write to [email protected] for access.