Documentation
Mbolo is three binaries: mbolo-node runs a node, mbolo-cli talks to it, mbolo-sim replays the test simulator.
Install
The script detects your OS and CPU, downloads the archive from assets.zaleo.dev, checks its SHA-256 sum and copies the binaries to ~/.local/bin.
$ curl -fsSL https://mbolo.zaleo.dev/install.sh | sh $ mbolo-cli --version mbolo-cli 0.1.0
Two variables change what it does: MBOLO_INSTALL_DIR (where to install) and MBOLO_VERSION. By hand:
$ curl -fsSLO https://assets.zaleo.dev/mbolo/0.1.0/mbolo-0.1.0-linux-x86_64.tar.gz $ curl -fsSLO https://assets.zaleo.dev/mbolo/0.1.0/SHA256SUMS $ sha256sum -c --ignore-missing SHA256SUMS mbolo-0.1.0-linux-x86_64.tar.gz: OK
The Linux archives are static (musl), so they run on any distribution. To uninstall, delete ~/.local/bin/mbolo-*.
Start a cluster
--peers lists every node and its peer address; the entry for a node’s own id is where it listens. --client is the client address, --data the log directory.
$ P=1=127.0.0.1:7001,2=127.0.0.1:7002,3=127.0.0.1:7003 $ for i in 1 2 3; do mbolo-node --id $i --peers $P --client 127.0.0.1:800$i --data mbolo/n$i 2>/dev/null & done $ mbolo-cli status $ pkill mbolo-node # stop everything; the data stays in mbolo/
For five nodes, add 4=…:7004,5=…:7005 to P, loop to 5 and give the client all five addresses with MBOLO_NODES. Across machines, swap 127.0.0.1 for real addresses: traffic is neither encrypted nor authenticated, so keep it on a private network.
mbolo-cli
The client tries nodes in order, follows redirects to the leader and resends the same request (same sequence number) on failure, so a write is never applied twice.
| Command | What it does |
|---|---|
get KEY | Linearizable read. Prints the value, or (nil). |
put KEY VALUE | Writes the value. Prints OK once a majority has it. |
cas KEY FROM TO | Compare-and-set. - as FROM means “the key does not exist”. |
status | Role, term, leader, commit, applied and last index, snapshot, fsync count and voters, for each node. |
transfer N | Asks the leader to hand over to node N. |
add N, remove N | Adds or removes a voter (one change at a time). |
--nodes A,B,C | The nodes’ client addresses. Otherwise MBOLO_NODES, otherwise 127.0.0.1:8001-8003. |
If a command fails with “may or may not have been applied”, the cluster did not answer in time: the operation may or may not have happened. Read before retrying.
mbolo-node
| Option | Default | What it does |
|---|---|---|
--id N | This node’s id, as listed in --peers. | |
--peers ID=ADDR,… | Every node and its peer address. | |
--client ADDR | Where clients connect. | |
--data DIR | Write-ahead log (CRC-checked) and snapshots. | |
--read-mode | readindex | log, readindex or lease: how a read proves it is up to date. |
--batch on|off | on | One fsync for every write waiting. |
--pipeline on|off | on | Several AppendEntries in flight per follower. |
--snapshot-every N | 10000 | Snapshot every N applied entries. |
--tick-ms | 10 | Tick length. Election after 15 to 30 ticks, heartbeat every 3. |
--voters 1,2,3 | all | Initial voters; other nodes wait for an add. |
--unsafe-no-fsync | For benchmark comparison only: loses committed writes on a crash. |
Membership changes
Start four nodes with three voters, then add the fourth and remove the first. A second change is refused until the first one commits.
$ P=1=127.0.0.1:7001,2=127.0.0.1:7002,3=127.0.0.1:7003,4=127.0.0.1:7004 $ for i in 1 2 3 4; do mbolo-node --id $i --peers $P --client 127.0.0.1:800$i --data mbolo/n$i --voters 1,2,3 2>/dev/null & done $ export MBOLO_NODES=127.0.0.1:8001,127.0.0.1:8002,127.0.0.1:8003,127.0.0.1:8004 $ mbolo-cli add 4 $ mbolo-cli remove 1
mbolo-sim
The simulator runs the same server code as mbolo-node, with time, network and disk simulated from a seed. A seed describes a whole run, so any failure replays exactly.
$ mbolo-sim --seeds 0..200 --threads 4 seeds: 200 failed: 0 conf changes: 0 wall: 1.1s (174.5 seeds/s) simulated: 3156082 events, 1544507 messages (133341 dropped), 858 crashes (465 torn), 600 partitions, 1076 elections, 162074 client ops ok, 1 indeterminate $ mbolo-sim --seeds 0..2000 --bug read-without-quorum --stop-on-failure FAIL seed 19: t=11999ms: linearizability violation on keys ["k0"] (976 ops) replay: mbolo-sim --seed 19 --bug read-without-quorum --trace
| Option | What it does |
|---|---|
--seed N, --seeds A..B | One seed, or a range. |
--threads T | Seeds in parallel. |
--trace | Prints role changes, faults and client replies. |
--read-mode | log, readindex (default) or lease. |
--drift F | Max clock error per node (0.3 = ±30%). |
--membership | 5 slots, voters {1,2,3}, adds and removes during faults. |
--no-crashes, --no-partitions | Drop one family of faults. |
--duration-ms MS | Simulated length (12,000 by default, faults during the first 8 seconds). |
--bug NAME | Injects a known bug, to check that the tests catch it. |
--stop-on-failure, --json PATH | Stop at the first failure; write the summary as JSON. |
Testing and numbers
After every simulated step, four Raft invariants are checked: one leader per term, log matching, leader completeness, committed entries never change. At the end every client operation is checked for linearizability (Wing-Gong-Lowe search, per key), replicas must converge and the cluster must answer again.
| Campaign | Result |
|---|---|
| Simulation | 30,000 seeds, 0 failures; 135,860 crashes (77,608 in the middle of a write), 89,031 partitions, clock drift up to ±50%. |
| Jepsen-style | 12 runs of 60 s on real processes, 162,028 operations, 58 kill -9, 98 cuts, 7 membership changes: 0 anomalies. |
| Mutations | 7 of 8 planted bugs caught. The figure 8 bug never is, in 20,000 seeds: the new leader’s no-op hides it. |
Performance
3 nodes on an i7-10510U (4 cores, WSL2, ext4 on a virtual disk), every write fsynced on the leader and a follower. Medians of 3 runs.
| Measure | Throughput | p50 | p99 |
|---|---|---|---|
| Writes, 64 clients | 2,295/s | 26 ms | 49 ms |
| Lease reads | 19,656/s | 3 ms | |
| ReadIndex reads | 12,011/s | 4.9 ms | |
| Reads through the log | 1,972/s | 30 ms | |
Failover after kill -9 of the leader, 3 nodes | 268 ms | 636 ms | |
| Failover, 5 nodes | 228 ms |
The published binaries don’t include the Jepsen harness or the benchmarks. The source is private for now; write to [email protected] for access.