A key-value store that survives failures.

Mbolo replicates every write across 3 or 5 nodes with Raft. Stop a node, cut the network: the cluster keeps answering, and loses nothing.

$ curl -fsSL https://mbolo.zaleo.dev/install.sh | sh

Linux and macOS, x86_64 and arm64v0.1.0Read the script

5-node cluster, live
t = 0.00 s
N1followerterm 0N2followerterm 0N3followerterm 0N4followerterm 0N5followerterm 0
N1
N2
N3
N4
N5
Electing the first leader…

Click a node to stop or isolate it.

Not an animation: Mbolo’s Raft code, compiled to WebAssembly, runs here inside its own test simulator. Squares are log entries, coloured by term; dashed ones are not committed yet.

Three nodes in two minutes.

Once installed, everything runs on your machine. No root, no Docker.

1

Start three nodes

Each node talks to its peers on port 700x, to clients on 800x, and keeps its log in mbolo/.

zsh
$ P=1=127.0.0.1:7001,2=127.0.0.1:7002,3=127.0.0.1:7003
$ for i in 1 2 3; do
    mbolo-node --id $i --peers $P --client 127.0.0.1:800$i --data mbolo/n$i 2>/dev/null &
  done
2

Write, read

The client finds the leader on its own. status shows who leads and how far each log has got.

$ mbolo-cli put city Dakar
OK
$ mbolo-cli get city
Dakar
$ mbolo-cli status | cut -c1-52
node 1: Follower  term 1    leader 2     commit 2
node 2: Leader    term 1    leader 2     commit 2
node 3: Follower  term 1    leader 2     commit 2
3

Kill the leader

A kill -9, no warning. The other two elect a new leader, and the next write goes through about 268 ms after the crash.

# the leader is node 2
$ pkill -9 -f 'mbolo-node --id 2'
$ mbolo-cli put city Thiès
OK
$ mbolo-cli get city
Thiès
$ mbolo-cli status | cut -c1-52
node 1: Follower  term 2    leader 3     commit 4
node 2: down (Connection refused (os error 111))
node 3: Leader    term 2    leader 3     commit 4
4

Bring it back

It reads its log back from disk, learns who leads, and catches up on what it missed.

$ mbolo-node --id 2 --peers $P --client 127.0.0.1:8002 --data mbolo/n2 2>/dev/null &
$ mbolo-cli status | cut -c1-52
node 1: Follower  term 2    leader 3     commit 4
node 2: Follower  term 2    leader 3     commit 4
node 3: Leader    term 2    leader 3     commit 4
$ mbolo-cli get city
Thiès

Broken on purpose, thousands of times.

A consensus bug shows up once in a thousand runs. So Mbolo replays thousands of failures, and checks every answer it gave its clients.

30,000
deterministic simulations
0 failures

135,860 crashes, 89,031 network partitions, drifting clocks. 25.2 million operations checked linearizable.

162,028
Jepsen-style operations
0 anomalies

12 runs on real processes: 58 kill -9, 98 cut TCP links, 7 membership changes.

7 of 8
planted bugs caught

To test the tests, Raft is broken on purpose. The eighth bug (figure 8 of the paper) slips through: see the limits.

Replay a seed.

A seed decides everything: latencies, failures, clients. The same seed gives the same run, here and in your terminal. At the end the history is checked, and the fingerprint must match.

$ mbolo-sim --seed 18
fingerprint: …
Seed 18, 12 simulated seconds
t = 0.00 s
N1followerterm 0N2followerterm 0N3followerterm 0
N1
N2
N3
  1. This seed’s faults will show up here.

Measured on a laptop.

Three nodes on one machine, every write forced to disk (fsync) on the leader and a follower. A 2019 i7 under WSL2, not a server.

2,295writes/s

With 64 clients, median 26 ms. One fsync covers every write waiting.

268 ms

From kill -9 of the leader to the first write the next one accepts (median of 40 trials).

Linearizable reads per second

3 nodes, 64 clients. Three ways to prove you read the latest value.

Leader lease19,656/s · 3 ms

No round trip, but assumes clocks are roughly right.

ReadIndex12,011/s · 4.9 ms

One heartbeat round confirms the leader is still the leader.

Through the log1,972/s · 30 ms

The read goes through consensus and the disk, like a write.

Medians of 3 runs. Full measurements are in the docs.

What isn’t done.

Mbolo is a serious learning project, not a production database. These are its known limits.

  • fsync blocks the event loop.The leader does not write to disk in parallel with replication: a write waits for two fsyncs, one after the other.
  • One member at a time.Voters are added and removed one by one, with no joint consensus and no learner phase.
  • No formal proof.Random simulation misses the figure 8 bug. A TLA+ model would be the right tool for those corners.
  • Jepsen on one machine.A kill -9 doesn’t drop the page cache; the simulator covers torn writes instead.
  • Lease reads trust the clocks.No stale read at 30 and 50 % drift in the tests: a sample, not a proof.
  • Small data.Snapshots go in one message and live in memory. No multi-key transactions.

Download Mbolo 0.1.0.

Three binaries: mbolo-node, mbolo-cli and mbolo-sim. The script checks the SHA-256 sum and installs to ~/.local/bin.

$ curl -fsSL https://mbolo.zaleo.dev/install.sh | sh
PlatformArchiveSHA-256
Linux x86_64mbolo-0.1.0-linux-x86_64.tar.gz 1.5 MB9d21b487806dda84b840f36601e5a70fd2f6b9df989243bcbcdd997599553a7d
Linux arm64 (aarch64)mbolo-0.1.0-linux-aarch64.tar.gz 1.5 MBa12dd54e0b799bec6a7c12ec0e66efd9e7772bfa4d179b23ae32064396012394
macOS Apple siliconmbolo-0.1.0-macos-arm64.tar.gz 1.2 MB50ada0a1a57735e4e85c64cb3ece15a1c0cfb11f962cc7b359b17879335cb744
macOS Intelmbolo-0.1.0-macos-x86_64.tar.gz 1.3 MBf3d6cf786629555232109d065873cfbd71c895ad9327a9bef987d6a2e8f5a200

Every command, the node and simulator options, and how the testing works.

Read the docs