A key-value store that survives failures.
Mbolo replicates every write across 3 or 5 nodes with Raft. Stop a node, cut the network: the cluster keeps answering, and loses nothing.
$ curl -fsSL https://mbolo.zaleo.dev/install.sh | shClick a node to stop or isolate it.
Not an animation: Mbolo’s Raft code, compiled to WebAssembly, runs here inside its own test simulator. Squares are log entries, coloured by term; dashed ones are not committed yet.
Three nodes in two minutes.
Once installed, everything runs on your machine. No root, no Docker.
Start three nodes
Each node talks to its peers on port 700x, to clients on 800x, and keeps its log in mbolo/.
$ P=1=127.0.0.1:7001,2=127.0.0.1:7002,3=127.0.0.1:7003 $ for i in 1 2 3; do mbolo-node --id $i --peers $P --client 127.0.0.1:800$i --data mbolo/n$i 2>/dev/null & done
Write, read
The client finds the leader on its own. status shows who leads and how far each log has got.
$ mbolo-cli put city Dakar OK $ mbolo-cli get city Dakar $ mbolo-cli status | cut -c1-52 node 1: Follower term 1 leader 2 commit 2 node 2: Leader term 1 leader 2 commit 2 node 3: Follower term 1 leader 2 commit 2
Kill the leader
A kill -9, no warning. The other two elect a new leader, and the next write goes through about 268 ms after the crash.
# the leader is node 2 $ pkill -9 -f 'mbolo-node --id 2' $ mbolo-cli put city Thiès OK $ mbolo-cli get city Thiès $ mbolo-cli status | cut -c1-52 node 1: Follower term 2 leader 3 commit 4 node 2: down (Connection refused (os error 111)) node 3: Leader term 2 leader 3 commit 4
Bring it back
It reads its log back from disk, learns who leads, and catches up on what it missed.
$ mbolo-node --id 2 --peers $P --client 127.0.0.1:8002 --data mbolo/n2 2>/dev/null & $ mbolo-cli status | cut -c1-52 node 1: Follower term 2 leader 3 commit 4 node 2: Follower term 2 leader 3 commit 4 node 3: Leader term 2 leader 3 commit 4 $ mbolo-cli get city Thiès
Broken on purpose, thousands of times.
A consensus bug shows up once in a thousand runs. So Mbolo replays thousands of failures, and checks every answer it gave its clients.
135,860 crashes, 89,031 network partitions, drifting clocks. 25.2 million operations checked linearizable.
12 runs on real processes: 58 kill -9, 98 cut TCP links, 7 membership changes.
To test the tests, Raft is broken on purpose. The eighth bug (figure 8 of the paper) slips through: see the limits.
Replay a seed.
A seed decides everything: latencies, failures, clients. The same seed gives the same run, here and in your terminal. At the end the history is checked, and the fingerprint must match.
$ mbolo-sim --seed 18 fingerprint: …
- This seed’s faults will show up here.
Measured on a laptop.
Three nodes on one machine, every write forced to disk (fsync) on the leader and a follower. A 2019 i7 under WSL2, not a server.
With 64 clients, median 26 ms. One fsync covers every write waiting.
From kill -9 of the leader to the first write the next one accepts (median of 40 trials).
What isn’t done.
Mbolo is a serious learning project, not a production database. These are its known limits.
- fsync blocks the event loop.The leader does not write to disk in parallel with replication: a write waits for two fsyncs, one after the other.
- One member at a time.Voters are added and removed one by one, with no joint consensus and no learner phase.
- No formal proof.Random simulation misses the figure 8 bug. A TLA+ model would be the right tool for those corners.
- Jepsen on one machine.A
kill -9doesn’t drop the page cache; the simulator covers torn writes instead. - Lease reads trust the clocks.No stale read at 30 and 50 % drift in the tests: a sample, not a proof.
- Small data.Snapshots go in one message and live in memory. No multi-key transactions.
Download Mbolo 0.1.0.
Three binaries: mbolo-node, mbolo-cli and mbolo-sim. The script checks the SHA-256 sum and installs to ~/.local/bin.
$ curl -fsSL https://mbolo.zaleo.dev/install.sh | sh| Platform | Archive | SHA-256 |
|---|---|---|
| Linux x86_64 | mbolo-0.1.0-linux-x86_64.tar.gz 1.5 MB | 9d21b487806dda84b840f36601e5a70fd2f6b9df989243bcbcdd997599553a7d |
| Linux arm64 (aarch64) | mbolo-0.1.0-linux-aarch64.tar.gz 1.5 MB | a12dd54e0b799bec6a7c12ec0e66efd9e7772bfa4d179b23ae32064396012394 |
| macOS Apple silicon | mbolo-0.1.0-macos-arm64.tar.gz 1.2 MB | 50ada0a1a57735e4e85c64cb3ece15a1c0cfb11f962cc7b359b17879335cb744 |
| macOS Intel | mbolo-0.1.0-macos-x86_64.tar.gz 1.3 MB | f3d6cf786629555232109d065873cfbd71c895ad9327a9bef987d6a2e8f5a200 |
Every command, the node and simulator options, and how the testing works.
Read the docs