seedfall: crash testing for Go services

Find the bugs that only appear when a service crashes at the wrong moment, and replay each one exactly. By Kunal Roy Choudhury.

The problem

Databases, queues and caches that write to disk usually pass their tests. Then, in production, one of them loses data it already told a client was saved. The cause is rarely a logic error. It is a crash at one moment: halfway through a write, or after a rename and before the directory was synced to disk. Normal tests almost never hit that moment, and when production does, nobody can reproduce it.

How it works

seedfall compiles each node of your service to WebAssembly (Go's GOOS=wasip1 target) and runs it on wazero. On that target everything nondeterministic reaches the outside world through three calls: random bytes, the clock and sleeping. seedfall implements all three, and it simulates the disk and the network, so your code keeps using plain os.File and net.

The clock is virtual, so ten seconds of time.Sleep cost no real time. The disk has an explicit durability model: a write is visible at once but survives a crash only after fsync, and a crash can keep a torn prefix of the pending writes. The network can partition, delay, reorder and reset.

A run is named by a seed, which is just a number. Scheduling, fault timing, disk tearing and latency all draw from streams seeded from it, so the same seed gives the same trace byte for byte on any machine. When a run breaks one of your checks, seedfall shrink replays it with fewer faults and keeps a change only while it still fails the same way.

examples/kvlog is a small key-value server that syncs before it answers "saved":

$ seedfall run -c seedfall.always.json -seeds 1-100
FAIL seed 1 at t=2.891820635s (737 scheduler steps, 445ms real)
  client#1: acknowledged write lost: k96=v96, server now answers "NONE"

$ seedfall shrink -c seedfall.always.json -seed 1
faults before the failure (2):
  427.268ms  crash server during disk op #56, restart at 761.268ms
     802ms  crash server, restart at 802ms

Seed 1 had 33 faults. Shrinking cut it to 2 in 16 runs. The first crash leaves a half-written line, the server appends after it, and the second restart cannot read the record that follows.

What it found

I ran it on real open-source Go storage projects. Status as of 2026-11-09; not all of it is accepted upstream.

A strict file system is one where a created or renamed file survives a crash only after its directory is synced. hashicorp/raft (600 seeds) and etcd-io/raft (100 seeds) pass cleanly. I also asked Claude for a crash-safe key-value store nine times with one prompt. Every sample's own tests passed, and seedfall found two that can refuse to open forever after a specific crash. Details are in the findings.

Seeds as a grader

The properties that make a seed useful to a person also make it a good reward function for an agent. An agent cannot tune its actions to a seed it never sees, so a score over secret seeds measures behavior that holds up under faults it did not predict. A replay is exact, so anyone can recompute a score and replay a disputed one. And a failure shrinks to the few events that caused it, which is feedback an agent or a person can read. An agent that does see a seed can overfit to it, so the setup is a few visible seeds and some kept secret.

Operator mode

In operator mode an agent acts on a live simulated cluster, one step at a time: crash, restart, pause, partition or reconfigure a node, between two scheduler steps. Every action is logged with its virtual time. The reward comes from replaying that log from scratch on the episode's seed and on other seeds the caller names, and it is the share of replays that pass. The interface is seedfall env, one JSON object per line, with reset, step, observe, finish and verify. The same five operations exist as MCP tools.

{"cmd":"reset","config":"opcluster.json","seed":1}
{"cmd":"step","advance":"1s"}
{"cmd":"step","op":{"kind":"set_args","node":"n1","args":["-id","1","-version","2"]}}
{"cmd":"step","op":{"kind":"restart","node":"n1","args":["300ms"]}}
{"cmd":"verify","seeds":[2,3,4,5]}

The scenario is a 3-node etcd-io/raft key-value store with three clients, under planned crashes and partitions. The operator must roll every node to version 2 by 12s. The run fails if client availability drops under 40% in any 2s window, if a node is not on version 2 at the deadline, or if the history is not linearizable. I wrote four action lists and ran each on seeds 1 to 21:

ActionsResult
good: restart n1, n2, n3 one at a time, 300ms down each21 pass
idle: no actions21 fail on version
restart-two: n1 and n2 down together for 4s21 fail on availability
restart-all: all three down together for 4s21 fail on availability

The grader tells a careful rolling restart from doing nothing and from restarting too much at once, the same way on every seed. On a laptop with an Intel Core i5-9300H, reset takes 188ms and a step with an action 22µs. A full episode takes 7.7s, and eight in parallel run at 0.734 episodes a second. The time goes to the raft nodes, not the environment. The protocol and every action are in the operator mode docs.

Limits

Links

GitHub, Apache-2.0. Getting started. How it works. Findings. Operator mode. All docs.

Kunal Roy Choudhury