View as Markdown, for AI agents Edit on GitHub

Operator mode

Operator mode lets an agent or a person act on a live simulated cluster, one step at a time, and scores the actions by replaying them. The cluster is an ordinary seedfall run: the same nodes, faults, disks and network. What changes is that something outside the cluster, the operator, can crash, restart, pause, partition and reconfigure nodes while the run is paused between two scheduler steps.

The reward does not come from the live episode. The operator's actions are logged with their virtual times, and verify replays that log from scratch on the episode's seed and on any other seeds the caller names. The reward is the share of those replays that pass. A caller that keeps some of the seeds secret gets a score the operator could not tune to.

Actions

An action is {"at": "2.5s", "kind": "restart", "node": "n2", "args": ["300ms"]}. at is the virtual time the action applies at. A session sets it for you.

Kind Node Args Effect
crash yes none Kills the node's process as a crash does, with the disk model of the config. It stays down until a restart
restart yes optional delay Starts a node that is down after the delay. A node that is running is crashed first, then started after the delay
pause yes none The node stops running until a resume
resume yes none Ends a pause
partition no a duration, then the nodes of one side Cuts the nodes of that side from the rest for the duration
heal no none Ends partitions, clogs and slow nodes now
slow yes a factor of at least 1, then a duration Stretches the node's latencies and timers
clock_jump yes a shift, such as -50ms Moves the node's wall clock
set_args yes the node's whole new argument list Replaces the node's arguments. They apply from its next start, so follow it with restart
upgrade yes none Restarts the node on its upgrade_to build. The config must set upgrade_to for that node
wait no a duration Does nothing and is logged. advance in a session is a wait followed by time passing

crash, restart and upgrade need a node that takes faults, since a crash needs the disk model. Actions are not suppressed by a quiet period or by faults_after_setup, which only silence planned faults. Every action is traced as a line starting with operator, and enters the determinism hash.

Planned faults still happen around the operator's actions. A planned crash does nothing to a node that is already down, including one the operator stopped.

When an action applies

An action with at T applies after every scheduler step at or before T has run, and before the first step after T. A session holds a virtual clock that starts at 0. Advance(d) moves the clock to T+d and runs every step at or before T+d, so everything an action does is applied between steps, never inside one. All actions at the same at apply together, in list order, before any step runs.

This rule is what makes a log replay exactly. A session applies its actions at the pause where the next step is later than the clock. A replay with the same actions in the schedule reaches the same point of the same run, with the same random streams in the same state. Advance must be positive, since an advance of zero would let steps run between two actions of the same time that a replay applies together.

The Go API

Runner.Operate(seed, schedule) returns a *Session paused at virtual time 0. Planned faults come from the seed as in any run.

Call Does
Do(op) Applies the action now. It is checked against the config first
Advance(d) Logs a wait, then lets d of virtual time pass
Observe() Node states and arguments, the trace lines since the last Observe, messages in flight and the client's operation counts. Peek() is the same without the log
Ops() The action log, ready for Schedule.Operator
Finish() Runs to the end and returns the Result: properties, linearizability, failure class and reason, hash

Schedule.Operator carries the same list into Replay, Run and Shrink, and it is saved in schedule files under operator.

The seedfall env protocol

seedfall env [-j N] reads one JSON object per line from stdin and writes one per line to stdout. Every response has ok, and an error when ok is false. A command that fails leaves the episode as it was.

Request Response
{"cmd":"reset","config":"seedfall.json","seed":1} observation. Starts an episode, replacing the one in progress. profile and set work as the flags do. The compiled nodes are kept for the next reset of the same config
{"cmd":"step","op":{"kind":"crash","node":"n1"}} observation after the action
{"cmd":"step","advance":"2s"} observation after the time passed
{"cmd":"observe"} observation, with the log lines since the last one
{"cmd":"state"} observation without the log
{"cmd":"finish"} verdict. Runs to the end
{"cmd":"verify","ops":[...],"seeds":[2,3]} verify. Replays ops on the episode's seed and on each seed listed. Without ops it uses the episode's own log

An observation has at, over, failed and reason, nodes (state of runnable, blocked, paused, down, exited or not started, plus instance, until, args and starts), log, in_flight and client (invoked, ok, failed, unknown, open, counted from the operations the workload records with sim.Invoke).

A verdict has passed, finished, failure and reason, determinism_hash, every properties entry with its hits and true counts, linearizability and ops, the action log. A replay passes when it did not fail and the workload finished. A run that reached the horizon with the workload still going does not pass.

verify returns reward, the share of seeds that passed, passed, and one entry per seed with passed, failure, reason and determinism_hash. The replays run in parallel, -j at a time.

seedfall replay -c seedfall.json -seed N -ops ops.json replays an action log, a JSON array of actions, and prints the trace and hash as any replay does. A failing replay saves its artifacts with the actions in faults.json, so the printed replay command reproduces it.

Over MCP, seedfall_env_reset, seedfall_env_step, seedfall_env_observe, seedfall_env_finish and seedfall_env_verify do the same for one episode held by the server. seedfall_env_verify takes only extra seeds and replays the episode's own log.

The opcluster scenario

examples/etcdraft/opcluster.json is the 3-node etcd-io/raft key-value store of that example, run for an operator. Planned faults are crashes with restarts of 50 to 400ms, and partitions that isolate one node for 100 to 500ms, a crash about every 33s per node and a partition about every 33s. The workload runs for 12s and then checks the versions for up to 3s.

The operator's job is to roll every node to version 2 by the 12s deadline. A node reports the version in its -version argument, which starts at 1, so only set_args and restart on each node changes it. Three clients write and read at about 100 operations a second, and the run fails if:

  • fewer than 40% of the clients' operations succeed in any 2s window after the first (client availability meets the SLO in every window);
  • a node does not report version 2 at the deadline (every node runs the new version by the deadline);
  • the history is not linearizable.
Operator actions in examples/etcdraft/ops Seeds 1 to 21
good.json: set version 2 and restart n1 at 1s, n2 at 3.5s, n3 at 6s, each down for 300ms 21 pass
idle.json: no actions 21 fail on the version at the deadline
restart-two.json: n1 and n2 down together for 4s at 3s, n3 at 8s 21 fail on availability
restart-all.json: all three down together for 4s at 3s 21 fail on availability

good.json also passes on seeds 1 to 41. The first seed is the episode's, the rest are the extra seeds of verify:

cd examples/etcdraft
printf '{"cmd":"reset","config":"opcluster.json","seed":1}\n{"cmd":"verify","ops":%s,"seeds":[2,3,4,5]}\n' "$(cat ops/good.json)" | seedfall env

A session that acts on what it sees gets the same score. The check below drives one by hand on seed 7: wait until no node is down, set the version, restart, advance 2.5s, for each node in turn. Its finish hash, the hash of verify on seed 7 and the hash of seedfall replay -seed 7 -ops on the logged actions are all dcc04807e70edb22.

Measurements

Measured with go test ./cmd/seedfall -bench Env on opcluster.json, which has 3 raft nodes and 3 clients, on a Windows laptop with an Intel Core i5-9300H (4 cores, 8 threads), Go 1.27.1. The nodes are compiled and cached before each measurement. An episode here is a full run of about 15s of virtual time.

Measure Result
reset, a fresh episode from a cached config 188ms, 80MB allocated
step with an action 22µs
step with an advance of 100ms of virtual time 63ms
Episodes a second at -j 1 0.129 (7.7s each)
Episodes a second at -j 8 0.734 (8 episodes in 10.9s)
Heap per open episode, 3s of virtual time in 38.7MiB with 1 open, 38.7MiB with 8 open
Heap per open episode, run to its end 42.9MiB with 1 open, 46.6MiB with 8 open

The time goes to the raft nodes and clients, not to the environment: an action costs microseconds. Eight jobs give 5.7 times the throughput of one on this machine. Heap is what Go reports in use after a collection, measured as the growth from before the episodes opened, so it excludes the compiled nodes.

Limits

  • One episode runs at a time per seedfall env process and per MCP server. reset replaces it.
  • The operator sees the trace, the node states and the client's operation counts, not the nodes' memory or disks.
  • An action list replays against other seeds at the same virtual times. It cannot react to what a different seed does, so a good list leaves room for the planned faults.
  • restart and upgrade are the same action for a node that has upgrade_to: as with a crash, the first restart runs the upgraded build.
  • A session pauses only between scheduler steps, so an action never lands inside a guest's step.
  • verify runs every seed to the end, so its cost is the number of seeds times the episode time above.