Recovery and the journal
The journal records runs, in-flight changes, uncertain outcomes, and pending refreshes. It exists to finish interrupted work. It is not an inventory of what Rivet manages: every run re-inspects the host instead of trusting the journal.
Inspect it
This prints unresolved intents and refreshes as JSON. It opens the journal read-only and doesn’t create one if none exists. After a clean run, the output is [].
Don’t delete the journal to force a clean run. Check what actually happened on the host, then fix the configuration or the host if needed.
What happens after a crash
Each change is recorded before it starts and marked complete only after Rivet verifies it. A crash at any point between those steps, for example before a write, after a file is replaced, or after a restart, leaves unresolved work in the journal.
The next apply recovers it automatically:
- Temporary files that Rivet recorded, such as file and home-directory candidates, are removed.
- Affected resources are re-inspected and converged again. The report notes that the earlier outcome is uncertain.
- Exec commands re-check their guards. A command with
always = trueruns again, so an interrupted command can repeat its side effects. Use guards when repetition matters. - Pending refreshes are delivered. If the crash happened before a refresh was acknowledged, it is delivered again. Delivery is at least once, not exactly once.
No manual acknowledgement is needed. Plan and noop never modify the journal.
Work the catalog no longer wants
If you change the configuration before the next run, some recorded work may no longer apply: the resource was removed, its desired state changed, or a notify was dropped. That work is marked superseded and reported, and doesn’t block the run. Superseding doesn’t mean the original operation succeeded. Rivet doesn’t undo its side effects or clean up its temporary files, and superseded refreshes aren’t delivered.
Interrupted SSH runs
If an SSH connection drops after an apply starts, the outcome is unknown. The remote run may have finished, stopped partway, or still be running; closing the connection doesn’t guarantee every remote process stopped. Inspect the same host, as the same user, with the same state directory:
For a direct root login, drop --sudo. --state-dir is a path on the remote host. Without it, root uses /var/lib/rivet, and an unprivileged user has a separate default journal. Add --installed-only to avoid uploading a runner just to inspect.
Rivet doesn’t retry a disconnected apply. A fresh apply recovers as described above, including rerunning always = true commands, so check for side effects first when repetition matters.
Interrupted plugin work
Rivet records intent before each plugin change. A crash, timeout, or lost reply can leave the outcome uncertain; the plugin session isn’t restarted or replayed. The next run needs the same plugin implementation that recorded the work, and asks it to recover before converging. If that plugin isn’t installed or doesn’t support recovery, the run refuses to continue rather than guessing. See plugin recovery.
Resource-specific recovery
- Modules: a module’s notifications are delivered only once every resource inside it, and their refreshes, succeed. A partial failure keeps the notifications pending.
- Timers: recovery restores the timer and service definitions and the timer’s activation, without running the job.
- Authorized keys and fstab: these shared files are re-read and merged under the host lock, preserving unrelated lines.
- File backups: backups made with
backup = truelive in/var/lib/rivet/backups, even when--state-dirpoints elsewhere. They are kept until you remove them, may contain secrets, and are never restored automatically.
Changing or removing for_each keys supersedes the old instances’ pending work on the next real apply. It doesn’t delete their host objects.
State and locks
The journal is a SQLite database under /var/lib/rivet by default, written with full synchronous durability. apply and journal accept --state-dir for another private absolute directory, but every real apply uses the same host lock, /run/rivet/apply.lock. Two state directories can’t be used to run applies in parallel.
The state directory must be owned by the runner with mode 0700. The journal and lock files must be regular files with mode 0600 and a single link. Symlinks, unsafe parent directories, and managed paths inside the state or lock areas are refused.
Each Rivet version reads exactly one journal version. A journal of any other version is refused and left untouched; it is never migrated.
Retained history
After each real run, Rivet keeps the 10 most recent full run reports and the 100 most recent resolved records of each kind (intents and refreshes), plus any records belonging to those 10 runs. Unresolved work is always kept, however old, while it still applies to the current catalog. Pruning never resolves work.
A single report can include up to 16 MiB of command output. Export reports if you need longer audit history. SQLite reuses freed space, but Rivet doesn’t VACUUM, so the file stays at its largest size. Plan, noop, and journal never prune.