> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://rivet.chrisboulton.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://rivet.chrisboulton.com/_mcp/server. # Recovery and the journal > How Rivet records in-flight work, recovers from interruptions, and keeps history. The journal records runs, in-flight changes, uncertain outcomes, and pending refreshes. It exists to finish interrupted work. It is not an inventory of what Rivet manages: every run re-inspects the host instead of trusting the journal. ## Inspect it ```sh sudo rivet journal --local sudo rivet journal --local --state-dir /var/lib/rivet-site ``` This prints unresolved intents and refreshes as JSON. It opens the journal read-only and doesn't create one if none exists. After a clean run, the output is `[]`. Don't delete the journal to force a clean run. Check what actually happened on the host, then fix the configuration or the host if needed. ## What happens after a crash Each change is recorded before it starts and marked complete only after Rivet verifies it. A crash at any point between those steps, for example before a write, after a file is replaced, or after a restart, leaves unresolved work in the journal. The next apply recovers it automatically: * Temporary files that Rivet recorded, such as file and home-directory candidates, are removed. * Affected resources are re-inspected and converged again. The report notes that the earlier outcome is uncertain. * Exec commands re-check their guards. A command with `always = true` runs again, so an interrupted command can repeat its side effects. Use guards when repetition matters. * Pending refreshes are delivered. If the crash happened before a refresh was acknowledged, it is delivered again. Delivery is at least once, not exactly once. No manual acknowledgement is needed. Plan and noop never modify the journal. ### Work the catalog no longer wants If you change the configuration before the next run, some recorded work may no longer apply: the resource was removed, its desired state changed, or a `notify` was dropped. That work is marked `superseded` and reported, and doesn't block the run. Superseding doesn't mean the original operation succeeded. Rivet doesn't undo its side effects or clean up its temporary files, and superseded refreshes aren't delivered. ## Interrupted SSH runs If an SSH connection drops after an apply starts, the outcome is unknown. The remote run may have finished, stopped partway, or still be running; closing the connection doesn't guarantee every remote process stopped. Inspect the same host, as the same user, with the same state directory: ```sh rivet journal --target app-1 --sudo --state-dir /var/lib/rivet-site ``` For a direct root login, drop `--sudo`. `--state-dir` is a path on the remote host. Without it, root uses `/var/lib/rivet`, and an unprivileged user has a separate default journal. Add `--installed-only` to avoid uploading a runner just to inspect. Rivet doesn't retry a disconnected apply. A fresh apply recovers as described above, including rerunning `always = true` commands, so check for side effects first when repetition matters. ## Interrupted plugin work Rivet records intent before each plugin change. A crash, timeout, or lost reply can leave the outcome uncertain; the plugin session isn't restarted or replayed. The next run needs the same plugin implementation that recorded the work, and asks it to recover before converging. If that plugin isn't installed or doesn't support recovery, the run refuses to continue rather than guessing. See [plugin recovery](/contributing/plugins#resource-lifecycle-and-recovery). ## Resource-specific recovery * **Modules:** a module's notifications are delivered only once every resource inside it, and their refreshes, succeed. A partial failure keeps the notifications pending. * **Timers:** recovery restores the timer and service definitions and the timer's activation, without running the job. * **Authorized keys and fstab:** these shared files are re-read and merged under the host lock, preserving unrelated lines. * **File backups:** backups made with `backup = true` live in `/var/lib/rivet/backups`, even when `--state-dir` points elsewhere. They are kept until you remove them, may contain secrets, and are never restored automatically. Changing or removing `for_each` keys supersedes the old instances' pending work on the next real apply. It doesn't delete their host objects. ## State and locks The journal is a SQLite database under `/var/lib/rivet` by default, written with full synchronous durability. `apply` and `journal` accept `--state-dir` for another private absolute directory, but every real apply uses the same host lock, `/run/rivet/apply.lock`. Two state directories can't be used to run applies in parallel. The state directory must be owned by the runner with mode 0700. The journal and lock files must be regular files with mode 0600 and a single link. Symlinks, unsafe parent directories, and managed paths inside the state or lock areas are refused. Each Rivet version reads exactly one journal version. A journal of any other version is refused and left untouched; it is never migrated. ## Retained history After each real run, Rivet keeps the 10 most recent full run reports and the 100 most recent resolved records of each kind (intents and refreshes), plus any records belonging to those 10 runs. Unresolved work is always kept, however old, while it still applies to the current catalog. Pruning never resolves work. A single report can include up to 16 MiB of command output. Export reports if you need longer audit history. SQLite reuses freed space, but Rivet doesn't `VACUUM`, so the file stays at its largest size. Plan, noop, and `journal` never prune.