Troubleshooting
Rivet’s error messages never include configuration values, secrets, URLs, or private paths. When it can say more without them, the underlying cause follows a colon: an OS error such as Permission denied (os error 13), a JSON position such as JSON syntax error at line 3 column 7, an SQLite code, an HTTP status, or a storage error category.
Configuration and compilation
| Symptom | What to check |
|---|---|
| Unsupported block, expression, or function | Compare with the syntax and function references. Plugin functions are called as plugin::ALIAS::NAME(...). |
| Missing fact or invalid index | Look at the field in a collected snapshot. Optional fields are left out when unavailable, and baseline target files only have the four basic facts (distribution, version, architecture, init_system). |
| Type mismatch | Rivet doesn’t convert types implicitly. Use real booleans and integers, and quote modes such as "0644". |
| Resource attribute not available | Only declared static exports can be referenced. IDs the host allocates can’t be read at compile time; use names instead. |
| Ownership conflict | Two declarations manage the same file, service, package, or account, or their group memberships overlap. |
| Dependency cycle | Look at references, depends_on, before, notifications, and the implicit rules for files, accounts, and services. rivet graph explains edges in a bundle that compiles. |
| Module lock or cache missing | Run rivet init. validate and compile never fetch modules. |
| Missing or unknown variable, or duplicate key | Check the --vars-file or hierarchy against the declared types, including nested duplicate keys, required fields, and nullable. |
| Hierarchy merge or removal fails | deep merge needs a map or object variable. Removal paths are lists of literal keys. --vars-file replaces whole variables. validate --explain-inputs shows where each value came from. |
| Optional hierarchy layer still fails | optional only skips missing files. Missing selector facts, bad selector values, symlinks, malformed files, and SOPS errors still fail. |
| SOPS fails | Check that sops is installed with working credentials, the file decrypts, and decryption finishes within 30 seconds. Rivet hides SOPS’s own output. |
| Function error | Check argument count and types, number limits, format and regex syntax, and convert type constraints. try and can catch evaluation errors. |
| Template or file read refused | Files must be regular UTF-8 files inside the module’s directory. Pass variables to templatefile explicitly, and escape ${ as $${ for a second rendering pass. |
| Sensitive value used as an expansion key | for_each keys must not be sensitive. Marking a whole variable sensitive also marks its keys. |
| Can’t read a module’s resource | Use the module’s outputs. To depend on a module, use module.NAME for all of it or an exported handle for one resource. A module can’t notify a resource inside itself. |
| Output path already exists | Compile to a new path; Rivet never overwrites a bundle. |
| Cloud download fails | Check compiler credentials, region, the exact object key, and the checksum. AWS shared profiles and SSO, and federated GCS credentials, aren’t supported. |
Plugins
| Symptom | What to check |
|---|---|
| Plugin path or checksum refused | Check the selected --plugin-lock, parent directory permissions, symlinks, and the binary’s checksum. Root runs need root-owned files. Locks under /tmp are refused. See plugin installation. |
| Plugin requirement not met | The plugin block’s alias, source, and version must match an approved installation. Requirements never download or approve anything. |
| Host plugins don’t match | Install the plugin on the host and approve it there. Uploading a runner doesn’t install plugins. |
| Plugin call failed during apply | Check the journal and what actually happened on the host. Keep the same plugin release installed so the next run can recover. |
Releases, agents, and the compile service
| Symptom | What to check |
|---|---|
| Agent configuration missing or invalid | Configure exactly one of source or server. An invalid /etc/rivet/agent.hcl is always an error. Agent files accept only literal values. |
| Environment missing or promotion refused | Check the name, that the release is complete, the expected generation, and whether someone else promoted at the same time. A deleted environment fails even if an old pointer is cached. |
| Release fails on modules, file reads, or variables | Releases support only top-level *.hcl files and resource assets, and variables need defaults. Use direct rivet compile for anything else. |
| Authentication denied | Check the audience or issuer, the service account’s project and email or the certificate’s URI SAN, certificate and CRL expiry, and that an allow rule names this exact principal and environment. |
trusted unavailable or claim missing | Trusted claims exist only when the compile service compiles. Local and source-mode compilation can’t read them. Optional claims are left out when absent. |
| Config file or cache refused | Files must be regular files, not symlinks, with the required ownership; keys and tokens must be private. The cache must be an absolute private directory owned by the caller, with no managed paths inside it. |
| Signed download or refresh failed | Check signing permissions, that assets belong to the release, URL lifetime, and whether the service restarted. The agent refreshes URLs at most once and never switches releases. |
| No offline catalog | Run online successfully first, with the same mode, source or server, environment, and host. Check the maximum age. Offline runs never download missing files. |
| HTTP 429 | The service is at its limit of concurrent compiles. Spread out scheduled runs. |
Acquisition failures stop the run before any change. See releases, agent runs, and compile service.
SSH
| Symptom | What to check |
|---|---|
| Host key or authentication fails | Enroll the host key and make sure plain ssh -o BatchMode=yes HOST works. Rivet can’t answer prompts. |
| No compatible runner | A Mac build can’t run on Linux. Pass a Linux build with --runner-binary, or install a runner on the host. --installed-only disables uploads. |
| Runner binary rejected | It must match the host’s CPU architecture and run on Ubuntu 24.04. |
| Sudo fails, or apply needs root | Log in as root or add --sudo. The login user needs passwordless sudo for /bin/sh; a rule allowing only Rivet isn’t enough. |
| Unsafe path or corrupt cache | Check ownership, writable parent directories, symlinks, and the cached files. Root and user caches are separate. |
| Capability or target mismatch | Collect facts again, check that the host has the required tools, and recompile. A newer runner doesn’t fix missing host tools. |
| Connection dropped during apply | Run rivet journal --target ... before applying again. See interrupted SSH runs. |
On the host
| Symptom | What to check |
|---|---|
| Command rejected on macOS | --local needs Linux. Use --target from a Mac, or rivet run --prepare-only to build an agent bundle. |
| Platform or capability mismatch | Collect fresh facts on the host, check that the needed tools are installed, and recompile. |
| Permission denied during plan | The user running plan must be able to read every managed object. An unreadable object is an error, not “absent”. |
| Unsafe filesystem object | Check for symlinked parents, hard links, special files, writable parent directories, ACLs, extended attributes, and security labels. Changing an object’s type needs replace_type = true. |
| Missing parent or non-empty directory | Declare the parent directory or set create_parents = true. Rivet never removes non-empty directories. |
| File validator fails | Check that the validator exists on the host and takes {candidate} correctly. Plan doesn’t run validators. |
| Package manager locked, or package half-installed | Wait for or stop the other package manager, or install missing dependencies, then apply again. |
| Held package refused | Set held = false before changing or removing it. |
| Service won’t start or enable | Set masked = false, check the unit exists, and note that static, generated, and transient units can’t be enabled. |
| User or group change refused | UIDs can’t change. Check for users whose primary group is being removed, name or ID collisions, home directory safety, logged-in users, and overlapping membership resources. |
| Timer refused | Check the executable path is absolute, the schedule, that persistent has a calendar trigger, unmanaged drop-ins or masks, and whether the job is running during removal. |
| Authorized key refused | Check the key, the user’s home directory, and that every key in the same file uses the same permission settings. |
| Mount drift or busy unmount | Check the source, type, options, and on_change, duplicate fstab entries, and what is using the filesystem. Rivet never force-unmounts. |
| Backup failed | A failed backup blocks the replacement. Check /var/lib/rivet/backups. |
| Shared file conflict | Resources sharing a file, such as authorized_keys, must agree on ownership and permissions. |
| Resource blocked or deferred | Look at its prerequisite’s result; the resource itself may be fine. |
| Journal version refused | Use a matching Rivet build to inspect it, or start with a new state directory. See recovery. |
Good to know
Catalogs run with the runner’s authority. Commands, guards, and file validators are not sandboxed, so review configuration and assets as you would a root shell script. Guards run even during plan and noop, so keep them read-only.
Timeouts stop Rivet’s own processes, but a process that escapes its group or is stuck in the kernel can outlive them. To explore resources that create packages, accounts, services, timers, SSH keys, or mounts, use a disposable Linux host.