> For clean Markdown of any page, append .md to the page URL. > For a complete documentation index, see https://rivet.chrisboulton.com/llms.txt. > For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://rivet.chrisboulton.com/_mcp/server. # Troubleshooting > Common errors and refusals, and what to check. Rivet's error messages never include configuration values, secrets, URLs, or private paths. When it can say more without them, the underlying cause follows a colon: an OS error such as `Permission denied (os error 13)`, a JSON position such as `JSON syntax error at line 3 column 7`, an SQLite code, an HTTP status, or a storage error category. ## Configuration and compilation | Symptom | What to check | | --------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Unsupported block, expression, or function | Compare with the [syntax](/language/syntax) and [function](/language/functions) references. Plugin functions are called as `plugin::ALIAS::NAME(...)`. | | Missing fact or invalid index | Look at the field in a collected snapshot. Optional fields are left out when unavailable, and baseline target files only have the four basic facts (`distribution`, `version`, `architecture`, `init_system`). | | Type mismatch | Rivet doesn't convert types implicitly. Use real booleans and integers, and quote modes such as `"0644"`. | | Resource attribute not available | Only declared [static exports](/resources/conventions#static-exports) can be referenced. IDs the host allocates can't be read at compile time; use names instead. | | Ownership conflict | Two declarations manage the same file, service, package, or account, or their group memberships overlap. | | Dependency cycle | Look at references, `depends_on`, `before`, notifications, and the implicit rules for files, accounts, and services. `rivet graph` explains edges in a bundle that compiles. | | Module lock or cache missing | Run `rivet init`. `validate` and `compile` never fetch modules. | | Missing or unknown variable, or duplicate key | Check the `--vars-file` or hierarchy against the declared types, including nested duplicate keys, required fields, and `nullable`. | | Hierarchy merge or removal fails | `deep` merge needs a map or object variable. Removal paths are lists of literal keys. `--vars-file` replaces whole variables. `validate --explain-inputs` shows where each value came from. | | Optional hierarchy layer still fails | `optional` only skips missing files. Missing selector facts, bad selector values, symlinks, malformed files, and SOPS errors still fail. | | SOPS fails | Check that `sops` is installed with working credentials, the file decrypts, and decryption finishes within 30 seconds. Rivet hides SOPS's own output. | | Function error | Check argument count and types, number limits, format and regex syntax, and `convert` type constraints. `try` and `can` catch evaluation errors. | | Template or file read refused | Files must be regular UTF-8 files inside the module's directory. Pass variables to `templatefile` explicitly, and escape `${` as `$${` for a second rendering pass. | | Sensitive value used as an expansion key | `for_each` keys must not be sensitive. Marking a whole variable sensitive also marks its keys. | | Can't read a module's resource | Use the module's outputs. To depend on a module, use `module.NAME` for all of it or an exported handle for one resource. A module can't notify a resource inside itself. | | Output path already exists | Compile to a new path; Rivet never overwrites a bundle. | | Cloud download fails | Check compiler credentials, region, the exact object key, and the checksum. AWS shared profiles and SSO, and federated GCS credentials, aren't supported. | ## Plugins | Symptom | What to check | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Plugin path or checksum refused | Check the selected `--plugin-lock`, parent directory permissions, symlinks, and the binary's checksum. Root runs need root-owned files. Locks under `/tmp` are refused. See [plugin installation](/contributing/plugins#installation-and-requirements). | | Plugin requirement not met | The `plugin` block's alias, source, and version must match an approved installation. Requirements never download or approve anything. | | Host plugins don't match | Install the plugin on the host and approve it there. Uploading a runner doesn't install plugins. | | Plugin call failed during apply | Check the journal and what actually happened on the host. Keep the same plugin release installed so the next run can [recover](/reference/recovery#interrupted-plugin-work). | ## Releases, agents, and the compile service | Symptom | What to check | | -------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Agent configuration missing or invalid | Configure exactly one of `source` or `server`. An invalid `/etc/rivet/agent.hcl` is always an error. Agent files accept only literal values. | | Environment missing or promotion refused | Check the name, that the release is complete, the expected generation, and whether someone else promoted at the same time. A deleted environment fails even if an old pointer is cached. | | Release fails on modules, file reads, or variables | Releases support only top-level `*.hcl` files and resource assets, and variables need defaults. Use direct `rivet compile` for anything else. | | Authentication denied | Check the audience or issuer, the service account's project and email or the certificate's URI SAN, certificate and CRL expiry, and that an `allow` rule names this exact principal and environment. | | `trusted` unavailable or claim missing | Trusted claims exist only when the compile service compiles. Local and source-mode compilation can't read them. Optional claims are left out when absent. | | Config file or cache refused | Files must be regular files, not symlinks, with the required ownership; keys and tokens must be private. The cache must be an absolute private directory owned by the caller, with no managed paths inside it. | | Signed download or refresh failed | Check signing permissions, that assets belong to the release, URL lifetime, and whether the service restarted. The agent refreshes URLs at most once and never switches releases. | | No offline catalog | Run online successfully first, with the same mode, source or server, environment, and host. Check the maximum age. Offline runs never download missing files. | | HTTP 429 | The service is at its limit of concurrent compiles. Spread out scheduled runs. | Acquisition failures stop the run before any change. See [releases](/operations/distribution), [agent runs](/operations/agent), and [compile service](/operations/compile-service). ## SSH | Symptom | What to check | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ | | Host key or authentication fails | Enroll the host key and make sure plain `ssh -o BatchMode=yes HOST` works. Rivet can't answer prompts. | | No compatible runner | A Mac build can't run on Linux. Pass a Linux build with `--runner-binary`, or install a runner on the host. `--installed-only` disables uploads. | | Runner binary rejected | It must match the host's CPU architecture and run on Ubuntu 24.04. | | Sudo fails, or apply needs root | Log in as root or add `--sudo`. The login user needs passwordless sudo for `/bin/sh`; a rule allowing only Rivet isn't enough. | | Unsafe path or corrupt cache | Check ownership, writable parent directories, symlinks, and the cached files. Root and user caches are separate. | | Capability or target mismatch | Collect facts again, check that the host has the required tools, and recompile. A newer runner doesn't fix missing host tools. | | Connection dropped during apply | Run `rivet journal --target ...` before applying again. See [interrupted SSH runs](/reference/recovery#interrupted-ssh-runs). | See [SSH from a controller](/operations/ssh). ## On the host | Symptom | What to check | | ------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Command rejected on macOS | `--local` needs Linux. Use `--target` from a Mac, or `rivet run --prepare-only` to build an agent bundle. | | Platform or capability mismatch | Collect fresh facts on the host, check that the needed tools are installed, and recompile. | | Permission denied during plan | The user running plan must be able to read every managed object. An unreadable object is an error, not "absent". | | Unsafe filesystem object | Check for symlinked parents, hard links, special files, writable parent directories, ACLs, extended attributes, and security labels. Changing an object's type needs `replace_type = true`. | | Missing parent or non-empty directory | Declare the parent directory or set `create_parents = true`. Rivet never removes non-empty directories. | | File validator fails | Check that the validator exists on the host and takes `{candidate}` correctly. Plan doesn't run validators. | | Package manager locked, or package half-installed | Wait for or stop the other package manager, or install missing dependencies, then apply again. | | Held package refused | Set `held = false` before changing or removing it. | | Service won't start or enable | Set `masked = false`, check the unit exists, and note that static, generated, and transient units can't be enabled. | | User or group change refused | UIDs can't change. Check for users whose primary group is being removed, name or ID collisions, home directory safety, logged-in users, and overlapping membership resources. | | Timer refused | Check the executable path is absolute, the schedule, that `persistent` has a calendar trigger, unmanaged drop-ins or masks, and whether the job is running during removal. | | Authorized key refused | Check the key, the user's home directory, and that every key in the same file uses the same permission settings. | | Mount drift or busy unmount | Check the source, type, options, and `on_change`, duplicate fstab entries, and what is using the filesystem. Rivet never force-unmounts. | | Backup failed | A failed backup blocks the replacement. Check `/var/lib/rivet/backups`. | | Shared file conflict | Resources sharing a file, such as `authorized_keys`, must agree on ownership and permissions. | | Resource blocked or deferred | Look at its prerequisite's result; the resource itself may be fine. | | Journal version refused | Use a matching Rivet build to inspect it, or start with a new state directory. See [recovery](/reference/recovery). | ## Good to know Catalogs run with the runner's authority. Commands, guards, and file validators are not sandboxed, so review configuration and assets as you would a root shell script. Guards run even during plan and noop, so keep them read-only. Timeouts stop Rivet's own processes, but a process that escapes its group or is stuck in the kernel can outlive them. To explore resources that create packages, accounts, services, timers, SSH keys, or mounts, use a [disposable Linux host](/contributing/testing).