> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://rivet.chrisboulton.com/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://rivet.chrisboulton.com/_mcp/server.

# YAML inputs and data hierarchies

> Layer YAML or JSON values by environment and host, with merges, removals, explanations, and SOPS.

A data hierarchy lets you keep the values that differ between environments or hosts in YAML or JSON files, layered from general to specific, while HCL declares the resources and typed variables. Rivet merges the layers when you compile, choosing files with facts such as the host's environment or hostname. The bundle contains the result, so hosts never need the data files.

```sh
rivet validate examples/hierarchy \
  --facts examples/hierarchy/facts.json \
  --data-hierarchy examples/hierarchy/data/hierarchy.yaml \
  --explain-inputs

rivet compile examples/hierarchy \
  --facts examples/hierarchy/facts.json \
  --data-hierarchy examples/hierarchy/data/hierarchy.yaml \
  --out host.bundle
```

`examples/hierarchy/facts.json` is sample data; use facts collected from a real host when compiling for one. Both commands also accept `--vars-file FILE` for final overrides. Hierarchies are only used when you pass `--data-hierarchy`; nothing is discovered automatically, and nothing is looked up at apply time.

## Input formats

Files ending in `.json` are parsed as strict JSON. Everything else, including `.yaml`, `.yml`, and files without an extension, is parsed as YAML. A hierarchy can mix both.

Inputs contain exactly one document with string mapping keys. Nested duplicate keys, non-string keys, custom tags, and non-finite numbers are rejected. YAML anchors and aliases are accepted within expansion limits; YAML `<<` keys have no implicit merge behavior. Use the hierarchy's `merge` policy to combine layers. Scalars retain their YAML types, so quote numeric-looking strings such as modes (`"0644"`); `true` and `false` are booleans, and plain `yes`, `no`, `on`, and `off` are strings. Comments and block strings are supported:

```yaml
message: |
  managed by Rivet
port: 8443
certificate: null
```

YAML and JSON inputs that express the same values produce identical catalogs.

## Typed variables

Supported input types are `string`, `bool`, integer-only `number`, `list(T)`, `set(T)`, `tuple([...])`, `map(T)`, and `object({...})`. Within object types, `optional(T)` permits omission or null; omitted optional fields become null after input resolution. Defaults can be recursive literal values. Required object fields and collection elements cannot be null. The same declared-type rules apply to evaluated module arguments and outputs; nested nulls require an optional object member. A variable's `nullable` policy governs its root value and defaults to true. Resource handle types can be passed between HCL modules but cannot be fabricated by data files. Sets normalize using the same deterministic deduplication as module inputs; tuples retain their declared length and order. No type coercion occurs.

```text
variable "web_config" {
  type = object({
    listen = object({ address = string, port = number })
    networks = list(string)
    headers = map(string)
    certificate = optional(string)
  })
  nullable = false
}
```

Read fields with `var.web_config.listen.port`, literal keys with `var.web_config["listen"]`, and list elements with integer indices; typed collections also support evaluated indices. Collections can feed resource parameters, locals, outputs, and module arguments. Sets, tuples, expansion, and modules are implemented; see [expansion](/language/expansion). Integer interpolation such as `"port=${var.web_config.listen.port}"` renders the number as text; booleans also interpolate as text; null and collections need explicit encoding or conversion.

## Hierarchy manifest

Paths are relative to the manifest directory, independently of the controller working directory. Layers are listed from lowest to highest priority. Variables default to whole-value replacement; only those listed as `deep` merge objects recursively.

```yaml
merge:
  web_config: deep
layers:
  - path: common.yaml
  - path: environments/${facts.deployment.environment}.yaml
  - path: nodes/${facts.networking.hostname}.yaml
    optional: true
```

The repository tutorial's `common.yaml` supplies the complete base value:

```yaml
web_config:
  listen:
    address: 127.0.0.1
    port: 8080
  networks:
    - 127.0.0.0/8
  headers:
    X-Legacy: enabled
    X-Application: rivet-example
  certificate: null
```

`environments/production.yaml` supplies only its overrides:

```yaml
web_config:
  listen:
    port: 8443
  networks: []
```

With deployment environment `production`, deep merge retains the listen address and headers, changes the port to 8443, and clears networks. Adding the removal shown below deletes X-Legacy while retaining X-Application. The optional node layer may be missing. `examples/hierarchy/facts.json` supplies the public selector fixture.

Each layer file is one mapping keyed by declared root variable names. Manifest and layer schemas reject unknown fields or variable names; duplicate mapping keys are rejected recursively. `merge` accepts only `replace` and `deep`. Deep merge is available only for declared map or object variables. A layer may opt into SOPS decryption with `decrypt: sops`; other decryption modes are rejected.

Path substitution accepts literal dotted fact traversals from the supplied snapshot, including the existing platform aliases. Selectors must be available strings consisting of ASCII letters, digits, underscores, dots, and hyphens; empty values, `.` and `..` are refused. Paths must be relative, without parent/current-directory components or symlinks. A missing optional file or directory skips that layer, including its removals. Missing selector facts, malformed or unreadable files, symlinks, and nonregular files always fail.

## Merge contract

1. Resolve hierarchy files in listed order. Missing keys inherit. With deep merge, two objects recursively merge by key; arrays and scalar values replace. Null is an explicit replacement and follows the declared type. An empty array clears an array; an empty object contributes no keys to an existing object.
2. Validate every supplied layer before merging. Deep-merged objects may omit required fields, but their supplied fields must have valid names and types. Arrays replace atomically, so object elements inside an array must be complete within that layer. Invalid values fail even when a later layer would replace them.
3. Apply `--vars-file` last. Each supplied variable replaces its complete hierarchy value, regardless of merge policy. Partial object overrides in this file fail.
4. Use an HCL variable default only if no source supplied that variable. Defaults do not fill partial hierarchy objects. Put mergeable defaults in `common.yaml`.
5. Validate final completeness and only then normalize missing optional fields to null. This ensures an omitted optional field in a partial layer cannot overwrite an inherited value.

Explicit null remains supplied even if the variable has a nonnull default. Unknown object fields fail; maps permit arbitrary member names with their declared value type. Mapping key order never changes merge results. Lists keep their order. No array concatenation, positional matching, string deletion markers, or implicit resource patching occurs.

## Removing inherited keys

Put removal paths on a manifest layer, separate from its ordinary data:

```yaml
path: environments/production.yaml
remove:
  - [web_config, headers, X-Legacy]
```

Paths are arrays of literal member names, so names containing dots or slashes are unambiguous. Removal runs in listed order against the accumulated hierarchy value, before that layer's values merge. It must name an existing nested object or map member in a deep-merged variable. Removing a whole variable, an array element, or an unknown path fails. Duplicate or overlapping removal paths fail if an earlier removal makes a later path nonexistent.

The layer can rebuild a removed subtree by supplying a new object at that path. Subsequent layers can also reintroduce a removed key. A required field removed without replacement fails final validation; a removed optional field normalizes to null. HCL defaults are not brought back by removing a field. Removing a declaration from an expanded configuration would stop management rather than delete host objects; declare `ensure = "absent"` to remove something.

## Explanations, privacy, and limits

`validate --explain-inputs` prints paths as JSON arrays with selected, shadowed, and removed sources. Arrays are explained as units. Sensitive variable explanations collapse to the declared variable name so data-derived map keys are not exposed. Explanations never contain values. Errors identify the input file or layer template and, for typed inputs, a JSON array field path and the expected type or missing or unknown field. Sensitive field paths stop at the declared variable name. Syntax errors include line and column when available, but never parser excerpts or supplied values. Generated missing optional nulls follow the type contract rather than a separate data-source assignment.

Missing optional layers are quiet by default. To include their original path templates and the missing-file reason, enable both flags:

```sh
rivet validate examples/hierarchy \
  --facts examples/hierarchy/facts.json \
  --data-hierarchy examples/hierarchy/data/hierarchy.yaml \
  --explain-inputs --explain-skipped-layers
```

The opt-in reports templates such as `nodes/${facts.networking.hostname}.yaml`, without printing substituted fact values. It does not change which layers load or the catalog digest.

Sensitivity is declared on the HCL variable or supplied automatically by a SOPS layer, and propagates through traversals, locals, outputs, resource metadata, and bundled assets. `sensitive` is redaction, not encryption. Bundles can contain plaintext secrets and are published with mode 0600 without overwriting existing destinations. Raw input maps, acquisition paths, and explanations are not new catalog fields. Only resulting desired state, outputs, and asset content affect the semantic digest; unused inputs and data-source locations do not independently change it.

The manifest, selected layers (including ciphertext and decrypted output), and `--vars-file` have a combined 16 MiB read limit, with at most 1024 layers and a recursive input and type depth ceiling of 128; parsers may reject nesting earlier. Required variables, nullability, types, removals, and all catalog validation complete before resource and cloud asset acquisition. The combined inputs also have an expanded-data budget of one million nodes (including mapping keys) and 16 MiB of string and key bytes, so aliases cannot bypass the read limit. Operational input I/O and SOPS process failures return 1; invalid YAML/JSON/configuration returns 2. Failed compilation publishes no bundle.

## SOPS layers

Mark an encrypted layer with `decrypt: sops`:

```yaml
merge:
  web_config: deep
layers:
  - path: common.yaml
  - path: secrets/production.sops.yaml
    decrypt: sops
```

Install `sops` on the Unix controller and configure its credentials using the normal SOPS mechanisms. Rivet leaves key management to SOPS, including age, KMS, and PGP. The decrypted document uses the same root-variable mapping and merge rules as plaintext layers. Both validate and compile decrypt; plan and apply use the completed bundle offline.

Rivet reads bounded ciphertext from the confined layer path, checks its YAML/JSON structure for duplicate keys, and feeds those exact bytes to `sops decrypt --input-type yaml --output-type json` via stdin. A `.json` layer selects `--input-type json`; other names select YAML. Stdout is captured directly in memory and passes through the same duplicate-key, expansion, and type checks as ordinary inputs. No shell or plaintext temporary file is used. Credentials and encryption keys remain owned by the installed SOPS executable.

Each decrypt has a 30-second deadline and shares the combined input size budget. Process failures, unavailable executable/credentials, integrity failures, timeouts, and output limits stop compilation; raw stderr and stdout never appear in diagnostics. Rivet terminates the owned process group and reaps the child. `optional: true` skips only a nonexistent encrypted file or directory, never a decryption failure. SOPS decryption is supported on Unix controllers.

Every declared variable supplied by a decrypted layer becomes sensitive automatically, including unencrypted fields in that layer. The mark remains even if a later plaintext layer or `--vars-file` completely replaces the variable. It propagates through traversals, locals, outputs, resource metadata, and bundled assets; explanations and typed errors redact its nested keys. This way a secret can't become public by accident when another layer overrides it.

SOPS protects your source files. Compiled bundles still contain the decrypted values in plain text, so keep them private.