Formats & integrations

YAML, JSON, env, binary; tooling.

A locksmith who can read the label on your box behaves differently from one who cannot. Hand SOPS (Secrets OPerationS, the secrets-encryption tool that now lives under the CNCF, the Cloud Native Computing Foundation, rather than Mozilla where it started) a file whose shape it understands, and it behaves like a wall of hotel deposit boxes: every value goes into its own numbered drawer, and the labels on the front stay readable. Hand it something it cannot parse and it reaches for shrink wrap instead. The whole file becomes one sealed brick with nothing showing through.

Both are real encryption. Same cipher (AES256-GCM, which hides the data and also detects tampering), same data key, same MAC (Message Authentication Code, a fingerprint that proves nobody edited the ciphertext behind your back). What changes is everything wrapped around the encryption: what a Git diff shows, what a reviewer can meaningfully approve, and whether your key-selection rules do anything at all. And the thing that decides which behaviour you get has nothing to do with the contents of the file. It is the last few characters of its name.

How SOPS Picks a Parser

Inside the source there is a function called FormatForPath, and it is about as sophisticated as a bouncer checking IDs by glancing at the last letter. It reads the end of the path string and nothing else. A path ending in .yaml or .yml gets the YAML parser (YAML is the indentation-based config format Kubernetes and most build systems speak). A .json ending gets JSON (JavaScript Object Notation). A .env ending gets dotenv, the flat KEY=value format that shells and Docker read. A .ini ending gets INI, the old sectioned format with [headings]. Everything else falls through to binary, with no warning and no error.

Those are strict suffix checks, not fuzzy pattern matches. The dotenv rule fires for a file called .env or app.env. It does not fire for .env.production, .env.local, or env.prod, all of which sail straight past into binary. Same story for values.secret, secrets.yaml.tpl, config.toml, id_rsa and tls.key. Keep that in your head. Most of this lesson is consequences of it.

$ export SOPS_AGE_KEY_FILE=~/.config/sops/age/keys.txt
$ cat app.env
DB_PASSWORD=hunter2
STRIPE_KEY=sk_live_51H7qVdKp2mQx8LrTn4Ba
LOG_LEVEL=info
 
$ sops encrypt app.env

DB_PASSWORD=ENC[AES256_GCM,data:RnYd71ipUQ==,iv:2QsgeO9WB4PVwciMHdyTAryqKYBvTri8JHWpWSeJN/0=,tag:tVOoT/zCcfshesz7JEeCDQ==,type:str]
STRIPE_KEY=ENC[AES256_GCM,data:7TkI3H3UyWbASgNf48qRk6YPVYxHEFzFdKYgLDA=,iv:mr/N4BWJ8C136t+8gyVW6u/Vvz48Rubx1ZHXlz62wiM=,tag:Ujgw2J3jj0J9KhmDSvgCnw==,type:str]
LOG_LEVEL=ENC[AES256_GCM,data:O4CzDg==,iv:AdzZvudPmRE0Cuu3usLl1KSmCmnHaZAoZtYaXViCg3o=,tag:nW+ilxDvZKpRjP1lJuYfcQ==,type:str]
sops_age__list_0__map_enc=-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWL0tCdEpxZmNZQitqaGFF\nVUZQL3RQNmdWdmFwYVFYS1cyV2FDNVdtalNNCnpVbGMzcGxpQXlwd2REbHdqNllt\nMDRLTXBvc3YwZU96dm1ucFo3ekVXbUEKLS0tIE9xYzNIeU9FMEEyWTFOR0dSMFVL\nekUxd3pxMEtMMUpLYUN0NTNRTVdTcTAKH6qZjnZTcjQziwcqi6dThP6Ei2AVvNL2\niwYRH6NEeVJ11jGBqx/HuycM3uFQYLTXzC4rrr4mrjgGchtQrrFt7Q==\n-----END AGE ENCRYPTED FILE-----\n
sops_age__list_0__map_recipient=age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
sops_lastmodified=2026-07-22T09:14:31Z
sops_mac=ENC[AES256_GCM,data:lA3jAM22VhK5L+bLEFQRnkyclPE/gXgLfuRVZscDAHFFmKoxF8Ftloms3otBtJ/5A2G3Mqvl0rDRfxY++gHoI4d+wEaGOmSjLaZ1QbPdzBnMjFxngixJ9PuSG29NZc1pKmvu6lbVKX0SjXALxMQdFspIyk2vRQuf3luNM46awVM=,iv:C1/ApfG8BMgbNMlULterpibWZ80D5sU1W0G5s+Kv17I=,tag:kknhxbJTADPnPiWHzaobPw==,type:str]
sops_unencrypted_suffix=_unencrypted
sops_version=3.13.2

Read that output slowly, because it shows the whole design. The key names stayed in cleartext and only the values turned into ENC[...] strings. SOPS encrypts values, never keys, which is why git diff can tell you that STRIPE_KEY changed without telling you what it changed to. Now look at the tail. The metadata block got flattened into sops_-prefixed variables sorted alphabetically, with names like sops_age__list_0__map_enc, because a dotenv file has nowhere to hang a nested map. Every format carries identical metadata. Each one bends it into a shape the format can physically hold.

Two flags override the guess. The --input-type flag says how to parse what comes in, --output-type says how to serialise what goes out, and both accept yaml, json, dotenv, ini or binary. They are independent settings, which is exactly where people trip. Set only the input type and SOPS still derives the output type from the filename, which is how you end up decrypting a perfectly good YAML file and getting a complaint about binary stores.

The Four Structure-Aware Formats

YAML is the sensible default, because it holds nesting, lists and multi-line strings, and because Kubernetes manifests already look like this. One behaviour catches everyone out the first time: SOPS treats YAML comments as content and encrypts them too. A comment sitting next to a secret comes back as #ENC[AES256_GCM,...,type:comment]. That is good if your comment says something like "rotated after the SEC-4412 incident", and annoying if you expected reviewers to read it. Write operational notes in a separate unencrypted file, not inside the secret. (If you later add an encrypted_regex, comments follow the same selection rule and stop being encrypted along with everything else that does not match.)

JSON is the strictest of the four: no comments, no trailing commas, and the plaintext SOPS emits is re-indented with tab characters. Reach for it when the consumer is a program rather than a person. INI gives you exactly one level of nesting through its [section] headers, which is enough for tools that already speak INI and not enough for anything else. dotenv is flat and only flat, one level of KEY=value with no structure underneath. Multi-line values still survive, because the dotenv store escapes real newlines to a literal backslash-n on the way out and turns them back on the way in, so a PEM-encoded private key (PEM is the base64 text wrapper that starts with a BEGIN line) can live in a dotenv value as one very long line.

Because parsing and serialising are separate steps, the format on disk does not constrain the format you deliver. Store the file as YAML for reviewability and hand JSON to a service that only speaks JSON.

$ sops decrypt --output-type json secrets.enc.yaml

{
	"database": {
		"host": "db.acme.internal",
		"port": 5432,
		"password": "s3cr3t-p0stgr3s-pw"
	},
	"api": {
		"token": "tok_9f2aQx7Lm4Vd"
	}
}

Ask for a format that cannot hold your data and SOPS refuses rather than guessing. That refusal is a feature. A tool that silently flattened database.password into DATABASE_PASSWORD would be making a naming decision on your behalf, in a file where a wrong name means an app boots with an empty credential.

$ sops decrypt --output-type dotenv secrets.enc.yaml
$ echo $?

Error dumping file: cannot use complex value in dotenv file: [{host db.acme.internal} {port %!s(int=5432)} {password s3cr3t-p0stgr3s-pw}]
4

Look hard at that error before you move on. SOPS printed the offending branch to prove which one it choked on, and your database password went along for the ride, into your terminal scrollback and into whatever log your build system keeps of stderr. An error message is not a safe place for plaintext. If you are scripting this, send stderr somewhere you control instead of letting it land in a job log that half the company can read.

You get the same rule enforced from a completely different code path. Running sops exec-env on that nested file prints "cannot use complex value in environment:" followed by the same branch, and exits 1 rather than 4. Two implementations, one rule: flat destinations take flat data. If your secrets have structure and your consumer wants environment variables, either keep a separate flat file or flatten it yourself with something like yq, deliberately and in code you can review.

Binary Is a Different Animal

$ sops encrypt tls.key > tls.key.enc     # .key is not a known suffix
$ cat tls.key.enc

{
	"data": "ENC[AES256_GCM,data:SQlKePiHyF2vnaY0igbbi21wG7uFcuYzkOpLijoqqOgxKH9PCn0hLreZq9Z4YVUis2tH9KOHwq7dckpKSW3iph2Zexp6KrSEj9NqoW3i8oIV2w036FmAC7lJsevco4RWUoxs92mUjMO0D6Pj9DHbu4L+ffJdMmF7aPOgXU4rHwns2aN2Y7quISJuvXPk4Chj4cwBcEx0ZkJmkw+WHeU+e29+SDDREv5Dp46cFJYMzkg7BjprP3digiE4yP6kPhcJmnmAWHgwEuOCs0PSqji74jW+MMDUfNXCi/iq8cgd2i2PPB+UU+rBdSuQTl/u+2Iw0I2PyQAbNRt+mUanHof4IYcyhgD4hYCdH/Wyr+kceDHJ1wYi28bYVpvS66e4MqY1leCri7Duk7sSBltxsaQFi/eBkROhhjCiEOIppmkYGPLDcUR8Kdxbtw82UDHml7e/KQgIFJt/t1qdctQ8HPEDdiQ61yV3ZXuFEamG9R0m5UCXax0isyln/z6sPZnRipRm7pw3yXIK8eVUCW1nz0zFIA0kdGbuKaaDG4YHV49oiZhkaFM9AQzEaNikq242kneWCUv0LngS74x43AUyANgqeeTTG5oTYFQN2lgcIS7K6qBek4Da60ue2BajEv+30nlWr7/O6k2Upezf6FR626IuBIvnHjiiVi4I99gmhePHi4OhJi9yLv7RXyaDHaYaggrC6CdAUr/Z1IILbKVfFf7GlMJuNcHDNHOBFZ9NxzSgDR7olCcAdPPQvPQ8Jgrw==,iv:rooPr8TkMwUipULLrpdmOHGrB5nO/FquEWRFbcTWj/M=,tag:Flj8nl+dOwgT7f/pvaIiiw==,type:str]",
	"sops": {
		"age": [
			{
				"recipient": "age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p",
				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3NGZtTmRtWjQxN1JYak5B\ncEZFazhhbnhQbVB0bUpBTFRTUHUzQmx5SWdVCjZhMS9jTHNEeldPZkpyazVKM3Jp\nbVl6SHByaTdXRzFwOTRnayttSTZZUlkKLS0tIERwckhCM0c5L1ZBb0llZll5dWhy\nVVJSL08reEN2Vmx5U0hqQ3k0RUZxZVEK/WSyt/Q3yU2xZ5jmBY8u6b0Vtb6Ev7ck\nCXI6sept6zMQKNEOsVKjHVv/Rl2dIfPwOYQUjz6D+yw5Lf6qFEwoAw==\n-----END AGE ENCRYPTED FILE-----\n"
			}
		],
		"lastmodified": "2026-07-22T09:21:07Z",
		"mac": "ENC[AES256_GCM,data:LrtM9ib3ziv005bqFKhPNByt04g8S7y7QjUhUueEFNIiSk38mhnnmp2RLhnHWM0v5BCIkmG1N7yvyNVooh88HmPio1YASrFuAPiy1nfymHutcWLdVGGMRMxisrI7HLcojUujYY/Y81IYAF1aSrwxfDI+sWNmwIMYnq2jL0uM+Is=,iv:cWZmIoJps4ataowrt7clRKlYOzoWukBp/6ABrJf34SI=,tag:v6wmkf7LOwuhyutekJ+yCA==,type:str]",
		"unencrypted_suffix": "_unencrypted",
		"version": "3.13.2"
	}
}

SOPS could not build a tree out of a PEM file, so it invented one: a single item whose key is literally "data" and whose value is your entire file as a string. That one value gets encrypted, and the result is wrapped in a JSON envelope carrying the usual metadata. Decrypt with --output-type binary and you get the original bytes back exactly, which makes binary the correct choice for TLS (Transport Layer Security) private keys, kubeconfigs, Java keystores and tarballs. Nothing about those files benefits from a line-by-line diff anyway.

The cost is everything partial encryption bought you. One value means one line in the diff, so every review of a binary secret is "a large base64 string changed, take my word for it". That much is obvious. The part that bites people is quieter, and it deserves its own section.

When Key Selection Meets a Blob

Remember that the binary store does have a key. Exactly one, named data. Your .sops.yaml key-selection rules do not know or care that the file is a blob, so encrypted_regex (a regular expression, meaning a text-matching pattern, listing which key names should be encrypted) gets applied to that one key like any other. If your pattern does not happen to match the word data, SOPS encrypts nothing at all.

$ cat .sops.yaml
creation_rules:
  - path_regex: .*
    encrypted_regex: ^(password|token)$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
 
$ sops encrypt tls.key > tls.key.enc
$ echo $?
$ sed -n 2p tls.key.enc
$ sops filestatus tls.key.enc
$ grep -c 'ENC\[' tls.key.enc

0
	"data": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDDdCPGgKxV8Zqa\nR7yYtRnMLpQ0vK8sXbFhT2mE4WcYd1oJ9uZgHnBvC3xKeSaPq6TdLmA0jN5wRfIu\n-----END PRIVATE KEY-----\n",
{"encrypted":true}
1

That is a production private key, sitting in cleartext, inside a file that looks encrypted. SOPS exited 0 and printed no warning. Worse, sops filestatus answered {"encrypted":true}, because all it checks is whether valid SOPS metadata is present, and the metadata is perfectly valid. The MAC got encrypted. Your key did not. Commit that and the secret is in Git history forever.

The ENC[ count is what exposes it. A healthy binary envelope has two occurrences, one for data and one for mac. This one has one. So the arithmetic you memorise is short: two means the blob is sealed, one means only the bookkeeping is sealed and your payload is naked. path_regex behaves differently again, because it matches on the filename rather than the contents, so a rule you wrote for structured config can still fire on a blob, pick a recipient list you did not intend, and hand the wrong team decryption rights.

creation_rules:
  # 1. Opaque blobs: TLS keys, kubeconfigs, keystores, certificates.
  #    Whole-file (binary) encryption, and deliberately NO encrypted_regex:
  #    a binary file has exactly one key, named "data", so a regex that
  #    misses it would leave the private key sitting in the clear.
  #    Note the alternation: a file named plain "kubeconfig" has no dot,
  #    so \.kubeconfig$ would never match it.
  - path_regex: (\.(key|pem|p12|jks|crt)|kubeconfig)$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
 
  # 2. Structured config: encrypt the sensitive fields only, so hostnames,
  #    ports and feature flags stay readable in review.
  - path_regex: \.(ya?ml|json)$
    encrypted_regex: ^(data|stringData|password|token|.*[-_](key|secret|token|password))$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
 
  # 3. Flat files use SCREAMING_SNAKE names, so match case-insensitively
  #    and without anchors. ^password$ never matches DB_PASSWORD.
  - path_regex: \.(env|ini)$
    encrypted_regex: (?i)(password|passwd|secret|token|apikey|_key$)
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

Creation rules are evaluated top to bottom and the first match wins, so the narrow blob rule has to sit above the broad structured ones. The split between rules 2 and 3 is not decoration. Anchoring a pattern with ^ and $ means the whole key has to match, and Go's regular expressions are case-sensitive by default, so the tidy-looking ^(password|token)$ silently skips DB_PASSWORD, STRIPE_KEY and API_TOKEN in every dotenv file you own. The (?i) prefix turns matching case-insensitive. Drop the anchors when your keys carry prefixes.

One setting is missing from that file on purpose. mac_only_encrypted: true narrows the file's MAC to the encrypted values, so a teammate or a bot can hand-edit a readable hostname without breaking anything. The price is that tamper detection stops covering the readable half: a quiet edit to a namespace or an image reference then sails through, which is the attack Partial encryption & structure walks you through. Leave it off and an outside edit to a plaintext field fails decryption with "MAC mismatch. File has B7156D73..., computed D0C5E154..." and exit code 51, which reads like tampering when it was a one-word hostname change. That noise is the check doing its job. Turn the setting on per file, on a path narrow enough to name the manifests a controller has to rewrite, never on every YAML and JSON file in the repo.

Proving It Did What You Meant

$ sops filestatus tls.key.enc
$ grep -c 'ENC\[' tls.key.enc
$ grep -c 'ENC\[' app.enc.env
$ sops decrypt --output-type binary tls.key.enc | sha256sum
$ sha256sum tls.key

{"encrypted":true}
2
4
eb34509a745606c14b33f40229dcf8435577caf7103ba958e56fcbba7095f880  -
eb34509a745606c14b33f40229dcf8435577caf7103ba958e56fcbba7095f880  tls.key

The filestatus subcommand arrived in 3.9 and picked up its own --input-type in 3.10. It answers exactly one question: does this file carry valid SOPS metadata? It cannot tell you whether the encryption did what you intended, as the cleartext private key above demonstrated. The ENC[ count is the check that catches real mistakes. Two for a binary envelope. For a structured file, one per encrypted value plus one for the MAC, which is why app.enc.env shows four when all three of its values were encrypted. Add a key-selection regex and that number should drop to exactly the count of fields you meant to protect, and no lower. If you expected a dozen and got two, you encrypted a blob by accident.

For binary files the sha256sum round trip is the proof that matters. Decrypt to standard output, hash it, compare against the original before you delete the plaintext. Do this once per new file type you start encrypting, and never delete a plaintext key until that comparison has passed.

Handing Secrets to a Running Process

The safest home for a decrypted secret is a process's memory, and sops exec-env is the shortest path there. It decrypts a flat file, sets each key as an environment variable, runs your command as a child, and lets the values die when the child exits. No plaintext file is written at any point, so there is nothing for a backup agent to sweep up, nothing left behind if the container is later inspected, and nothing on a disk image if the host is seized or stolen.

$ sops exec-env --pristine app.enc.env 'env'

LOG_LEVEL=info
STRIPE_KEY=sk_live_51H7qVdKp2mQx8LrTn4Ba
DB_PASSWORD=hunter2
PWD=/home/dev/app

The --pristine flag starts the child with only the decrypted values instead of inheriting yours, which is a cheap way to stop a stray AWS_SECRET_ACCESS_KEY from following your app into a subprocess. PWD is the shell adding its own bookkeeping, because SOPS runs your command through /bin/sh -c. Now the sharp edge: exec-env has no --input-type flag at all. The format comes from the filename and there is no way to override it.

$ sops encrypt .env.production > .env.production.enc   # unknown suffix -> binary
$ sops exec-env --pristine .env.production.enc 'env'

data=DB_PASSWORD=hunter2
STRIPE_KEY=sk_live_51H7qVdKp2mQx8LrTn4Ba
LOG_LEVEL=info
 
PWD=/home/dev/app

One variable, named data, holding the whole file as a string. Your app starts with DB_PASSWORD unset, connects to nothing, and the stack trace points at your database driver rather than at your secret handling. Rename the file so it ends in .env and the problem disappears, which is the entire fix. When a tool insists on reading a real path instead of the environment, use exec-file.

$ sops exec-file --input-type binary --filename kubeconfig kubeconfig.enc 'ls -l {}'

prw------- 1 dev dev 0 Jul 22 09:31 /tmp/.sops3554727244/kubeconfig

That leading p in prw------- means named pipe, also called a FIFO (first in, first out). The size is zero because nothing is stored: SOPS writes the plaintext into one end while your tool reads the other, so the data crosses through kernel memory and never lands on the filesystem. The {} in your command string is replaced with the pipe's path. The trade-off is that a pipe can be drained once, so a tool that opens the file twice hangs on the second read. Pass --no-fifo and SOPS writes a real temporary file instead, mode 0600, in the same .sops temp directory, named tmp-file followed by a random number unless you set --filename. That file is deleted when the command finishes, but until then the plaintext is on disk, so keep --no-fifo for tools that genuinely need it. Both exec commands also take --user to drop to an unprivileged account before running, which matters when your build runner starts as root.

The Tooling That Wraps It

When a wrapper is overkill and you want one value, --extract walks the decrypted tree with a path expression and prints a single leaf. It is the right tool inside a shell script, and it keeps the other secrets in the file out of your terminal history and scrollback.

$ sops decrypt --extract '["database"]["password"]' secrets.enc.yaml

s3cr3t-p0stgr3s-pw

Above that sits an ecosystem, all of it built on the same encrypt and decrypt calls. helm-secrets decrypts values files for the duration of a Helm release and cleans up afterwards. KSOPS plugs into Kustomize as a generator, so encrypted manifests become plain Secrets at build time. Flux decrypts SOPS files itself through a decryption.provider setting on the Kustomization, Argo CD usually goes through KSOPS, and the External Secrets Operator (ESO) points the other way entirely, pulling from a hosted secrets manager rather than from Git. Those each get their own lesson.

data "sops_file" "db" {
  source_file = "secrets/db.enc.yaml"
}
 
resource "aws_db_instance" "app" {
  # dotted path into the decrypted tree
  password = data.sops_file.db.data["database.password"]
}

The carlpett/sops Terraform provider deserves one specific warning before you wire it in. Every value you read through sops_file is written into terraform.tfstate in plaintext. That is not a provider bug, it is how Terraform state works for every sensitive attribute. So the moment you add that data source, your state file becomes exactly as sensitive as the age key that protects the secret (age is the modern encryption tool whose public keys start with age1), and it needs the same treatment: a remote backend with encryption at rest, versioning switched on, a bucket policy restricted to the roles that run plans, and no state file anywhere near a laptop or a build artifact. Teams get this wrong constantly. They encrypt a database password beautifully in Git, then leave it sitting in cleartext in an Amazon S3 (Simple Storage Service) bucket half the company can list.

Try this

Run sops encrypt app.env on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: an unknown suffix silently becomes a binary blob. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related