Your overlay sets disableNameSuffixHash: true for the app-db generator, and the Secret it produces is marked immutable. You change DB_PASSWORD in db.env, rebuild, and apply. What happens?

The generated name is unchanged, the apply is refused, and the running pods carry on serving with the previous password. A frozen name means the Deployment reference never moves, and an immutable Secret will not accept new data, so the rotation dies twice over.

A teammate copies your .sops.yaml but leaves out the encrypted_regex line, keeping path_regex and the age recipient. They seal secret.enc.yaml and open a pull request. What does the reviewer find?

Every value in the document is ciphertext, so the reviewer cannot tell which resource it is and tooling that keys off kind gives up. With no filter, apiVersion and kind get sealed alongside the password, so the diff says nothing useful and anything reading the file before decryption is blind.

A job on a shared CI runner executes kustomize build --enable-alpha-plugins --enable-exec overlays/prod | tee manifests.yaml, with KSOPS installed and working. Why should that line never have been written?

The rendered output holds real Secret data, so tee writes the live credential into a file and into the job log at once. Once the plugin decrypts, stdout carries the same value base64 -d would hand back, and job logs usually have a wider audience than the cluster does.

Courses KustomizeSecrets done safely

Secrets done safely

Generators, SOPS, and the pitfalls.

Advanced14 min · lesson 9 of 12

A Kubernetes Secret looks like a locked drawer, but the front panel is glass. The base64 encoding everyone points to (a way of rewriting raw bytes using 64 printable characters) is a language, not a lock. Anyone who can read the manifest can read the value sitting inside it. So storing secrets safely splits into two separate jobs. The first is getting real values into a Secret without ever pasting them into YAML (the text format Kubernetes manifests are written in) that you commit to Git. That job belongs to Kustomize's secretGenerator. The second is the awkward one. GitOps (running your cluster from whatever is committed in Git) wants the secret itself in the repository, so you have to make sure Git only ever sees ciphertext. That job belongs to SOPS (a tool that encrypts the values inside a file) and a decrypt-at-build plugin called KSOPS. The generator mechanics and the content hash are covered in the generators lesson. Here the whole focus is the security posture: what touches disk, what touches Git, and what leaks into your CI (continuous integration) logs.

The entire problem fits in one command. Take the base64 string out of a Secret's data field and decode it.

terminal

# base64 in a Secret's data field is reversible by anyone who can read it
$ echo 'UzNjcjN0IQ==' | base64 -d

output

S3cr3t!

There was no password prompt, no key, and no permission check. If that string is in a file you committed, the credential is in Git, readable by everyone with clone access and kept in history forever, even after you delete the line. So the first job is to keep the plaintext out of the manifest to begin with.

Keep the plaintext out of the manifest

Never hand-write a Secret with an inline data block. That base64 string is the credential, and once it lands in history it stays there. secretGenerator works like a recipe card that names its ingredients instead of copying them onto the card: it assembles the Secret at build time from files or env files that you keep out of version control, either gitignored or written fresh by CI. Prefer envs and files over literals, because a literal puts the value straight back into kustomization.yaml, which is the thing you were trying to avoid. The env file lists key=value pairs. A files entry turns the contents of a file (a TLS certificate, say, the credential that proves a server's identity) into the value for one key. This is also what keeps rotation honest: because the value lives in a file and not in the manifest, rotating it means editing one gitignored file, not surgery on YAML that sits in the repo.

kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
secretGenerator:
  - name: app-db
    type: Opaque
    envs:
      - db.env            # key=value pairs; this file is gitignored
    files:
      - ca.crt=tls/ca.crt # file contents become the value for key ca.crt
 
generatorOptions:
  disableNameSuffixHash: false  # keep the hash: a new value rolls the pods

db.env

# db.env  -  never committed (add *.env to .gitignore)
DB_USER=app
DB_PASSWORD=S3cr3t!

The name you get back is not app-db. It is app-db with a suffix like 9t2kd7f4hb, a short hash of the Secret's contents. That suffix is the rotation guarantee, and it is the reason to care about generators for security at all. Change the password, and the Secret's name changes, so the Deployment that mounts it now references a new object, and Kubernetes rolls the pods onto the new value. Leave disableNameSuffixHash at its default of false for anything sensitive. Flip it to true and the name stops moving, which quietly breaks rotation. Pair that with an immutable Secret and you cannot even update the value in place: the next apply is rejected and the pods keep the old credential.

terminal

$ kustomize build overlays/prod | grep 'name: app-db'

output

  name: app-db-9t2kd7f4hb

terminal

# rotate the value, rebuild: the content hash (and the pods) roll
$ sed -i 's/^DB_PASSWORD=.*/DB_PASSWORD=N3wP@ss/' db.env
$ kustomize build overlays/prod | grep 'name: app-db'

output

  name: app-db-6bd8hfk2t9

Encrypt before it reaches Git

secretGenerator keeps plaintext out of the manifest, but it does nothing about the secret you actually want to commit. For that you need encryption on the file itself, and SOPS (the name is short for Secrets OPerationS) is the standard tool. SOPS seals the values inside a YAML file while leaving the structure readable, the way an envelope hides the letter but not the address on the front. Your diffs still show which keys changed without showing what they changed to. Back it with age (a small public-key encryption tool) for a small team, or with a cloud KMS (Key Management Service, such as AWS KMS, GCP KMS, or Azure Key Vault) or HashiCorp Vault when you want one place to set key policy and read an audit trail. The choice is a security decision, not a convenience one. age keys are simple, but you manage their distribution and revocation yourself, while a KMS lets you revoke a key centrally and see who decrypted what.

A .sops.yaml file holds creation rules so an engineer runs sops on a file and the right key is chosen automatically, with no flags to remember. One sharp edge trips people up. SOPS matches those rules against the filename you hand it, so the plaintext file has to already match your path_regex before you encrypt it. Name the file secret.enc.yaml from the start, write your values in, then encrypt it in place so the filename never changes and the rule keeps matching. The setting that matters most for readable diffs is encrypted_regex. Scope it to data and stringData so the keys, the metadata, and the resource kind all stay in cleartext, and only the sensitive values turn into ciphertext. Get that scope wrong in either direction and you pay for it. Leave encrypted_regex off entirely and SOPS encrypts every value in the document, including the values of apiVersion and kind, so the file still decrypts but every diff turns to noise and any tool that reads the manifest before decryption breaks. Set it too narrowly and real secret values get left in cleartext. Scope it to '^(data|stringData)$', then open the committed file by eye and confirm the values, and only the values, show up as ENC[...] before you push.

.sops.yaml

creation_rules:
  - path_regex: .*\.enc\.yaml$
    encrypted_regex: '^(data|stringData)$'   # encrypt values only, keep keys diff-able
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8j

terminal

# author secret.enc.yaml locally with plaintext values, then seal it in place
$ sops --encrypt --in-place secret.enc.yaml
$ cat secret.enc.yaml

output

apiVersion: v1
kind: Secret
metadata:
    name: app-db
type: Opaque
data:
    password: ENC[AES256_GCM,data:9Fb2xQ7mKw3pL1vT,iv:Kp3n5wQ8vX2pL7wR4mZ0kBcT9gN1jY6dW5xE2sV4uH8=,tag:Rd8h2QpVnK7wZ3xL9cT0mA==,type:str]
sops:
    age:
        - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8j
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBB...
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-07-17T10:22:04Z"
    mac: ENC[AES256_GCM,data:vY7k2Qm9wX4pL1vT8sR3nB6hK0jY5dW2xE9sV4uH7aZ0kBcT5gN1jY6dW3xE8sV2uH5aQ9wX4pL7vT1sR6nB0h==,iv:Zx8vK2pL7wR4mZ0kBcT9gN1jY6dW5xE2sV4uH8aQ9wX=,tag:Bn6kT0mA9wX4pL7vT1sRuH==,type:str]
    encrypted_regex: ^(data|stringData)$
    version: 3.9.4

Decrypt at build with KSOPS

Kustomize has no idea SOPS exists. KSOPS (the Kustomize plugin that runs SOPS) bridges the gap. It is a KRM (Kubernetes Resource Model) generator plugin that Kustomize calls during a build: it reads your encrypted files, hands them to SOPS to decrypt in memory, and emits real Secret resources into the output stream. You wire it in through the generators field, pointing at a small KSOPS manifest, not through secretGenerator. Because KSOPS is an exec plugin, meaning it shells out to a separate binary, the build needs two opt-in flags, --enable-alpha-plugins and --enable-exec, and the ksops binary has to be installed in your Kustomize plugin home directory. In a GitOps setup this same plugin gets baked into the Argo CD repo-server or the Flux image so the controller can decrypt on the cluster, which is the GitOps lesson's territory.

secret-generator.yaml

# secret-generator.yaml  -  the resource KSOPS recognizes
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: app-secret
files:
  - ./secret.enc.yaml

kustomization.yaml

# kustomization.yaml  -  reference it under generators, not secretGenerator
generators:
  - ./secret-generator.yaml

terminal

# KSOPS is an exec plugin, so BOTH alpha flags are mandatory
$ kustomize build --enable-alpha-plugins --enable-exec overlays/prod

output

apiVersion: v1
data:
  password: UzNjcjN0IQ==
kind: Secret
metadata:
  name: app-db
type: Opaque

How a secret stays sealed until build

1age / KMS key

private key never enters the repo

2sops --encrypt

data values become ENC[...]

3commit .enc.yaml

Git only ever holds ciphertext

4KSOPS at build

decrypts in memory, emits a Secret

The private key gates the last step. Git and its history hold ciphertext only, and decryption happens in memory during kustomize build.

kustomize build prints your secrets in the clear

The moment KSOPS runs, kustomize build writes fully decrypted, base64-encoded Secrets to stdout (the stream a command prints to). Any step that echoes that manifest, or tees it to a file (writing it out while still printing it), drops the live credential into your logs. So does a GitOps controller that logs rendered resources at debug level. Those pipeline logs are often readable by more people than the cluster is. Never run kustomize build | tee manifests.yaml on a shared runner, mask the step that renders secrets, and check the log verbosity on your Argo CD repo-server and Flux controllers.

Check it actually worked

A defender does not trust that this worked; they check. Two checks catch almost everything. First, ask Git directly what it is storing. git show against the committed file should return ciphertext for every value under data, reaching back into history as well. Second, put yourself in the attacker's shoes: clone the repo, strip away the private key, and try to decrypt. Without an identity that matches one of the recipients, SOPS refuses and hands back nothing useful. That failing command is the proof that repo access alone buys an attacker encrypted bytes, not your database password.

terminal

# prove Git holds ciphertext, even in history
$ git show HEAD:overlays/prod/secret.enc.yaml | grep 'password:'

output

    password: ENC[AES256_GCM,data:9Fb2xQ7mKw3pL1vT,iv:Kp3n5wQ8vX2pL7wR4mZ0kBcT9gN1jY6dW5xE2sV4uH8=,tag:Rd8h2QpVnK7wZ3xL9cT0mA==,type:str]

terminal

# someone with clone access but no private key gets nothing
$ SOPS_AGE_KEY_FILE=/dev/null sops --decrypt secret.enc.yaml

output

Failed to get the data key required to decrypt the SOPS file.
 
Group 0: FAILED
  age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8j: FAILED
    - | no identity matched any of the recipients
 
Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

Make that first check automatic. Add a pre-push hook, or a CI step, that runs git show on each staged .enc.yaml and greps the data block: if any value is not wrapped in ENC[...], fail the push. A file that slips through unencrypted is expensive to undo, because scrubbing a secret out of Git history means rewriting every commit that touched it and forcing every clone to reset. Catch it before it ever reaches a remote.

Try this

Run echo 'UzNjcjN0IQ==' | base64 -d on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: kustomize build prints your secrets in the clear. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Components & reuseRemote bases & supply chain

What Helm is: charts & releasesHelm Kustomize cheat sheetCheat sheet Helm vs KustomizeComparison

Secrets done safely

Keep the plaintext out of the manifest

Encrypt before it reaches Git

Decrypt at build with KSOPS

Check it actually worked

Try this

Takeaway

Related