Kustomize: template-free overlays

Patch manifests per environment.

A recipe card written out in full, plus a sticky note on top that reads "for the party, triple the batch and use the good chocolate." The card still works on its own. The note takes two seconds to read. Kustomize does exactly that for Kubernetes manifests, the YAML files that tell a cluster what to run. (YAML is a plain-text format for structured data, all colons and indentation.) A base directory holds the full, valid manifests. An overlay directory holds a short list of what one environment changes.

Helm, the usual Kubernetes package manager, takes the other road. It hands you a form letter full of blanks and a separate file of answers to fill them in, so what sits in your Git repository is not valid Kubernetes YAML until a template engine renders it. Kustomize has no blanks and no template engine. Every file is a real manifest you could hand to the cluster right now. There is nothing to install either, because kustomize has shipped inside kubectl (the Kubernetes command line tool) since kubectl 1.14. Two subcommands do all the work: kubectl kustomize prints the result, and kubectl apply -k sends it to the cluster.

$ kubectl version --client

Client Version: v1.36.1
Kustomize Version: v5.8.1

That second line matters more than people expect. The copy of kustomize baked into kubectl is a frozen snapshot, and it lags behind the standalone kustomize binary. A kustomization.yaml that uses a field added last quarter will build cleanly with the standalone tool and fail with an unknown-field error under an older kubectl. Decide which one your pipeline uses, print its version into the build log, and run the same one on your laptop. Two engineers rendering different YAML from the same commit is a miserable way to spend a Friday.

The Base Holds Your Security Defaults

The base is the application with nothing environment-specific in it. One replica, the real image, the real ports. It is also the right home for your hardening, because anything you put here is inherited by every overlay, whether or not the person writing that overlay was thinking about security that afternoon. Defaults in the base cannot be forgotten. Defaults copied into each overlay get forgotten during the first busy week.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api
  labels:
    app: payments-api
spec:
  replicas: 1
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 10001
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: api
          image: registry.acme.internal/payments-api:1.4.2
          ports:
            - name: http
              containerPort: 8080
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              cpu: 500m
              memory: 256Mi

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
resources:
  - deployment.yaml
  - service.yaml

Read those pod settings again, because each one shuts a specific door. runAsNonRoot with runAsUser: 10001 stops the container starting as root, the account on a Linux system that is allowed to do anything. readOnlyRootFilesystem: true means an attacker who wins remote code execution (getting your process to run commands of their choosing) cannot drop a binary into the container's own filesystem. capabilities.drop: [ALL] hands back the Linux kernel privileges every container is granted by default, things like opening raw network sockets or changing the owner of a file. seccompProfile: RuntimeDefault switches on the runtime's system-call filter, so the process may only make the requests to the kernel (the core of the operating system, the part that talks to the hardware) that ordinary workloads need. And automountServiceAccountToken: false keeps the pod's cluster API credential out of the container unless the application genuinely calls the Kubernetes API.

None of this is Kustomize-specific, and that is the point. The base is plain Kubernetes, so every schema checker, scanner and editor already understands it. One caveat before you try it: do not run kubectl apply -f k8s/base/ against the whole directory. kubectl picks up every YAML file in there, including kustomization.yaml, and stops with a complaint that it does not recognise the Kustomization kind. Point it at deployment.yaml and service.yaml individually, or use kubectl apply -k.

The Overlay Lists Only the Differences

An overlay is a directory with its own kustomization.yaml that points back at the base and then states its changes. Nothing else belongs there. Reading the prod overlay should tell you the whole story of how production differs from the base. If it does not, somebody has hidden a difference somewhere it will never be reviewed.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
 
resources:
  - ../../base            # start from the shared, hardened manifests
 
namespace: payments       # every resource lands here
namePrefix: prod-         # prod-payments-api, not payments-api
 
labels:
  - pairs:
      environment: prod
      app.kubernetes.io/part-of: payments
    includeSelectors: false   # never rewrite matchLabels on a live Deployment
    includeTemplates: true    # but do label the pods themselves
 
images:
  - name: registry.acme.internal/payments-api
    digest: sha256:1c1a0f3f6e3a4d5b8c9e2f7a1b4c6d8e0f2a4b6c8d0e2f4a6b8c0d2e4f6a8b0c
 
patches:
  - path: replicas-patch.yaml

Four different mechanisms are at work in that file. Picture four rubber stamps coming down on every sheet in the stack. namespace and namePrefix are transformers: they rewrite one field across every resource that came from the base, so both the Deployment and the Service become prod-payments-api inside the payments namespace (a named partition of the cluster that keeps one team's objects apart from another's). labels stamps key/value pairs onto metadata. images rewrites the container image, and here it pins by digest, the SHA-256 content hash of the exact image bytes, rather than by newTag, so nobody can move the 1.4.2 tag under you between the render and the rollout. patches applies a file that edits named fields. There is also a replicas shorthand in kustomization.yaml if a pod count is the only thing you want to change.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payments-api      # the BASE name, before namePrefix is applied
spec:
  replicas: 5

That file is a strategic merge patch, which is Kubernetes' own idea of a smart edit. Treat it like marking up a printed page instead of retyping it. Every field you write overwrites the matching field in the target, and everything you leave off stays exactly as it was. Lists merge on a key rather than being replaced wholesale, so a patch listing - name: api under containers edits that one container and leaves its neighbours alone. Two details bite people. Setting a field to null deletes it, and hold that thought, because it comes back later in this lesson. And metadata.name has to be the name as it appears in the base, because patches run before the prefix and namespace transformers. Name the prefixed resource by mistake and the build stops.

# replicas-patch.yaml edited to say  name: prod-payments-api
$ kubectl kustomize k8s/overlays/prod

error: no resource matches strategic merge patch "Deployment.v1.apps/prod-payments-api.[noNs]": no matches for Id Deployment.v1.apps/prod-payments-api.[noNs]; failed to find unique target for patch Deployment.v1.apps/prod-payments-api.[noNs]

Render Before You Apply

The overlay is a promise. The render is the truth. kubectl kustomize prints the manifests that kubectl apply -k would send to the API server, and reading them costs you ten seconds.

$ kubectl kustomize k8s/overlays/prod | head -32

apiVersion: v1
kind: Service
metadata:
  labels:
    app: payments-api
    app.kubernetes.io/part-of: payments
    environment: prod
  name: prod-payments-api
  namespace: payments
spec:
  ports:
  - name: http
    port: 80
    targetPort: http
  selector:
    app: payments-api
  type: ClusterIP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: payments-api
    app.kubernetes.io/part-of: payments
    environment: prod
  name: prod-payments-api
  namespace: payments
spec:
  replicas: 5
  selector:
    matchLabels:
      app: payments-api

Check the interesting parts. Both objects picked up the prefix and both landed in the payments namespace. The two new labels appear under metadata.labels. The Service's spec.selector still reads app: payments-api on its own, because the overlay set includeSelectors: false, and the Deployment's matchLabels on the last line is untouched for the same reason. Further down, the image renders as registry.acme.internal/payments-api@sha256:1c1a0f..., the pod template carries the new labels, and the base securityContext block survives intact. Kustomize sorts its output by kind, which is why the Service prints before the Deployment.

Against a live cluster, kubectl diff -k does one better. It asks the API server what the outcome would be and shows it against what is running right now, including drift somebody caused by hand at two in the morning.

$ kubectl diff -k k8s/overlays/prod; echo "exit=$?"

diff -u -N /tmp/LIVE-1264753918/apps.v1.Deployment.payments.prod-payments-api /tmp/MERGED-3947281056/apps.v1.Deployment.payments.prod-payments-api
--- /tmp/LIVE-1264753918/apps.v1.Deployment.payments.prod-payments-api	2026-07-21 09:41:12.113455212 +0000
+++ /tmp/MERGED-3947281056/apps.v1.Deployment.payments.prod-payments-api	2026-07-21 09:41:12.117455212 +0000
@@ -20,7 +20,7 @@
   name: prod-payments-api
   namespace: payments
 spec:
-  replicas: 3
+  replicas: 5
   revisionHistoryLimit: 10
   selector:
     matchLabels:
@@ -40,7 +40,7 @@
     spec:
       automountServiceAccountToken: false
       containers:
-      - image: registry.acme.internal/payments-api:1.4.1
+      - image: registry.acme.internal/payments-api@sha256:1c1a0f3f6e3a4d5b8c9e2f7a1b4c6d8e0f2a4b6c8d0e2f4a6b8c0d2e4f6a8b0c
         imagePullPolicy: IfNotPresent
         name: api
         ports:
exit=1

Mind that exit code. kubectl diff returns 0 when there is no difference, 1 when there is one, and higher than 1 when the command itself failed. That is handy in a pipeline and hostile inside a shell script running under set -e, where a perfectly normal difference kills the run. Write it as kubectl diff -k k8s/overlays/prod || true and decide what to do with the output yourself.

The One-Line Change That Opens a Hole

Here is the review problem, and it is the reason this lesson exists. Someone opens a pull request (a proposed change sitting in a queue waiting for a colleague to approve it) against your prod overlay. The entire diff is one line.

$ git diff origin/main -- k8s/overlays/prod/

diff --git a/k8s/overlays/prod/kustomization.yaml b/k8s/overlays/prod/kustomization.yaml
index 8f3c1a2..b41d9e7 100644
--- a/k8s/overlays/prod/kustomization.yaml
+++ b/k8s/overlays/prod/kustomization.yaml
@@ -20,3 +20,4 @@ images:
 
 patches:
   - path: replicas-patch.yaml
+  - path: debug-patch.yaml

debug-patch.yaml is already in the repository. It went in months ago for a real debugging session, got reviewed then, and was forgotten. The diff looks harmless, the schema check in CI (continuous integration, the automated checks that run on every commit) passes because the YAML is perfectly valid, and the approver clicks the green button. So stop reviewing the overlay. Render both sides and compare the results.

$ git worktree add --detach /tmp/base-render origin/main
$ kubectl kustomize /tmp/base-render/k8s/overlays/prod > /tmp/before.yaml
$ kubectl kustomize k8s/overlays/prod              > /tmp/after.yaml
$ diff -u /tmp/before.yaml /tmp/after.yaml

Preparing worktree (detached HEAD 9f31c4a)
HEAD is now at 9f31c4a payments: prod overlay refresh
--- /tmp/before.yaml	2026-07-21 09:52:41.118392201 +0000
+++ /tmp/after.yaml	2026-07-21 09:52:41.640118773 +0000
@@ -52,13 +52,19 @@
             cpu: 100m
             memory: 128Mi
         securityContext:
-          allowPrivilegeEscalation: false
           capabilities:
             drop:
             - ALL
-          readOnlyRootFilesystem: true
+          privileged: true
+        volumeMounts:
+        - mountPath: /host
+          name: host-root
       securityContext:
         runAsNonRoot: true
         runAsUser: 10001
         seccompProfile:
           type: RuntimeDefault
+      volumes:
+      - hostPath:
+          path: /
+        name: host-root

One line of review became eight added lines and two deletions in what actually runs. privileged: true is the loud one. A privileged container is handed every device on the host, gets /proc and /sys mounted without the usual masking, and has its seccomp and AppArmor confinement switched off, so that RuntimeDefault filter still sitting three lines below is now decoration. The hostPath volume mounts the node's whole root filesystem at /host. A node is one of the machines the cluster runs on, so that one directory puts the kubelet's credentials (the kubelet is the agent Kubernetes runs on every machine to start and stop containers), every other pod's files, and the host's SSH keys inside reach of a single compromised process.

Now look at what disappeared, because that is the half nobody catches. readOnlyRootFilesystem: true is gone, deleted by a null in the patch. allowPrivilegeEscalation: false had to go as well, because the API server refuses any container that sets that field false while privileged is true, so making the first control fail quietly required removing a second one. Neither deletion appears as an added line anywhere. Reading the overlay diff shows you a filename. It cannot show you a field that stopped existing.

Make it a rule rather than a habit: the artifact you review is the rendered YAML, not the overlay. Render the merge base (the commit the branch grew out of), render the branch, diff the two, and fail the build on anything that grants host-level access. Policy engines handle this part well, since Conftest and the Kyverno command line tool both read rendered manifests on standard input, and Pod Security admission is your backstop inside the cluster, though only in namespaces you have actually labelled for it. Kustomize itself will never save you here. It renders that privileged container without a murmur, because merging YAML and printing it is the entire job.

What Breaks in Practice

Two Kustomize behaviours cause most of the real incidents, and both come down to identity: how the cluster works out which pods belong to which object. The first is selector labels.

# the overlay was switched to includeSelectors: true, against a running app
$ kubectl apply -k k8s/overlays/prod

service/prod-payments-api configured
The Deployment "prod-payments-api" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{"app":"payments-api", "app.kubernetes.io/part-of":"payments", "environment":"prod"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable

includeSelectors: true, along with the older commonLabels field which always behaves this way and now prints a deprecation warning, adds your labels to spec.selector.matchLabels on a Deployment and to a Service's spec.selector. On a brand new resource that is fine and usually what you wanted. On something already running it is fatal, because a Deployment's selector cannot be changed after creation: your choices are deleting and recreating the Deployment, which is an outage, or leaving the selector alone. Look closely at what the Service did in that output, though. A Service selector is mutable, so it took the change and reported configured. It is now hunting for pods labelled environment: prod, the Deployment that would have produced them just failed to apply, and your Service is pointing at nothing. When you retrofit Kustomize onto live workloads, use includeSelectors: false with includeTemplates: true, the pairing in the overlay above. Pods pick up the new labels for queries and for NetworkPolicy (the Kubernetes firewall rules, which choose their targets by label), and no selector is ever touched.

The second is renaming. Change namePrefix from prod- to production- and nothing gets renamed in the cluster. kubectl apply -k creates a fresh set of objects under the new names and leaves the old ones running, quietly doubling your pods and your bill. kubectl apply keeps no memory of what it applied last time. Cleaning up needs either --prune with a label selector, a sharp tool that is easy to aim at the wrong things, or a GitOps controller such as Argo CD or Flux. Those sit inside the cluster, watch the repository, keep a record of everything they created, and delete whatever leaves Git.

Helm and Kustomize Are Not Rivals

Helm is a package manager in the sense that apt or Homebrew are: templates, versioned releases, helm rollback, and a large catalogue of charts you can install. That is what you want for shipping software to strangers, or for installing a database you did not write. Kustomize is a customizer: no placeholders, valid YAML from end to end, built into kubectl, and a diff a human can actually read. That suits most teams for their own applications. Picking by ideology wastes meetings. Grown-up setups run both.

They compose in three directions, too. kubectl kustomize --enable-helm inflates a chart inside a kustomization, so you can patch a vendor's output without forking it. Helm's --post-renderer pipes a chart's rendered manifests through kustomize before install, which is the standard trick for bolting a missing securityContext onto third-party software. And Argo CD and Flux both spot either layout in a repository and render it for you.

Neither tool makes a manifest secure. Both are YAML producers, and whatever comes out the far end still has to stand on its own merits: non-root pods, dropped capabilities, resource limits, no host mounts, pinned image digests, no wildcard RBAC (role-based access control, the rules deciding who may do what in the cluster). Pin the version of everything external, whether that is a chart, a remote base, or the kustomize binary itself. Then render with helm template or kubectl kustomize and read the result, because the render is what runs.

Keep the Render

One habit is worth the ten minutes it costs to set up. Write the rendered YAML to a file on every build, gate on the difference, and keep the file as a build artifact tagged with the commit.

#!/usr/bin/env bash
set -euo pipefail
 
OVERLAY="k8s/overlays/prod"
BASE_REF="${1:-origin/main}"
WORK="$(mktemp -d)"
trap 'git worktree remove --force "$WORK/base" 2>/dev/null || true; rm -rf "$WORK"' EXIT
 
git worktree add --detach "$WORK/base" "$BASE_REF" >/dev/null 2>&1
kubectl kustomize "$WORK/base/$OVERLAY" > "$WORK/before.yaml"
kubectl kustomize "$OVERLAY"            > "$WORK/after.yaml"
 
# keep the render: this file is what actually ships
mkdir -p artifacts
cp "$WORK/after.yaml" "artifacts/prod-$(git rev-parse --short HEAD).yaml"
 
# diff exits 1 when the files differ, which is the normal case here
diff -u "$WORK/before.yaml" "$WORK/after.yaml" > "$WORK/render.diff" || true
 
# Read the file directly, never pipe into grep -q. It exits at the first match,
# the writer upstream takes SIGPIPE and returns 141, and under `set -o pipefail`
# that 141 becomes the status of the whole pipeline, so the gate quietly passes.
if grep -qE '^\+[^+].*(privileged: true|hostPath|hostNetwork: true|hostPID: true)' \
     "$WORK/render.diff"; then
  echo "BLOCKED: this change adds host-level access to production" >&2
  cat "$WORK/render.diff" >&2
  exit 1
fi

When a pod starts crash-looping at 03:00, or an auditor asks what was running in production on the 14th, you do not have to reconstruct the answer from a base, an overlay, three patches and whichever kustomize version happened to be on the runner that week. You open the file for that commit and read it.

Try this

Run kubectl version --client on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a remote base is someone else's YAML running in your cluster. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related