Kustomize in GitOps & securing it

Argo/Flux, build, scan, deploy.

A kitchen runs from one recipe book. The book is the truth. Whatever lands on the plate has to match it, and if a cook changes a dish on a whim, the head chef pulls the book, spots the difference, and puts the plate back to spec. GitOps (running your cluster straight from a Git repository, where Git holds the one true copy of what should be deployed) works the same way. Git stores the desired state. A controller living inside the cluster keeps asking one question: does what is running match what is written? When the answer is no, it re-applies until the answer is yes.

Kustomize sits inside this model cleanly. An overlay (a folder of small patches that bends a shared base configuration to fit one environment, like prod or staging) is plain YAML (a text format for describing configuration) in a Git folder. You stop running kustomize build or kubectl apply -k by hand. You point the controller at the overlay directory, it renders the YAML and applies it, and the cluster tracks whatever is in Git. That splits your security work in two. Make the thing that gets built easy to review. Give the thing that does the applying the smallest key you can.

How Argo CD And Flux Build Your Overlay

Two controllers do most of this work: Argo CD (a continuous delivery tool that syncs a cluster to Git) and Flux (a set of small Kubernetes controllers that do the same job). Both understand kustomize on their own, so you never commit a pre-rendered blob. You commit the overlay and let the controller render it. Here is how you tell Argo CD where to look.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shop-prod
  namespace: argocd
spec:
  project: shop
  source:
    repoURL: https://github.com/acme/shop-manifests.git
    targetRevision: main
    path: overlays/prod          # a kustomization.yaml lives here
  destination:
    server: https://kubernetes.default.svc
    namespace: shop
  syncPolicy:
    automated:
      prune: true                # delete what Git no longer declares
      selfHeal: true             # revert manual cluster edits

Argo CD sees a kustomization.yaml at that path and runs kustomize build itself, on its repo-server, inside the cluster. The syncPolicy.automated block is the part with teeth. prune: true deletes anything you remove from Git. selfHeal: true means if someone edits the live cluster by hand, Argo overwrites their change on the next reconcile (the controller's periodic check that what is running still matches what Git declares). Flux describes the same idea with two resources.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: shop
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/acme/shop-manifests.git
  ref:
    branch: main
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: shop-prod
  namespace: flux-system
spec:
  interval: 10m
  path: ./overlays/prod
  prune: true
  sourceRef:
    kind: GitRepository
    name: shop
  targetNamespace: shop

GitRepository is the source: where to fetch and which branch. The Flux Kustomization (a Flux custom resource, not the kustomization.yaml file itself) tells the kustomize-controller which path to build and how often to reconcile. Both controllers run kustomize build server-side, using their own credentials. Hold that thought, because those credentials are the prize an attacker wants.

This reconcile loop is a security control in its own right. If an attacker gets a foothold and runs kubectl to add a privileged sidecar (an extra container slipped into a pod next to your app, sharing its network and often its secrets), self-heal wipes it on the next pass and the cluster snaps back to what Git says. Every real change now flows through a commit, so the repository doubles as an audit log of who changed what and when. Require signed commits, which Argo CD and Flux can both verify, and that trail gets hard to forge.

Render And Scan Before The Merge

The moment Git becomes the deploy button, a bad merge is a bad deploy. So you put a gate in front of the merge. Continuous integration (CI, the automated checks that run on every proposed change) renders the overlay down to one file and scans that file. Nothing merges until the scans pass.

render-scan:
  image: registry.example.com/ci/kustomize-tools:5.4.3
  script:
    - kustomize build --enable-helm overlays/prod > out.yaml        # assemble the manifests
    - kubeconform -strict -summary out.yaml                         # schema validity
    - checkov -f out.yaml --framework kubernetes --quiet --compact  # security posture
    - trivy config out.yaml                                         # misconfiguration scan
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'            # gate the merge

Render first. Use a pinned, standalone kustomize binary so the version in CI matches what the controller runs, not whatever kubectl happens to bundle.

kustomize version
kustomize build --enable-helm overlays/prod > out.yaml
grep -c '^kind:' out.yaml

v5.4.3
24

Then validate the shape. kubeconform checks every resource against the Kubernetes schema, and -strict rejects unknown fields, so a typo like replcas or a misplaced key fails the build instead of the cluster.

kubeconform -strict -summary -kubernetes-version 1.30.0 out.yaml

Summary: 24 resources found in 1 file - Valid: 22, Invalid: 0, Errors: 0, Skipped: 2

The two skipped resources are custom resources kubeconform has no schema for, which is expected. Now check the security posture. checkov and trivy read the assembled manifests and flag the weak spots: a container that can run as root, a pod (the smallest thing Kubernetes schedules, one or more containers that share an address and a lifecycle) that can escalate its privileges, a filesystem left writable.

checkov -f out.yaml --framework kubernetes --quiet --compact

kubernetes scan results:
 
Passed checks: 96, Failed checks: 5, Skipped checks: 0
 
Check: CKV_K8S_23: "Minimize the admission of root containers"
	FAILED for resource: Deployment.shop.web
	File: /out.yaml:14-61
Check: CKV_K8S_20: "Containers should not run with allowPrivilegeEscalation"
	FAILED for resource: Deployment.shop.web
	File: /out.yaml:14-61

trivy config out.yaml

2026-07-17T10:22:05Z	INFO	[misconfig] Misconfiguration scanning is enabled
2026-07-17T10:22:06Z	INFO	Detected config files	num=1
 
out.yaml (kubernetes)
=====================
Tests: 32 (SUCCESSES: 27, FAILURES: 5, EXCEPTIONS: 0)
Failures: 5 (UNKNOWN: 0, LOW: 2, MEDIUM: 3, HIGH: 0, CRITICAL: 0)
 
AVD-KSV-0012 (MEDIUM): Container 'web' of Deployment 'web' should set 'securityContext.runAsNonRoot' to true
See https://avd.aquasec.com/misconfig/ksv012

Why The Rendered Output, Not The Diff

Here is the trap. A pull request shows you the diff of the overlay files, and that diff can be tiny while the manifests it produces change enormously. Watch. This is the whole change a reviewer sees:

git diff main -- overlays/prod/kustomization.yaml

diff --git a/overlays/prod/kustomization.yaml b/overlays/prod/kustomization.yaml
index 3f2a1c9..b7e4d02 100644
--- a/overlays/prod/kustomization.yaml
+++ b/overlays/prod/kustomization.yaml
@@ -3,3 +3,3 @@ kind: Kustomization
 resources:
-  - github.com/acme/base//workloads?ref=9c1e0b7
+  - github.com/acme/base//workloads?ref=main

One line. A remote base reference (a base configuration Kustomize fetches from another Git repository while it builds) moved from a commit fingerprint to the main branch. It looks like housekeeping. Now render both versions the way CI does and diff what the cluster would actually get.

kustomize build --enable-helm overlays/prod > after.yaml
git worktree add -q /tmp/base main
kustomize build --enable-helm /tmp/base/overlays/prod > before.yaml
diff before.yaml after.yaml

44,50c44
<       securityContext:
<         runAsNonRoot: true
<         allowPrivilegeEscalation: false
<         readOnlyRootFilesystem: true
<         capabilities:
<           drop:
<           - ALL
---
>       securityContext: {}
71c65
<         image: acme/web@sha256:9f2c1e0b4a...
---
>         image: acme/web:latest

The moved reference pulled a newer base that stripped the securityContext (the block that says this pod cannot become root, cannot escalate its privileges, and runs on a read-only filesystem) and swapped a pinned image digest (the content hash of one exact image, which changes if a single byte of the image changes) for the floating :latest tag. None of that shows up in the two-line overlay diff. It only shows up in the rendered output, which is the thing the cluster runs.

Pin Every Remote Reference

That attack works because a branch or a tag is a label, and labels can be moved. main gets new commits. A tag like v2 can be deleted and re-pushed to point at different code, and your repository never changes a byte. A commit SHA (the unique fingerprint of one exact snapshot of a repo, so different content always produces a different SHA) cannot be moved. Pin remote bases and Helm charts (packaged, templated Kubernetes apps you install with the Helm package manager) to immutable references.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - github.com/acme/platform-base//workloads?ref=9c1e0b7   # commit SHA: immutable
helmCharts:
  - name: ingress-nginx
    repo: https://kubernetes.github.io/ingress-nginx
    version: 4.11.3                                        # pinned chart version
    releaseName: ingress
    namespace: ingress

The commit SHA nails the base to exact content. The chart version pins the Helm dependency the same way, and the same rule applies to any container image: reference it by digest, not by a moving tag. Rendering an overlay that has helmCharts needs the --enable-helm flag on kustomize build, which your CI already passes. When upstream ships a new base, you bump the SHA in a commit, CI renders it, the scans run, and you see the change before the cluster does.

Give The Controller The Smallest Key

The controller holds cluster credentials and runs kustomize build inside the cluster. Two locks matter. First, RBAC (role-based access control, the rules that say which identity may touch which resources). Scope the controller to the namespaces and resource kinds it manages. Do not give it cluster-admin. If a malicious manifest ever slips past the gate, a scoped controller can damage one namespace; a cluster-admin controller can rewrite the whole cluster, including its own permissions.

Second, keep the build itself boring. Do not set --load-restrictor LoadRestrictionsNone, which lets a kustomization reach outside its own directory with ../ paths and read files it should never see, like a mounted service-account token or another team's secret. Keep the default, RootOnly. And keep exec plugins off (no --enable-exec, no alpha plugins in Argo CD), so a crafted overlay cannot run an arbitrary binary during the build.

Secrets never go in the repo as plaintext. Encrypt them with SOPS (Secrets OPerationS, which encrypts the values inside the YAML so only the cluster's key can decrypt them) or pull them at runtime with an external secrets operator (a controller that reads a secret from a vault and creates the Kubernetes Secret in the cluster). Either way, Git holds ciphertext or a pointer, never the raw password.

Check It Before Git Does

Before you trust a change, preview it against the live cluster. Flux renders the overlay locally and shows exactly what would move.

flux diff kustomization shop-prod --path ./overlays/prod

► GitRepository/flux-system/shop
✚ Deployment/shop/web created
► Kustomization/flux-system/shop-prod drifted
± Deployment/shop/api
  spec.template.spec.containers.0.image
-   acme/api:1.4.2
+   acme/api:1.4.3
► identified 2 resource(s)

Argo CD gives you argocd app diff for the same preview. Reading that output is the difference between finding a dropped guard in a pull request and finding it in an incident.

Run the exact CI scan commands on your own machine before you push: same kustomize version, same kubeconform, checkov, and trivy calls. The merge gate should confirm what you already checked, never be the first place you learn that your one-line change quietly turned off a security control.

Try this

Run kustomize version on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: review the build output, not the overlay diff. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related