Kustomization: reconciling

Apply and prune from a source.

A careful stockroom clerk works from one master packing list. On a schedule they walk the aisles. Anything on the list that is missing gets placed. Anything sitting in the wrong spot gets fixed. Anything on the shelf that is no longer on the list gets pulled and carried out. The shelf is never trusted on its own. The list wins, every time. A Flux Kustomization is that clerk for your cluster.

A Flux Kustomization is a Kubernetes custom resource (a new object type that Flux installs into your cluster) whose whole job is reconciling. It takes a source (a snapshot of a Git repository, an OCI image, or an object-storage bucket, where OCI means Open Container Initiative, the standard packaging format for container images) plus a path inside that source, turns the files it finds there into live Kubernetes objects, and keeps the cluster matching them on a fixed schedule. One naming trap: this is not Kustomize, the command-line tool, even though the reconciler does run Kustomize under the hood when it finds a kustomization.yaml at that path. GitOps (running your operations by treating Git as the single source of truth for what should be running) stands entirely on this loop.

The reconcile loop

Reconciliation is a loop, not a one-shot deploy. On each tick, Flux fetches the current source artifact, a tarball the source-controller built from your latest Git commit and tagged with a revision (the branch name plus the commit hash). It builds the manifests: run Kustomize on the path, or read plain YAML (YAML Ain't Markup Language, the text format Kubernetes objects are written in) when there is no kustomization.yaml. It applies any post-build variable substitutions. Then it does a server-side apply to create or correct every object. That is a Kubernetes feature where the API server, not your client, merges the change and records which controller owns each field. It prunes objects that used to be in its inventory but are gone from Git. If you asked it to wait, it blocks until those workloads report healthy. Finally it writes status and schedules the next tick. Nothing about the run depends on what a human did between ticks.

Defining one

Here is a Kustomization for a payments app. It lives in your Git repo next to everything else, so the reconciler is itself version-controlled and reviewed like any other change.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: payments
  namespace: flux-system
spec:
  interval: 5m           # reconcile every 5 minutes
  retryInterval: 1m      # back off to 1m after a failed apply
  timeout: 3m            # give up a run after 3 minutes
  sourceRef:
    kind: GitRepository
    name: payments
  path: ./k8s/overlays/prod
  prune: true            # delete resources removed from Git
  wait: true             # not Ready until workloads are healthy
  targetNamespace: payments
  serviceAccountName: payments-reconciler  # apply with limited RBAC

Read the fields as the clerk's standing orders. interval is how often they walk the aisles. Between walks, Flux still picks up a new commit quickly, because the kustomize-controller watches the source and reruns the moment the source-controller publishes a new revision. sourceRef and path say which list and which shelf. prune: true is the authority to remove things. wait: true means the clerk does not report the job done until the placed items are actually standing up (pods Ready, not merely created), which is what lets you chain one Kustomization behind another later. serviceAccountName pins the whole apply to a limited identity, which we come back to at the end.

Watching it reconcile

Start with the summary view. This is the single most useful command for an operator on call.

flux get kustomizations

NAME         REVISION                                          SUSPENDED  READY  MESSAGE
flux-system  main@sha1:2a5e3c9f1b8d4e6a7c0f2b1d3e5a7c9f0b2d4e6a  False      True   Applied revision: main@sha1:2a5e3c9f1b8d4e6a7c0f2b1d3e5a7c9f0b2d4e6a
payments     main@sha1:2a5e3c9f1b8d4e6a7c0f2b1d3e5a7c9f0b2d4e6a  False      True   Applied revision: main@sha1:2a5e3c9f1b8d4e6a7c0f2b1d3e5a7c9f0b2d4e6a

Three columns carry the story. REVISION is the exact commit now live, so you can match it back to a pull request. READY True means the last apply landed and, because wait is on, the workloads are healthy. SUSPENDED False means the clerk is still walking the aisles; a True there means someone paused enforcement, which you should care about a lot. You do not have to wait for the timer. Force a run for immediate feedback, pulling the newest commit first:

flux reconcile kustomization payments --with-source

► annotating GitRepository payments in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main@sha1:2a5e3c9f1b8d4e6a7c0f2b1d3e5a7c9f0b2d4e6a
► annotating Kustomization payments in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main@sha1:2a5e3c9f1b8d4e6a7c0f2b1d3e5a7c9f0b2d4e6a

To see exactly what this Kustomization owns, ask for its tree. Every object under it was placed by this reconciler, and every object under it is a candidate for pruning.

flux tree kustomization payments

Kustomization/flux-system/payments
├── Deployment/payments/api
├── Service/payments/api
├── ConfigMap/payments/api-config
└── HorizontalPodAutoscaler/payments/api

Prune and the inventory

The clerk does not guess what to remove. Flux keeps a written inventory in the Kustomization's status: a flat list of every object it applied last time. Look at it directly.

kubectl -n flux-system get kustomization payments -o yaml

status:
  conditions:
  - type: Ready
    status: "True"
    reason: ReconciliationSucceeded
    message: 'Applied revision: main@sha1:2a5e3c9f1b8d4e6a7c0f2b1d3e5a7c9f0b2d4e6a'
  lastAppliedRevision: main@sha1:2a5e3c9f1b8d4e6a7c0f2b1d3e5a7c9f0b2d4e6a
  inventory:
    entries:
    - id: payments_api_apps_Deployment
      v: v1
    - id: payments_api__Service
      v: v1
    - id: payments_api-config__ConfigMap
      v: v1
    - id: payments_api_autoscaling_HorizontalPodAutoscaler
      v: v2

Each id is namespace_name_group_kind. A core object like the Service has an empty group, which is why you see the double underscore in payments_api__Service. Pruning is a set difference: build the new manifests, compare them against this stored inventory, and delete anything that was in the old list but is missing from the new one. That is the exact promise from the intro. Delete a file from Git, commit it, and on the next tick the matching live object is gone. Deletion from the repository is deletion from the cluster.

You exclude a specific object from garbage collection with an annotation. The clerk still manages the object, but will never carry it out, even if its file disappears from Git.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-data
  namespace: payments
  annotations:
    kustomize.toolkit.fluxcd.io/prune: disabled
spec:
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 20Gi

Drift, and why attackers hate reconciliation

Server-side apply creates objects, and it also owns fields. Any field the Kustomization sets, it keeps set, because the API server records the reconciler as the owner of that field. So a change made by hand to a managed object is drift, and the next reconcile writes it straight back to whatever Git says. You get this as a security property for free. Suppose an attacker with cluster access swaps your image for a backdoored one:

kubectl -n payments set image deployment/api api=ghcr.io/attacker/backdoor:latest

deployment.apps/api image updated

That sticks for at most one interval. On the next walk, or the moment you force one, Flux sees that a field it owns no longer matches the source, and it corrects the object back.

flux reconcile kustomization payments
kubectl -n payments get deploy api -o jsonpath='{.spec.template.spec.containers[0].image}'; echo

► annotating Kustomization payments in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main@sha1:2a5e3c9f1b8d4e6a7c0f2b1d3e5a7c9f0b2d4e6a
registry.example.com/payments/api:v1.4.0

The backdoor image is gone and the trusted one is back, without anyone paging you. For a defender this cuts two ways. Live-editing production is a losing move for an attacker and a self-healing safety net for you. It also moves the real target. Whatever gets merged, Flux applies faithfully and enforces relentlessly, so the prize becomes Git itself and the source behind it. Branch protection, signed commits, and a reviewed pull request stopped being code hygiene the day you turned on GitOps. They are runtime security controls now.

There is one obvious way to defeat all of this, and an attacker knows it. flux suspend kustomization payments stops the clerk cold by setting spec.suspend on the object. A suspended Kustomization no longer enforces Git, so any manual change persists for as long as it stays suspended. An adversary who wants a backdoor to survive will suspend first, then tamper. Treat the SUSPENDED column flipping to True as a real signal: alert on spec.suspend being set, and periodically confirm that your critical Kustomizations still read Ready: True and Suspended: False.

Least privilege for the clerk

Think of the difference between a master key that opens every room in the building and a key cut for one office. By default the kustomize-controller applies with its own identity, which on a standard Flux install carries cluster-admin (the master key), so a compromised repo path could touch anything in the cluster. serviceAccountName swaps in the office key. Flux impersonates the ServiceAccount (a non-human identity in the cluster) you name, looked up in the Kustomization's own namespace, flux-system here, so the apply can only do what that identity's RBAC (Role-Based Access Control, Kubernetes' permission system) allows. You set this up once, ahead of time: create the payments namespace, create the payments-reconciler ServiceAccount in flux-system, and bind it with a Role and RoleBinding scoped to the payments namespace. Now a bad merge into that path cannot create a cluster-admin binding or reach into another team's namespace. The reconcile loop stays powerful. Its blast radius does not.

Wire two alerts before you call any of this production-ready. Fire one when a critical Kustomization reports Ready: False, which catches a broken deploy or a source that stopped fetching. Fire the other when spec.suspend turns true on a Kustomization nobody paused on purpose, which catches an attacker reaching for the off switch. Those two signals cover the loop working and the loop being switched off, and the second one is the alert most clusters are missing.

Try this

Run flux get kustomizations on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: prune will happily delete your database. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related