Image automation

Auto-bump images back to Git.

A new container image build (a fresh, packaged copy of your application ready to run) is like a shipment arriving at a loading dock. You could wire the dock straight through to the kitchen, so every crate that lands drops into the pot the second it shows up. Fast. Also terrifying, because nobody wrote down what changed or when. Flux image automation takes the other route: a new image becomes a line in your Git history first, and only then does it become a running container. Git is the ledger, and every image bump is an entry you can read, review, and undo.

Three moving parts make this work. A scanner watches the registry (the server that stores your container images) and writes down every tag it finds. A rule picks the one tag you actually want. A writer edits that tag into your YAML (a plain-text format for configuration files) in Git and commits it. Because the change lands as a commit, everything you already trust about GitOps (managing your systems through Git as the single source of truth) still holds: you can see who changed what, diff it (compare two versions line by line), require a review, or roll it back with git revert. This lesson wires all three together, and, more to the point, shows where the sharp edges are for anyone running it in production.

The two controllers, and who holds the write key

Flux keeps image automation out of the default install, so you opt in with two extra controllers (background programs that watch objects and steer them toward the state you declared, a process Flux calls reconciling). The image-reflector-controller is the scanner. It drives two objects: an ImageRepository, which points at one image in a registry and lists its tags, and an ImagePolicy, which picks a tag by a rule. The image-automation-controller is the writer. It drives an ImageUpdateAutomation, which edits Git and pushes. A standard Flux bootstrap (its one-command install that wires the cluster to your repo) does not start either controller, so you add them with two flags at bootstrap time.

flux bootstrap github \
  --owner=acme --repository=fleet \
  --branch=main --path=./clusters/prod \
  --components-extra=image-reflector-controller,image-automation-controller \
  --read-write-key=true

The --components-extra flag turns the two controllers on. The --read-write-key=true flag is the one to slow down on. The writer has to push commits to Git, which means it needs a credential (a login secret) that can write. A default Flux bootstrap hands out a deploy key (an SSH, or Secure Shell, key scoped to a single repository) that is read-only, so the automation controller fails every push against it. This flag makes that key writable. The key then lives in the cluster as a Kubernetes Secret (the cluster's built-in object for holding sensitive values) and can rewrite your deploy branch, so treat it like any other production credential: one key per repository, and branch protection (a rule that blocks unreviewed pushes) on the branches that matter.

Get that flag wrong and the failure hides in plain sight. If you bootstrapped without a writable key, flux get image update reports READY False with a message like unable to push: ERROR: The key you are authenticating with has been marked as read only. The catch is that the scan and the tag selection still succeed, so the ImagePolicy looks perfectly healthy while no commit ever lands in Git. Check the automation object, not only the policy.

kubectl -n flux-system get deploy | grep image

image-automation-controller   1/1     1            1           12d
image-reflector-controller    1/1     1            1           12d

Tell Flux which tag to want

The scanner needs two things: where to look, and how to choose. The ImageRepository is the where. It names one image, sets how often to scan it, and points at a pull secret (a login the cluster uses to read a private registry) when the registry is not public. The ImagePolicy is the how. Its policy block is a rule applied to the tags the scanner found. The common rule is semver (semantic versioning, the MAJOR.MINOR.PATCH numbering scheme, like 1.4.7), paired with a range that bounds what you will accept.

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
  name: payments
  namespace: flux-system
spec:
  image: registry.internal/payments
  interval: 5m
  secretRef:
    name: regcred
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
  name: payments
  namespace: flux-system
spec:
  imageRepositoryRef:
    name: payments
  policy:
    semver:
      range: ">=1.4.0 <2.0.0"

Read that range out loud: accept any 1.x version from 1.4.0 up, but never cross into 2.0.0. That keeps automation off a major version, where breaking changes live. The other rule types are numerical and alphabetical, for tags that are not clean semver. Continuous integration (the automated build pipeline, CI for short) often tags images with something messy like main-2f8a1c9-1721460872, a branch name, a commit hash, and a build timestamp. You strip out the sortable part with filterTags, a regex (regular expression, a text pattern for matching text), then order what is left.

spec:
  imageRepositoryRef:
    name: payments
  filterTags:
    pattern: '^main-[a-fA-F0-9]+-(?P<ts>[0-9]+)$'
    extract: '$ts'
  policy:
    numerical:
      order: asc

Here is the security crux. Sit with it for a second. Semver picks the highest matching tag. Numerical picks the largest number. Whatever the rule, the winner is chosen by whoever can publish a tag that sorts to the top. Your range narrows the field, but inside that field, the registry decides your running version. You can see what it has chosen at any moment from the command line.

flux get image policy payments

NAME      LATEST IMAGE                        READY  MESSAGE
payments  registry.internal/payments:1.4.7    True   Latest image tag for 'registry.internal/payments' resolved to 1.4.7

Mark the exact line to rewrite

The scanner and the rule produce a chosen tag, but Flux still has to know which line in which file to change. You tell it with a marker, a comment stuck on the image line, like a sticky note on one field of a form that says update me from the payments policy. The writer only touches lines that carry a marker. It never adds new fields on its own.

    spec:
      containers:
      - name: payments
        image: registry.internal/payments:1.4.0 # {"$imagepolicy": "flux-system:payments"}

The marker is JSON (JavaScript Object Notation, a structured text format) tucked inside the comment. The value flux-system:payments is the namespace and name of the ImagePolicy to read from. Bare like that, it rewrites the whole reference, repository and tag together. Add :tag on the end (flux-system:payments:tag) to rewrite only the tag and leave the repository alone, or :name to rewrite only the repository. That split matters when your manifest keeps the image name in one field and the tag in another, which is common with Helm (the package manager for Kubernetes) values files.

This is the second silent failure to watch for. A typo in the JSON, the wrong namespace:name, or a missing marker raises no error at all. Flux skips the line and writes nothing. The ImagePolicy still shows a healthy LATEST IMAGE, so it looks like automation is running while Git never moves. After any change to a marker, confirm that a commit actually appeared before you trust it.

Commit it back to Git

The ImageUpdateAutomation is the writer's job description. It spells out which Git source to edit, which branch to read from, where to push, and how to phrase the commit. The update.strategy: Setters value tells it to use the markers you placed. The choice that matters most is push.branch. Point it at a separate branch and a human merges the change through a pull request (a proposed change teammates review before it lands on the main branch). Leave push off and Flux commits straight onto the branch it deploys from, no human in the loop.

apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
  name: fleet
  namespace: flux-system
spec:
  interval: 30m
  sourceRef:
    kind: GitRepository
    name: flux-system
  git:
    checkout:
      ref:
        branch: main
    commit:
      author:
        name: fluxbot
        email: [email protected]
      messageTemplate: |
        Update {{range .Updated.Images}}{{println .}}{{end}}
    push:
      branch: image-updates
  update:
    path: ./apps
    strategy: Setters

Now the payoff. Every bump is a real commit, authored by fluxbot, naming the exact image and tag. That is your audit trail, and it is the whole reason to route image updates through Git instead of letting a controller mutate the running Deployment (the Kubernetes object that keeps your pods running) directly.

git -C fleet log --oneline -4 origin/image-updates

a1b9f3c Update registry.internal/payments:1.4.7
4e7c210 Update registry.internal/payments:1.4.6
7c33b90 Update registry.internal/payments:1.4.5
9d2f8a1 Add payments deployment

What an attacker gets, and what you see

Run the poisoning scenario in your head. An attacker steals your CI registry credentials, or slips a push into the registry through a compromised build. They publish payments:1.9.9. The scanner lists it on its next pass. The policy, range >=1.4.0 <2.0.0, selects it as the highest match. The writer commits image: registry.internal/payments:1.9.9. The kustomize-controller (the Flux component that applies your manifests to the cluster) rolls it out. If your automation pushes straight to the deploy branch, that malicious image is live with nobody in the loop. The tight range did its job, and the attacker stayed inside it anyway.

So what does the defender actually see, and where do you put the gate? You see a commit authored by fluxbot with the exact tag and timestamp, which is a far better place to start than a silently mutated pod. From there, three controls stack. First, keep ranges narrow so a bad tag has less room to move, knowing this bounds the blast radius (how far one bad tag can spread) rather than stopping an in-range poison. Second, push to a reviewed branch with branch protection, so a pull request review and required approvals sit between the commit and production. Third, verify signatures. Flux can check Git commit signatures at the source, and you bolt on an admission controller (a gatekeeper that inspects resources before they run, such as Kyverno or OPA, short for Open Policy Agent, Gatekeeper) plus cosign or Notation (tools that cryptographically sign container images) to refuse any unsigned image at deploy time. Image automation does not verify signatures on its own. That gate is separate, and you have to build it.

Verify it actually worked

Do not wait for the interval when you want an answer now. Force a scan, force a run, then check the three places a change has to show up: the resolved tag, the commit on your update branch, and, once that branch has merged into the branch Flux deploys, the live pod.

flux reconcile image repository payments
flux reconcile image update fleet
flux get image update fleet

► annotating ImageRepository payments in flux-system namespace
✔ ImageRepository annotated
◎ waiting for ImageRepository reconciliation
✔ ImageRepository reconciliation completed
✔ scan completed, found 37 tags
► annotating ImageUpdateAutomation fleet in flux-system namespace
✔ ImageUpdateAutomation annotated
◎ waiting for ImageUpdateAutomation reconciliation
✔ ImageUpdateAutomation reconciliation completed
NAME   LAST RUN               READY  MESSAGE
fleet  2026-07-20T09:15:04Z   True   committed and pushed commit 'a1b9f3c' to branch 'image-updates'

kubectl -n payments get deploy payments \
  -o jsonpath='{.spec.template.spec.containers[0].image}'

registry.internal/payments:1.4.7

If flux get image policy shows a LATEST IMAGE that never reaches the running pod, the break is in one of two places. Either the marker is wrong, so no commit was ever written, or the commit is parked on your update branch waiting for the merge into the branch Flux deploys from. Run flux logs --kind=ImageUpdateAutomation --level=error and it will point at which one, usually a push permission error or a no changes made line that leads straight back to the marker.

Try this

Run kubectl -n flux-system get deploy | grep image on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a wide policy plus a direct push makes the registry your deploy button. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related