Hardening Flux & the supply chain

Signed commits, verified images.

A GitOps pipeline (running your infrastructure by keeping its desired state in Git and having software make the cluster match it, over and over) is like a conveyor belt in a factory. Whatever Git (the version-control system that stores your code and its whole history) or a container registry (the server that stores and serves your container images) drops onto the belt rides straight into your Kubernetes cluster (the system that runs your containers across a pool of machines) and becomes live, running state. Flux (the GitOps tool that pulls that desired state and applies it) is the machine at the end of the belt. Out of the box it trusts the cargo completely: any commit on the branch it watches, any tag the registry serves, no questions asked. Hardening Flux means hiring an inspector who checks the identity stamp on every package before it moves onto the belt. Flux ships two inspectors, and both start switched off.

Here is the attack you are defending against. Someone steals a developer's push credentials, or takes over a bot account, and pushes a commit straight to the branch Flux watches. No review, no second pair of eyes. Within a minute or two Flux reconciles it, and their change is running in your cluster: a new image, an extra container, a firewall rule that quietly opens an outbound path (egress, traffic leaving your cluster). Nothing on the belt stopped it because nothing on the belt was looking. The two checks in this lesson are how you make the belt look.

Verify signed commits at the source

A signature is like a wax seal on an envelope. Anyone can read whose seal it is, but only the person holding the matching stamp could have pressed it. Git commits can carry that seal using PGP (Pretty Good Privacy, a long-standing system for signing with a public and private key pair). You set spec.verify on the GitRepository object, hand Flux a keyring (a file of trusted public keys), and Flux checks every commit's seal before it reconciles the change into the cluster. If the seal is missing, or was made with a key you never loaded, the source goes NotReady and nothing applies. The bad commit sits there, rejected. The keyring is an allow-list, not a wildcard: the only authors you trust are the ones whose public keys you put in the secret, and no one else.

This is a different guard from branch protection. Code review catches a bad change that travels through a pull request (the review step a change goes through before it merges). Signature verification catches the change that never went through one: the commit pushed straight to the branch with stolen credentials. You want both. Load only public keys here, never a private key, and keep the list short so a single leaked key is a small blast radius.

# export ONLY the public key of an author you trust
gpg --export --armor [email protected] > author.asc
 
kubectl create secret generic pgp-public-keys \
  --from-file=author.asc --namespace=flux-system

secret/pgp-public-keys created

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo
  ref:
    branch: master
  verify:
    mode: HEAD                 # check the signature on the tip commit
    secretRef:
      name: pgp-public-keys    # only these public keys are trusted

Apply it, then read the status. Flux records the result in a condition named SourceVerified, so you can see the verdict without guessing. When the commit is signed by a trusted key, the condition says so plainly. When it is not, the source is not Ready, and the reason tells you exactly why.

kubectl -n flux-system get gitrepository podinfo \
  -o jsonpath='{.status.conditions[?(@.type=="SourceVerified")].message}'

verified signature of commit master@sha1:6d4a3ba9c1f0

# a commit signed with a key that is not in your keyring lands on master
kubectl -n flux-system get gitrepository podinfo

NAME      URL                                        AGE   READY   STATUS
podinfo   https://github.com/stefanprodan/podinfo    2m    False   unable to verify Git commit: openpgp: signature made by unknown entity

Verify OCI artifacts with Cosign

When your desired state or your Helm charts (packaged, versioned bundles of Kubernetes manifests) ship as OCI artifacts (Open Container Initiative, the standard packaging format for container images and related files) instead of plain Git files, the seal changes shape. Now you check a Cosign signature. Cosign (a signing tool from the Sigstore project, which builds open tools for signing software) usually signs without any long-lived key at all, which sounds strange until you see how it works. This is keyless signing: at build time your CI (continuous integration, the automation that builds and publishes your code) proves who it is through OIDC (OpenID Connect, the standard way one service proves its identity to another), and Fulcio (Sigstore's certificate authority) issues a short-lived certificate stamped with that identity, for example 'built by the release workflow in the fluxcd/podinfo repository'. The signature and that certificate are stored right next to the image in the registry.

Set spec.verify with provider: cosign on the OCIRepository and Flux refuses any artifact whose Cosign signature does not check out. But a valid signature only proves that someone signed it, not that the right someone did. With keyless signing, anyone with a GitHub account can produce a perfectly valid signature from their own workflow. So you pin the identity: matchOIDCIdentity makes Flux compare the certificate's issuer and subject against regular expressions you set. No identity match, no trust. Leave it out and every validly-signed artifact on earth passes your gate.

apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: OCIRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 5m
  url: oci://ghcr.io/stefanprodan/manifests/podinfo
  ref:
    tag: latest
  verify:
    provider: cosign
    matchOIDCIdentity:                 # prove WHO signed, not only THAT it is signed
      - issuer: "^https://token.actions.githubusercontent.com$"
        subject: "^https://github.com/stefanprodan/podinfo.*$"

kubectl -n flux-system get ocirepository podinfo \
  -o jsonpath='{.status.conditions[?(@.type=="SourceVerified")].message}'

verified signature of revision latest@sha256:3b6cbcd1f0e2a7d9

One honest gap, because it surprises people and gets exploited. Flux's image-automation feature (two controllers: image-reflector-controller, which scans a registry for new tags, and image-automation-controller, which writes the chosen tag back into Git) does no signature checking at all. It picks tags by name and by rules like 'highest semver' (semantic version, the 1.4.2-style number that orders releases), never by trust. So an attacker who can push a tag that sorts higher than your current one can get that tag written into Git automatically. A signed OCIRepository does not save you here, because image automation is a separate path. The thing that actually stops an unsigned or untrusted image from running is an admission policy inside the cluster, checked at deploy time, which we get to at the end.

Harden the inspectors themselves

An inspector you can bribe is worse than no inspector, because it hands you false confidence. If an attacker swaps the Flux controller image for a tampered one, every check above becomes theater: a tampered controller can wave anything through and still report that every signature was fine. So the controllers that do the verifying have to be verified and locked down themselves. The good news is Flux ships them hardened already. Each controller runs as a non-root user, with a read-only root filesystem, with every Linux capability dropped, and with seccomp (secure computing mode, a kernel feature that limits which system calls a process is allowed to make) set to the runtime default. Confirm it rather than assume it.

kubectl -n flux-system get deploy source-controller \
  -o json | jq '.spec.template.spec.containers[0].securityContext'

{
  "allowPrivilegeEscalation": false,
  "capabilities": {
    "drop": [
      "ALL"
    ]
  },
  "readOnlyRootFilesystem": true,
  "runAsNonRoot": true,
  "seccompProfile": {
    "type": "RuntimeDefault"
  }
}

The other half is the network. A controller only needs to reach a few places: your Git host, your container registries, and the Kubernetes API server (the control-plane endpoint the cluster is driven through). Everything else it tries to reach is a signal worth investigating. A NetworkPolicy (a Kubernetes firewall rule that filters pod traffic) in the flux-system namespace that denies all egress except DNS and port 443, plus the API server port, closes off the odd ports a compromised controller might use to open a reverse shell or beacon out. Port-only rules still let traffic reach any host on 443, so once this is in place you tighten it further by naming the specific Git and registry address ranges the controllers are allowed to talk to.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: flux-restrict-egress
  namespace: flux-system
spec:
  podSelector: {}                 # every pod in flux-system
  policyTypes: [Egress]
  egress:
    - ports:                      # DNS lookups
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    - ports:                      # HTTPS to Git and registries, plus the API server
        - { protocol: TCP, port: 443 }
        - { protocol: TCP, port: 6443 }

Now verify Flux's own supply chain the same way you verify your workloads. Flux's release manifests and container images are Cosign-signed with keyless GitHub OIDC, so check a controller image before you install or upgrade. Run flux check --pre before you bootstrap (the one-time command that installs Flux and points it at your Git repo), and flux check afterward, which confirms every controller is running and prints the image version of each one, so you can match them against the release you meant to install. Those controllers depend on a set of CRDs (Custom Resource Definitions, the way you teach Kubernetes new object types like GitRepository and OCIRepository) that the bootstrap installed alongside them.

cosign verify ghcr.io/fluxcd/source-controller:v1.3.0 \
  --certificate-oidc-issuer=https://token.actions.githubusercontent.com \
  --certificate-identity-regexp='^https://github.com/fluxcd/.*$'

Verification for ghcr.io/fluxcd/source-controller:v1.3.0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
 
[{"critical":{"identity":{"docker-reference":"ghcr.io/fluxcd/source-controller"},"image":{"docker-manifest-digest":"sha256:1c..."},"type":"cosign container image signature"},"optional":{"Issuer":"https://token.actions.githubusercontent.com","Subject":"https://github.com/fluxcd/source-controller/.github/workflows/release.yaml@refs/tags/v1.3.0"}}]

flux check

► checking prerequisites
✔ Kubernetes 1.29.4 >=1.28.0-0
► checking version in cluster
✔ distribution: flux-v2.3.0
✔ bootstrapped: true
► checking controllers
✔ source-controller: deployment ready
► ghcr.io/fluxcd/source-controller:v1.3.0
✔ kustomize-controller: deployment ready
► ghcr.io/fluxcd/kustomize-controller:v1.3.0
✔ helm-controller: deployment ready
► ghcr.io/fluxcd/helm-controller:v1.0.1
✔ notification-controller: deployment ready
► ghcr.io/fluxcd/notification-controller:v1.3.0
✔ all checks passed

So put a last gate in the cluster itself. An admission policy (a rule the API server enforces the instant something tries to run) from Kyverno, Gatekeeper, or the Sigstore policy-controller checks the actual container images your pods are about to start, and can require the same signed identity you pinned on the source. Source verification stops a bad change from being stored. The admission policy stops a bad image from running, including the images that image automation pulled in without a check. Set both, and a valid-but-untrusted artifact has nowhere left to pass.

Try this

Run gpg --export --armor [email protected] > author.asc on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a signature that exists is not one you trust. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related