What Flux is: the GitOps toolkit

Composable reconciler controllers.

A professional kitchen does not run on one giant machine that does everything. It has stations. One person pulls ingredients from the pantry. One chops. One works the grill. One plates the dish and calls out when the order is ready. Each station owns a single job, and work moves down the line. Flux is built the same way. Rather than one big application running your whole cluster, Flux is a handful of small programs, each with one responsibility, that you combine into a pipeline. The whole set has a name: the GitOps Toolkit.

GitOps (running your systems by keeping their desired state in a Git repository and having software continuously make reality match that state) needs two basic moves: read what you want, then make the cluster look like it. Flux takes those two moves, plus a few more, and hands each one to its own controller. A controller, in Kubernetes (the software that schedules and runs your containers across a fleet of machines), is a program that watches for one type of object and works to make the running system agree with what that object declares. Think of it as a thermostat. It reads the target, checks the room, and nudges reality toward the target, over and over. Flux's controllers each watch their own object types and mind their own business.

The controllers, one job each

The source-controller is the pantry runner. It watches source objects (a GitRepository, an OCIRepository stored as an Open Container Initiative artifact, which uses the same packaging as a container image, a HelmRepository, or a storage Bucket), fetches the desired state from that origin, checks it, packs it into a compressed archive it calls an artifact, and serves that archive to the rest of the cluster over HTTP (the ordinary web protocol). It never creates or deletes your workloads. It only fetches and stores. The kustomize-controller and the helm-controller are the cooks. The kustomize-controller takes an artifact and applies the plain Kubernetes YAML (a text format for configuration) inside it, stitching together any overlays (small patches layered on top of a shared base) first. The helm-controller does the same job for Helm charts (pre-packaged bundles of Kubernetes resources). The notification-controller is the expediter who calls out orders. It sends events outward to Slack or Microsoft Teams, and it takes webhooks inward (automatic HTTP callbacks a service fires when something happens), so a Git push can poke Flux to reconcile now, meaning re-check the source and re-apply, instead of waiting for the next interval. Two more controllers, the image-reflector-controller and the image-automation-controller, scan container registries for new image tags and write the chosen tag back into Git.

flux version

flux: v2.3.0
distribution: flux-v2.3.0
helm-controller: v1.0.1
kustomize-controller: v1.3.0
notification-controller: v1.3.0
source-controller: v1.3.0

A default install runs four of these: source, kustomize, helm, and notification. The two image controllers are opt-in, because plenty of teams do not automate image bumps. That is the whole point of the toolkit shape. You install the controllers you need and leave out the ones you do not. Every controller you add is a small, replaceable piece instead of a feature buried inside one big program. Each controller is an ordinary Deployment in the flux-system namespace, so you inspect them the way you inspect anything else.

kubectl get pods -n flux-system

NAME                                       READY   STATUS    RESTARTS   AGE
helm-controller-b6767d66-4jw8n             1/1     Running   0          6d
kustomize-controller-7c87d9f45b-2vhkq      1/1     Running   0          6d
notification-controller-5f7c9d8b6-lm4rt    1/1     Running   0          6d
source-controller-6b8d9c7f45-x9pqz         1/1     Running   0          6d

Everything is a resource

Every object Flux acts on is a Custom Resource, backed by a CRD (Custom Resource Definition, which is how you teach Kubernetes a brand-new type of object so the API server stores and serves it like anything built in). There is no separate database and no default web console. A GitRepository is a Kubernetes resource. So is a Kustomization, a HelmRelease, an Alert. That single design choice pays off everywhere. The tools you already run keep working: kubectl to read and edit, RBAC (Role-Based Access Control, the rules for who may do what) to lock things down, admission controllers (gatekeepers that inspect or reject a resource before the API server saves it) to enforce policy, and Flux itself to manage Flux. Ask the API server what Flux added, and it answers plainly.

kubectl api-resources --api-group=source.toolkit.fluxcd.io

NAME               SHORTNAMES   APIVERSION                         NAMESPACED   KIND
buckets                         source.toolkit.fluxcd.io/v1beta2   true         Bucket
gitrepositories    gitrepo      source.toolkit.fluxcd.io/v1        true         GitRepository
helmcharts         hc           source.toolkit.fluxcd.io/v1        true         HelmChart
helmrepositories   helmrepo     source.toolkit.fluxcd.io/v1        true         HelmRepository
ocirepositories    ocirepo      source.toolkit.fluxcd.io/v1beta2   true         OCIRepository

Two objects do most of the day-to-day work. A source says where the truth lives. A Kustomization says what to do with it. Here is a minimal pair that deploys the podinfo demo app.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1m
  url: https://github.com/stefanprodan/podinfo
  ref:
    branch: master

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: podinfo
  path: ./kustomize
  prune: true
  targetNamespace: default

Read them top to bottom and the handoff is clear. The GitRepository tells the source-controller to clone that URL, track the master branch, and re-check every minute. The Kustomization names that source, points at a folder inside the fetched artifact, and applies what it finds. prune: true means when you delete a file in Git, Flux deletes the matching resource in the cluster on the next pass. The two controllers never call each other directly. The source-controller publishes an artifact, and the kustomize-controller consumes it. Swap the source from Git to an OCI registry and the Kustomization does not change at all.

The loop you can watch

The loop is boring on purpose. Fetch the desired state, apply it, wait for the interval, repeat. If someone edits a Deployment by hand, the next pass overwrites their change back to what Git says, which is how GitOps corrects drift. You do not have to guess whether it is working. The flux command-line tool (its CLI, the program you type commands into) reads the same resources the controllers do and shows you their status.

flux get sources git

NAME     REVISION                                              SUSPENDED  READY  MESSAGE
podinfo  master@sha1:1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e  False      True   stored artifact for revision 'master@sha1:1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e'

flux get kustomizations

NAME     REVISION                                              SUSPENDED  READY  MESSAGE
podinfo  master@sha1:1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e  False      True   Applied revision: master@sha1:1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e

READY tells you the last pass succeeded. REVISION is the exact Git commit now live in the cluster, so you can match what is running to a line in your history. To see what a Kustomization actually placed on the cluster, ask for its tree. To stop waiting and reconcile right now, tell Flux to reconcile the Kustomization together with its source.

flux tree kustomization podinfo

Kustomization/flux-system/podinfo
├── Deployment/default/podinfo
├── HorizontalPodAutoscaler/default/podinfo
└── Service/default/podinfo

flux reconcile kustomization podinfo --with-source

► annotating GitRepository podinfo in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision master@sha1:1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e
► annotating Kustomization podinfo in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision master@sha1:1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e

Where the power actually lives

Modular or not, the kustomize-controller and helm-controller create and delete real resources in your cluster, which means they run with real permissions. A ClusterRoleBinding works like a master key handed to a member of staff: it decides which doors that identity may open. By default, Flux is handed the master key that opens every door. The bootstrap process binds both controllers' service accounts (the identities pods use to talk to the API server) to cluster-admin through a single ClusterRoleBinding. Read it yourself.

kubectl get clusterrolebinding cluster-reconciler-flux-system -o yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  creationTimestamp: "2024-06-01T09:14:22Z"
  labels:
    app.kubernetes.io/instance: flux-system
    app.kubernetes.io/part-of: flux
    app.kubernetes.io/version: v2.3.0
  name: cluster-reconciler-flux-system
  resourceVersion: "1487"
  uid: 6f3a2b1c-8d4e-4a9f-b0c1-2d3e4f5a6b7c
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kustomize-controller
  namespace: flux-system
- kind: ServiceAccount
  name: helm-controller
  namespace: flux-system

The other soft spot is trust in the source. The source-controller fetches whatever the origin serves and, out of the box, does not check who produced it. If an attacker rewrites a commit or poisons the registry Flux pulls from, the cluster follows on the next pass. Close that gap the way a courier trusts a wax seal on an envelope: refuse anything whose seal does not match. You set that seal with spec.verify. On a GitRepository, Flux checks the commit's PGP signature (Pretty Good Privacy, a long-standing standard for signing and encrypting data) against public keys you supply. On an OCIRepository, it checks a Cosign or Notation signature (Cosign is the Sigstore project's tool for signing container images and artifacts) before it stores the artifact. To shrink the blast radius further, set spec.serviceAccountName on a Kustomization and give that account only the RBAC a given tenant needs, so a bad manifest cannot reach past its own namespace.

For detection, three signals earn their place on a dashboard. flux logs surfaces failed applies and permission errors as they happen. The notification-controller can ship every reconcile event to a channel you watch, so an unexpected change is loud instead of silent. And an alert on any edit to a GitRepository's spec.url or spec.ref catches the quiet attack where someone repoints Flux at a repository you do not control. Wire those three up before you spread Flux across clusters, not after.

Try this

Run flux version on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: flux reconciles as cluster-admin by default. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related