Dependencies & health

Order and readiness gates.

Pouring a concrete foundation and having that foundation pass inspection are two different events. One says the work happened. The other says the work is sound enough to build on. A crew that hangs framing the second the concrete is wet, before it cures and before the inspector signs off, is buying itself a collapse.

Flux keeps a Kubernetes cluster (the system that schedules and runs your containers across many machines) matching what is written in Git. That is the GitOps idea: your repository holds the desired state, and a controller (a small program running inside the cluster that watches and acts on its own) drags the running cluster toward it. For ordering and readiness, Flux hands you two separate controls, and the whole game is remembering they are separate. One sets the order: which piece goes first. The other sets the gate: how Flux decides the first piece is done and not merely poured. Wire both correctly and your database is up and answering before the API (the service other programs call over the network) that depends on it starts. Wire them wrong and Flux cheerfully releases a stampede of workloads that crash-loop against services, secrets, and resource types that do not exist yet.

Order With dependsOn

A dependency in Flux is a note on the work order that reads: do not start this until that one is signed off. The field is dependsOn, and it is a list of references to other Flux objects of the same kind. A Kustomization (Flux's object for applying a folder of Kubernetes manifests, where a manifest is the YAML text file that spells out what you want running) can depend on other Kustomizations. A HelmRelease (Flux's object for installing a Helm chart, where Helm is the package manager for Kubernetes) can depend on other HelmReleases. You cannot cross the streams: a HelmRelease cannot list a Kustomization in its dependsOn, and the reverse fails too. When you genuinely need that edge, bridge it with an intermediate object of the right kind. Flux refuses to reconcile the dependent (reconcile is Flux's word for dragging the cluster into line with Git) until every dependency reports Ready: True.

Two things it does not do. It does not create the dependency for you; the object you point at has to exist on its own. And it takes no part in teardown, so when you prune (Flux's word for deleting objects that left Git), the order is not reversed for you. References default to the dependent's own namespace (a namespace is a folder-like partition inside the cluster), so for a cross-namespace edge you set namespace explicitly. That case is everywhere: infrastructure controllers often live in one namespace while the apps that need them live in another.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  dependsOn:
    - name: infra-controllers      # same namespace (flux-system) is assumed
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./apps
  prune: true

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: backend
  namespace: apps
spec:
  interval: 5m
  dependsOn:
    - name: redis
      namespace: redis           # cross-namespace edge: set it explicitly
  chart:
    spec:
      chart: backend
      sourceRef:
        kind: HelmRepository
        name: my-charts
        namespace: flux-system

flux get helmreleases -A

NAMESPACE  NAME     REVISION  SUSPENDED  READY  MESSAGE
apps       backend            False      False  dependency 'redis/redis' is not ready
redis      redis    17.15.6   False      True   Helm install succeeded for release redis/redis.v1 with chart [email protected]

Read that output like a defender would. redis is Ready: True, and the message names the real Helm action, Helm install succeeded, so you know the chart genuinely installed and did not merely render its templates. backend still sits at Ready: False with dependency 'redis/redis' is not ready. This is a snapshot caught a beat after redis went green: backend has not run its own reconcile yet, so it is still showing the message from its last check, and the ordering guarantees Flux will not touch backend until that next reconcile confirms the dependency is up. If you ever catch a dependent going Ready while its dependency is still failing, your references are wrong, and that is worth an alert.

Applied Is Not Healthy

Here is the trap that catches everyone. A Kustomization flips to Ready: True the moment its manifests apply cleanly, not when the Pods behind them are running. (A Pod is the smallest unit Kubernetes runs: one or more containers scheduled together.) Apply means the API server accepted the YAML and wrote it down. It says nothing about whether the container ever started, whether the image (the packaged filesystem a container boots from) pulled, or whether an admission webhook (a service the API server calls to approve or reject each object before storing it) is answering. So a dependent gated on that Kustomization gets released into a half-built site.

To make Ready mean the workloads are actually up, you opt in. Set wait: true and Flux blocks on the health of every object it applied, computing readiness with kstatus (the shared status library that decides whether a Kubernetes object is Current, InProgress, or Failed, the same one the Flux command-line tool uses to report status). When waiting on everything is too broad or too slow, drop wait and list specific objects under healthChecks; Flux then gates on those alone. Either way, an unhealthy or timed-out object drops the Kustomization to NotReady, which stalls everything downstream that depends on it. That backpressure is the point. One default will bite you here: timeout falls back to the reconcile interval when you leave it unset, so a one-hour interval means a stuck gate can take up to an hour to surface as a visible failure. Set timeout explicitly whenever the interval is long.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: infra-controllers
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./infra/controllers
  prune: true
  timeout: 5m                      # flip to NotReady if health does not settle in time
  # wait: true                     # or gate Ready on EVERY applied object via kstatus
  healthChecks:                    # gate Ready on exactly these, not on everything
    - apiVersion: apps/v1
      kind: Deployment
      name: kyverno-admission-controller
      namespace: kyverno
    - apiVersion: apps/v1
      kind: Deployment
      name: ingress-nginx-controller
      namespace: ingress-nginx

flux get kustomizations

NAME               REVISION                                            SUSPENDED  READY  MESSAGE
flux-system        main@sha1:7f3a9c1b8e2d5a4c6f0938e1d7c2b5a4f6e8d0c9  False      True   Applied revision: main@sha1:7f3a9c1b8e2d5a4c6f0938e1d7c2b5a4f6e8d0c9
infra-controllers  main@sha1:7f3a9c1b8e2d5a4c6f0938e1d7c2b5a4f6e8d0c9  False      False  Health check failed after 5m0s, timeout waiting for: [Deployment/kyverno/kyverno-admission-controller status: 'InProgress']
apps                                                                   False      False  dependency 'flux-system/infra-controllers' is not ready

Now the story is honest. The Kyverno admission controller never came up inside the 5m budget, so infra-controllers failed its health check and is Ready: False. Because apps depends on it, apps stays parked and never applies a single workload. The rollout stopped at the exact place the inspector failed, which is what you want. Without those healthChecks, infra-controllers would have read Ready: True the instant its manifests applied, and apps would already be live against an enforcement layer that is not answering.

Teaching Flux About Custom Resources

Flux ships built-in health checks for the common Kubernetes kinds and for its own objects. Give it a Deployment or a Pod and it knows what healthy looks like. Hand it a type it has never seen and it shrugs, the way a building inspector who only knows concrete and steel goes quiet when you point at some custom composite he has no spec sheet for. Your cluster is full of those types: a cert-manager Certificate, a Crossplane claim, some operator's bespoke resource. For them, healthCheckExprs lets you write the spec sheet yourself in CEL (Common Expression Language, a small, safe language for writing true/false checks). You declare when an object is current (healthy) and, optionally, failed, as expressions evaluated against each object's status. Flux runs them while a wait or healthChecks gate is active. That is how you make a rollout wait for real TLS (Transport Layer Security, the encryption behind an HTTPS address) to be issued before the ingress (the component that routes outside web traffic to services inside the cluster) that serves it comes up.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: platform-certs
  namespace: flux-system
spec:
  interval: 10m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./platform/certs
  prune: true
  wait: true
  timeout: 3m
  healthCheckExprs:
    - apiVersion: cert-manager.io/v1
      kind: Certificate
      # CEL, evaluated against each Certificate's .status
      current: status.conditions.exists(e, e.type == 'Ready' && e.status == 'True')
      failed: status.conditions.exists(e, e.type == 'Ready' && e.status == 'False')

kubectl -n istio-system get certificate wildcard-example

NAME              READY   SECRET                AGE
wildcard-example  False   wildcard-example-tls  90s

flux get kustomization platform-certs

NAME            REVISION                                            SUSPENDED  READY    MESSAGE
platform-certs  main@sha1:7f3a9c1b8e2d5a4c6f0938e1d7c2b5a4f6e8d0c9  False      Unknown  Reconciliation in progress

The Certificate is not ready, so the CEL current expression returns false, so platform-certs holds at Unknown and anything depending on it waits. The moment cert-manager issues the certificate and flips the Ready condition to True, the expression passes, the gate opens, and the dependents proceed. You taught Flux to read a type it shipped no knowledge of.

Verify It, And Why A Defender Cares

Trust the gate, then check the gate. Plain kubectl (the Kubernetes command-line tool) shows you the same readiness the controller acted on, printed straight from each object's Ready condition, so you can audit the whole chain without any Flux-specific tooling.

kubectl -n flux-system get kustomizations

NAME               AGE   READY   STATUS
apps               42h   False   dependency 'flux-system/infra-controllers' is not ready
flux-system        42h   True    Applied revision: main@sha1:7f3a9c1b8e2d5a4c6f0938e1d7c2b5a4f6e8d0c9
infra-controllers  42h   False   Health check failed after 5m0s, timeout waiting for: [Deployment/kyverno/kyverno-admission-controller status: 'InProgress']

Now the operations and security payoff. Your infrastructure layer installs Kyverno, a policy engine that enforces rules through an admission webhook. One rule blocks privileged Pods. If your app Kustomization does not gate on Kyverno being live, Flux will apply your workloads during the window where the webhook is not yet answering, and depending on the webhook's failure policy those Pods can slip in unchecked. An attacker with an existing foothold loves that window; nudging the controller to restart at reconcile time is enough to widen it. With a health gate on the infrastructure Kustomization, the app Kustomization stays NotReady until enforcement is genuinely up, the privileged workload never gets applied, and your alert on Ready == False fires instead of a policy quietly failing open. The same logic covers a secrets operator that must decrypt before apps mount the secret, and cert-manager issuing a real certificate before traffic reaches it.

Try this

Run flux get helmreleases -A on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: ready is not healthy unless you asked. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related