Notifications & alerts

Events out, webhooks in.

Every Flux controller keeps up a running commentary on its own work. Each time the source-controller fetches a Git repository, the kustomize-controller applies a change, a health check fails, or a resource that drifted gets pulled back into line, the controller writes down what happened as a Kubernetes Event (a small, time-stamped record the cluster keeps for about an hour). It is like a busy restaurant kitchen where every cook calls out each plate as it leaves the line: useful in the moment, gone in a blur, impossible to follow unless you were standing right there. Run kubectl get events and that is what you get, a wall of chatter that scrolls past while nobody reads it.

Here is the part the kitchen analogy hides. Every Flux controller sends each of those events two ways at once. One copy lands in the Kubernetes API as the record you just watched scroll by. The other copy goes straight to the notification-controller over the cluster network. That second controller is the cook at the pass, the one who decides what actually leaves the kitchen. Think of an old telephone switchboard operator, the person who once connected one phone line to another by hand: the notification-controller takes that direct feed of events, keeps the few that matter, and routes them outward to Slack, Microsoft Teams, PagerDuty, or a Git host's status API (Application Programming Interface, the machine-to-machine doorway a service exposes so other programs can call it). It also works the other way, holding open inbound webhooks (doorbells that let an outside service ring into the cluster) so a git push can wake Flux the instant a change lands instead of waiting out the next poll. Two directions, one controller. Events out, webhooks in.

Events Out: Providers And Alerts

Two small objects split this job, and it pays to keep them straight. A Provider is an address-book entry. It says where a message goes and how you prove you are allowed to send it: a Slack channel, plus the Secret (the Kubernetes object built to hold sensitive values like passwords and URLs) that stores the incoming webhook URL. An Alert is a mail-sorting rule. It says what to forward. It points at one Provider and lists the Events it cares about by kind and name, with * as a wildcard that means all of them. The controller compares every Event against every Alert and forwards the matches.

You also set a severity floor, and this is where an on-call room lives or dies. Set eventSeverity: info and you get everything, down to the steady drip of 'reconciliation succeeded' (Flux's note that it checked the cluster against Git and found everything already matching). Set error and the channel stays silent until something genuinely breaks. An exclusionList of regular expressions (text patterns) drops lines you have learned to ignore, like a dependency that is briefly not ready during a rollout. And eventMetadata stamps each message with labels such as cluster or environment, so one shared Slack channel stays readable even when ten clusters are all talking into it at once.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: slack
  namespace: flux-system
spec:
  type: slack
  channel: deployments
  secretRef:
    name: slack-url          # Secret key 'address' = the incoming webhook URL
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: on-call
  namespace: flux-system
spec:
  providerRef:
    name: slack
  eventSeverity: error        # only failures; use 'info' to see every apply
  eventSources:
    - kind: Kustomization
      name: '*'               # all Kustomizations in this namespace
    - kind: HelmRelease
      name: '*'
  exclusionList:
    - "waiting for dependency"   # drop this recurring, harmless message
  eventMetadata:
    cluster: prod-eu-1          # stamped on every message to this channel

kubectl -n flux-system create secret generic slack-url \
  --from-literal=address=https://hooks.slack.com/services/T00/B00/XXXXXXXX

secret/slack-url created

There is no flux create secret notification command, so you make an ordinary generic Secret by hand, and the key must be address. Treat that webhook URL like a password: anyone who holds it can post into your channel, so it belongs in a Secret, never in the Alert and never in a committed file. If you would rather not write the Provider and Alert yourself, the flux command-line tool can generate both for you.

flux create alert-provider slack \
  --type slack --channel deployments --secret-ref slack-url
 
flux create alert on-call \
  --provider-ref slack --event-severity error \
  --event-source Kustomization/'*' --event-source HelmRelease/'*'

✚ generating Provider
► applying Provider
✔ Provider created
◎ waiting for Provider reconciliation
✔ Provider reconciliation completed
✚ generating Alert
► applying Alert
✔ Alert created
◎ waiting for Alert reconciliation
✔ Alert reconciliation completed

Whichever way you created them, confirm the controller accepted the objects. A ready Alert is one the controller has parsed and is now matching against live Events.

flux get alerts

NAME     SUSPENDED  READY  MESSAGE
on-call  False      True   Reconciliation succeeded

Webhooks In: Receivers

Polling a GitRepository once a minute works, but it means your fix can sit unnoticed for up to a minute after you push it. A Receiver removes that wait. It is a doorbell you wire from your Git host straight to the controller. On every push the Git host presses the button (it sends an HTTP POST, a HyperText Transfer Protocol request that pushes data to a URL), and the controller reacts by annotating the resources you listed, which forces them to reconcile right now: re-check the cluster against Git and apply any difference. The GitRepository refetches, the Kustomization re-applies, and the change is live seconds after it merges.

The doorbell has a lock built in. The endpoint path is not a friendly name; it is a SHA-256 (Secure Hash Algorithm 2, a one-way fingerprint) hash derived from a token Secret, which makes the URL long and unguessable. With type: github, every genuine request also carries an HMAC signature (Hash-based Message Authentication Code, a tamper-proof seal computed from a shared secret), and the controller verifies it before acting. A random POST from someone who found the path but does not hold the token is rejected. One catch: the notification-controller is not reachable from the internet by default, so you publish this single endpoint through an Ingress or a Gateway (the Kubernetes objects that expose an in-cluster service to the outside world).

apiVersion: notification.toolkit.fluxcd.io/v1
kind: Receiver
metadata:
  name: github-webhook
  namespace: flux-system
spec:
  type: github
  events:
    - "ping"
    - "push"
  secretRef:
    name: webhook-token       # Secret key 'token', shared with GitHub's webhook secret
  resources:
    - apiVersion: source.toolkit.fluxcd.io/v1
      kind: GitRepository
      name: flux-system       # apiVersion, kind, name must match the source exactly

TOKEN=$(head -c 12 /dev/urandom | sha256sum | cut -d ' ' -f1)
kubectl -n flux-system create secret generic webhook-token \
  --from-literal=token="$TOKEN"
 
kubectl -n flux-system get receiver/github-webhook \
  -o jsonpath='{.status.webhookPath}'

secret/webhook-token created
/hook/bed6d00b5555b1603e1f59b94d7bccc0af9ba498b0be4d5f9c2e1a3b4c5d6e7f

Take that path, prefix it with your Ingress host, and register https://webhooks.example.com/hook/bed6...e7f in the Git host's webhook settings, using the same token value as the webhook's shared secret and application/json as the content type. Push a commit, then check that the Receiver reports itself ready and is counting deliveries.

flux get receivers

NAME            SUSPENDED  READY  MESSAGE
github-webhook  False      True   Receiver initialized

One sharp edge comes attached to good hygiene. A Receiver's URL path is a hash of its token Secret, not a fixed name, so the day you rotate that Secret the /hook/<hash> path changes with it. The Git host keeps pressing the old button. Nothing errors inside the cluster, because nothing arrives. On the Git side, deliveries quietly start returning 404 (the HTTP status code for a URL that is not there) and reconciliation slips back to slow polling without a word of complaint. After any token change, re-read status.webhookPath and update the webhook in your Git host.

Closing The Loop Back To Git

The most useful Provider never messages a human. Instead of Slack, set type: github (or gitlab, gitea, bitbucketserver, azuredevops) with the repository URL and a token: a PAT (Personal Access Token, a scoped stand-in for your password) limited to the repo:status permission. Attach an Alert to your GitRepository and Kustomization Events, and the controller writes a commit status back onto the exact revision it just handled. Every commit earns a green check when Flux applies it cleanly and a red cross when reconciliation fails, shown right on the pull request. Git stops being a place you throw changes into and hope. It tells you whether the cluster agreed.

apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Provider
metadata:
  name: github-status
  namespace: flux-system
spec:
  type: github
  address: https://github.com/my-org/my-fleet   # the repo to annotate
  secretRef:
    name: github-pat          # Secret key 'token' = a PAT with repo:status scope
---
apiVersion: notification.toolkit.fluxcd.io/v1beta3
kind: Alert
metadata:
  name: apply-status
  namespace: flux-system
spec:
  providerRef:
    name: github-status
  eventSources:
    - kind: Kustomization
      name: apps

For a defender, that red cross is an early-warning light. If someone edits a live resource by hand, or a bad manifest slips through review, the failed apply surfaces as an Event, the Alert forwards it, and the commit that caused it wears the failure in plain sight. Because the status API is keyed to the precise commit hash, a stale or never-pushed revision never gets a mark at all, so a green check means Flux applied this exact commit cleanly, not some neighboring build. When the checks go quiet, though, do not read silence as calm.

A matcher that matches nothing looks identical to one with nothing to report. Both are silent. Start with flux events to confirm the controllers are still producing events at all. Then read the notification-controller's own logs, where an unmatched event leaves a very specific footprint: the controller says it is discarding an event because no Alert covers that object. Two traps cause most of these. The first is the Receiver's resources list, which matches on exact apiVersion, kind, name, and namespace, so bumping a source's API version quietly stops the trigger. The second is namespaces. A Provider can only read a Secret in its own namespace, and an Alert only matches Events from objects sitting in the same namespace as the Alert, unless you name another namespace on the source and the cluster still allows cross-namespace references (locked-down clusters often switch them off). Keep the Provider, its Secret, the Alert, and the objects it watches together in flux-system, and this whole class of problem never shows up.

flux events --all-namespaces

LAST SEEN  TYPE     REASON                   OBJECT                      MESSAGE
5m12s      Normal   GitOperationSucceeded    GitRepository/flux-system   stored artifact for commit 'fix: bump image tag'
5m10s      Normal   ReconciliationSucceeded  Kustomization/apps          Reconciliation finished in 1.4s, next run in 10m0s
92s        Warning  HealthCheckFailed        Kustomization/apps          Health check failed after 30s: timeout waiting for: [Deployment/web]

kubectl -n flux-system logs deploy/notification-controller | grep -i discard

{"level":"info","ts":"2026-07-20T10:14:58.221Z","logger":"event-server","msg":"discarding event, no alerts found for the involved object","reconciler kind":"Kustomization","name":"payments","namespace":"apps"}

Try this

Run flux get alerts on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: type: generic leaves the door unlocked. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related