>_SecOpsLog_
CoursesHelmHooks & release lifecycle

Hooks & release lifecycle

Run jobs at install/upgrade.

Intermediate12 min · lesson 8 of 12

A Helm release runs a bit like a stage play. The templates are your set and cast, the visible thing the audience watches. But between scenes something has to happen offstage: a database gets migrated, a cache warms up, an old secret rotates out. Hooks are the stagehands who work on cue. Helm pauses the main action, lets them do their job, and only raises the curtain on the next scene once they report back clean. A hook is any ordinary Kubernetes object, a Job that runs a task, a Pod that runs a container, even a ConfigMap that holds settings, that you tag with a special annotation (a free-form key-value label you can attach to any Kubernetes object) so Helm runs it at a chosen moment instead of folding it in with the rest of the app.

The release lifecycle

Every helm install, upgrade, and rollback is a versioned event. Helm writes each one into a ledger as a numbered revision you can list, inspect, and reverse, and that ledger is the release lifecycle. helm history shows every revision with its status and chart version. helm rollback winds the release back to an earlier revision, and records the rollback itself as a brand new revision, so you never lose the trail. helm status reports where the release stands right now. helm uninstall closes the ledger: on the way out it fires the release's delete hooks, and unless you pass --keep-history it purges the revision records too.

Hooks hang off the phases of that lifecycle. You mark a normal resource as a hook with the helm.sh/hook annotation, and Helm lifts it out of the regular manifest (the full set of resource definitions the chart would otherwise apply in one go), applies it at the phase you named, and waits for it to finish before doing anything else. The phases fire in a fixed order: pre-install then post-install, pre-upgrade then post-upgrade, pre-rollback then post-rollback, pre-delete then post-delete. There is one more, test, which runs only when you call helm test (that one belongs to the linting and testing lesson). A pre-install hook runs after Helm renders your templates but before it loads a single one of the chart's own resources into the cluster, which is exactly why it is the right place to gate a rollout.

terminal
# install, upgrade, and rollback each write a numbered revision
helm history web
helm status web
output
REVISION  UPDATED                   STATUS      CHART      APP VERSION  DESCRIPTION
1         Fri Jul 17 09:14:02 2026  superseded  web-1.4.0  2.0.0        Install complete
2         Fri Jul 17 10:02:41 2026  superseded  web-1.4.1  2.1.0        Upgrade complete
3         Fri Jul 17 11:20:18 2026  deployed    web-1.4.0  2.0.0        Rollback to 1
 
NAME: web
LAST DEPLOYED: Fri Jul 17 11:20:18 2026
NAMESPACE: default
STATUS: deployed
REVISION: 3
TEST SUITE: None

A pre-upgrade migration Job

The most common hook is a Job that has to finish before new code goes live, and a database schema migration is the textbook case. A Job here means a Kubernetes workload that starts a pod (the smallest unit Kubernetes runs, one or more containers that share an address and a lifecycle), runs it once to completion, and then stops. You want the migration to run on a fresh install and on every upgrade, and you want it to run before the new pods appear, so new code never talks to an old schema. Fix the wiring before you throw the switch. You get all of that from three annotations on an otherwise ordinary Job.

templates/db-migrate-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "web-chart.fullname" . }}-migrate
  annotations:
    # Run before the new pods roll out, on install AND upgrade
    "helm.sh/hook": pre-install,pre-upgrade
    # Lower weights run first; default is 0, so -5 sorts ahead of everything
    "helm.sh/hook-weight": "-5"
    # Clear a same-named leftover first, then delete this one when it passes
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  backoffLimit: 2               # retry twice before Kubernetes calls it failed
  template:
    spec:
      restartPolicy: Never      # each attempt is a fresh pod
      containers:
        - name: migrate
          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
          command: ["/app/bin/migrate", "up"]

Read the annotations top to bottom. helm.sh/hook: pre-install,pre-upgrade attaches the Job to both phases, so it runs on the first install and on every upgrade after. helm.sh/hook-weight: "-5" is a sort key: within a phase Helm runs hooks from lowest weight to highest, the default is 0, and a -5 guarantees this migration goes ahead of anything else in the phase. helm.sh/hook-delete-policy controls cleanup, which the next section covers. backoffLimit: 2 lets the Job retry twice before Kubernetes marks it failed, and restartPolicy: Never makes each attempt a fresh pod. When you run the upgrade, Helm creates the Job, waits for its pod to exit 0, and only then applies the new Deployment (the Kubernetes object that keeps a set of identical pods running at the count you asked for).

terminal
# roll out image 2.1.0; the pre-upgrade Job must pass before new pods appear
helm upgrade web ./web-chart --set image.tag=2.1.0
output
Release "web" has been upgraded. Happy Helming!
NAME: web
LAST DEPLOYED: Fri Jul 17 10:02:41 2026
NAMESPACE: default
STATUS: deployed
REVISION: 2
TEST SUITE: None

Because the delete policy includes hook-succeeded, Helm removes the migration Job the moment it passes, so a clean run leaves nothing behind to collide with next time. The safety payoff shows up when the migration does not pass.

One upgrade, phase by phase
1helm upgrade
revision N+1 written to the ledger
2pre-upgrade hooks
run by weight, each to completion
3chart manifest applied
new Deployment rolls out
4post-upgrade hooks
warm cache, notify, verify
If any hook fails to finish, Helm marks the release FAILED and stops; the phase after it is never applied and the old pods keep serving.

Ordering, cleanup, and failure

Two annotations give you the control. helm.sh/hook-weight orders hooks that share a phase. Helm reads the weights as whole numbers and runs the hooks from lowest to highest, so negative values go first. Hooks that carry the same weight fall back to alphabetical order by name, which is a coin toss you do not want to lean on, so set explicit weights whenever order actually matters. helm.sh/hook-delete-policy decides when Helm removes the hook object: before-hook-creation deletes a leftover of the same name before creating a new one, hook-succeeded deletes after a clean run, hook-failed deletes after a failure. Set no policy and before-hook-creation is the default. This matters most for Jobs, because a Job's name cannot be reused while an old one with that name still exists.

Failure handling is strict, and that strictness is the whole point. If a hook never reaches its finished state, Helm marks the release failed at that phase and stops. It does not apply the rest of the manifest, and it does not roll back on its own. Pass --atomic and the rule changes: any failure during the upgrade, a failed hook included, reverts the entire release to the previous revision, and --timeout bounds how long Helm waits before giving up (five minutes by default). Here is the same broken migration, first bare and then guarded.

terminal
# push a broken migration; the pre-upgrade hook fails and blocks the rollout
helm upgrade web ./web-chart --set image.tag=2.2.0
output
Error: UPGRADE FAILED: pre-upgrade hooks failed: 1 error occurred:
	* job web-migrate failed: BackoffLimitExceeded
terminal
# the failed hook Job is still there, so read exactly why it broke
kubectl logs job/web-migrate
 
# and confirm the release state
helm history web
output
migrate: applying 20260717_add_orders_index.up.sql
ERROR: relation "orders" does not exist (SQLSTATE 42P01)
migrate: 1 migration(s) failed
 
REVISION  UPDATED                   STATUS    CHART      APP VERSION  DESCRIPTION
1         Fri Jul 17 09:14:02 2026  deployed  web-1.4.0  2.0.0        Install complete
2         Fri Jul 17 10:41:55 2026  failed    web-1.5.0  2.2.0        pre-upgrade hooks failed

The old pods are still serving the previous version. The pre-upgrade hook blocked before the new Deployment was ever applied, so the bad build never reached a user. The failed Job is still in the namespace too, because hook-succeeded only fires on success, which is what lets you read its logs after the fact. helm history shows revision 1 still deployed and revision 2 sitting in failed, and helm status agrees; to get back to a clean state you either fix the migration and upgrade again or run helm rollback web 1. Add --atomic and Helm does that rollback for you the instant the hook fails.

terminal
# make the failure self-healing: revert the whole upgrade on any error
helm upgrade web ./web-chart --set image.tag=2.2.0 --atomic --timeout 5m
output
Error: UPGRADE FAILED: release web failed, and has been rolled back due to atomic being set: pre-upgrade hooks failed: 1 error occurred:
	* job web-migrate failed: BackoffLimitExceeded

Hooks are a trust boundary

A hook runs with the service account its pod names (the identity a pod presents to the Kubernetes API when it makes requests), at a moment when nobody is watching the screen, and Helm will not clean it up on uninstall. Turn that around and it is an attacker's dream. A pre-install hook Job buried in a chart you pulled from some public registry can read every secret that account can reach and phone home before a single application pod exists, and helm uninstall will leave that Job sitting in the namespace afterward. So treat hooks as code you are about to execute, not as configuration. For a release that is already installed, ask Helm directly what it carries.

terminal
# dump every hook a release will run, with its phase and weight
helm get hooks web
output
---
# Source: web-chart/templates/db-migrate-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: web-migrate
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  backoffLimit: 2
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: "registry.example.com/web:2.1.0"
          command: ["/app/bin/migrate", "up"]

helm get hooks prints every hook the release carries, so you can see precisely what fires on install, upgrade, and delete. For a chart you have not installed yet, helm template ./web-chart renders the hooks right alongside the rest of the manifest, so a grep for helm.sh/hook surfaces them before you commit to anything. Run charts under a scoped service account rather than a cluster-admin one, and scan that rendered output the same way you would scan any other Kubernetes manifest.

Hook resources are not tracked by the release
Helm applies a hook, confirms it finished, and then forgets about it. The object is never adopted into release state, a later helm upgrade does no three-way merge on it (the reconcile Helm normally runs to fold together your edits, the live object, and the last-applied version), and helm uninstall does not delete it. To clean one up you need a helm.sh/hook-delete-policy, or a Job ttlSecondsAfterFinished field (a timer after which Kubernetes deletes a completed Job), and that policy fires when the hook runs, never at uninstall, so any hook without one lingers in the namespace after the release is gone. The subtle trap is the cleanup you thought you had. The default policy is before-hook-creation, which clears a same-named leftover before the next run, but the moment you set an explicit delete-policy and leave before-hook-creation out of the list, you lose that guard. A named Job that failed under a hook-succeeded-only policy survives with its fixed name, and the next upgrade dies with jobs.batch "web-migrate" already exists. Keep before-hook-creation in the list whenever you name a hook Job.
Quick check
01A pre-upgrade migration Job exits non-zero during helm upgrade and you did not pass --atomic. Where does that leave the cluster and the release ledger?
Incorrect — The hook has to reach its finished state before Helm loads any of the chart's own resources, so the new pods are never created in the first place.
Incorrect — Cleanup fires only on the outcome you asked for. hook-succeeded means a pass, so a failed Job stays put and kubectl logs job/web-migrate still gives you the SQLSTATE error.
Correct — The phase blocks and the bad build never reaches a user. helm history shows revision 1 still deployed next to revision 2 failed, and helm status agrees.
Incorrect — Nothing reverts on its own. Reverting the whole upgrade on any error is exactly what --atomic buys you, and without it the release just sits in failed until you act.
02Two hooks share the pre-upgrade phase: the migration Job at helm.sh/hook-weight "-5" and a cache warmer with no weight annotation at all. Which one goes first?
Correct — Weights are read as whole numbers and sorted ascending, so anything negative gets ahead of the pack. That is the entire reason the migration carries -5.
Incorrect — A missing annotation is not the same as no ordering. Helm fills in 0 for that hook, which places it behind any negative weight in the same phase.
Incorrect — File layout has no say here. Two hooks in one phase are ordered by weight no matter what their filenames are or where they sit in the chart.
Incorrect — That is the rule inverted. Weight leads and the name settles a tie between equal weights, which is why you set weights explicitly whenever order actually matters.
03A hook Job named web-migrate carries only helm.sh/hook-delete-policy: hook-succeeded. It fails once, and the next helm upgrade dies with jobs.batch "web-migrate" already exists. What went wrong?
Incorrect — Retry budget and name collision are separate problems. Even a Job that used every attempt only blocks the next upgrade because the object itself is still sitting there.
Incorrect — --atomic reverts the release to the previous revision. It does not hand your hook a delete policy, so the stale Job stays in the namespace either way.
Incorrect — Hook Jobs are meant to keep a stable name so you can find them afterwards, which is what makes kubectl logs job/web-migrate work. The fix lives in the delete policy.
Correct — Listing hook-succeeded on its own replaces the default rather than adding to it. Put before-hook-creation back alongside it and Helm clears the stale Job before creating the new one.

One habit pays for itself: before you install or upgrade anything that carries hooks, run helm get hooks against the current release and diff it against what the new chart renders. A hook that changed phase, picked up a new weight, or started running under a different service account is the kind of change that never appears in a Deployment diff, and it is the one most worth catching before it runs.

Try this

Run helm history web on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: hook resources are not tracked by the release. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Subcharts & dependenciesLibrary charts & reuse

Related

What Kustomize is: the overlay modelKustomizeHelm cheat sheetCheat sheetHelm vs KustomizeComparison