CoursesInfrastructure as Code & automationHelm: the Kubernetes package manager

Helm: the Kubernetes package manager

Template and release K8s apps.

Intermediate14 min · lesson 15 of 23

A Kubernetes app is almost never one file. A payments service needs a Deployment (the controller that keeps a set number of copies of your container running), a Service (a stable internal address for those copies), a ConfigMap to hold its settings, an Ingress (the door in from outside the cluster), a ServiceAccount for its identity, and probably a HorizontalPodAutoscaler, which adds and removes copies as load changes. That is six files of YAML, the indented text format Kubernetes reads, for one service. Now build dev, staging, and production versions of all six, where four values differ between them. Copy and paste gives you eighteen files that drift apart within a month.

Mail merge solved this shape of problem for letters decades ago. Write the letter once with blanks where the name and address go, feed it a spreadsheet, print a thousand personalized copies. Helm is mail merge for Kubernetes YAML, with a filing cabinet bolted on the side. The letter with blanks is a chart. The spreadsheet row is a values file. The dated copies of everything you actually sent, kept so you can pull one back out later, are the release history.

Now the precise version. A chart is a directory of templated manifests plus default values, carrying its own version number. A release is one installation of a chart, into one namespace, under a name you choose. Every install or upgrade of that release creates a new revision, and Helm stores the exact YAML that revision applied, which is what makes helm rollback real rather than hopeful. If you have written Terraform, a chart sits close to a module: shared code, per-environment inputs, versioned and reviewed.

One architectural fact shapes the whole security story. Helm runs entirely on your side of the wire. Nothing from Helm sits inside your cluster. Helm 2 shipped a piece that did, a service called Tiller that teams routinely handed cluster-admin, and it sat there as a standing target. Helm 3 deleted it and Helm 4 kept it deleted. Today the helm binary on your laptop or CI runner (the machine your build pipeline runs on) reads your kubeconfig and talks to the Kubernetes API server, the front door every cluster change passes through, as you, under your RBAC (role-based access control, the rules deciding which identities may touch which objects). A chart can do whatever you can do. Nothing less, nothing more.

terminal
$ helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
$ helm repo update
$ helm search repo ingress-nginx --versions | head -4
output
"ingress-nginx" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "ingress-nginx" chart repository
Update Complete. ⎈Happy Helming!⎈
NAME CHART VERSION APP VERSION DESCRIPTION
ingress-nginx/ingress-nginx 4.15.1 1.15.1 Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx 4.15.0 1.15.0 Ingress controller for Kubernetes using NGINX a...
ingress-nginx/ingress-nginx 4.14.5 1.14.5 Ingress controller for Kubernetes using NGINX a...

What Is Actually Inside a Chart

helm create writes a working chart for you. It is the fastest way to see the shape of one without inventing it yourself.

terminal
$ helm create payments-api
$ find payments-api | sort
output
Creating payments-api
payments-api
payments-api/.helmignore
payments-api/Chart.yaml
payments-api/charts
payments-api/templates
payments-api/templates/NOTES.txt
payments-api/templates/_helpers.tpl
payments-api/templates/deployment.yaml
payments-api/templates/hpa.yaml
payments-api/templates/ingress.yaml
payments-api/templates/service.yaml
payments-api/templates/serviceaccount.yaml
payments-api/templates/tests
payments-api/templates/tests/test-connection.yaml
payments-api/values.yaml

Think of it as a parcel with a label, a packing list, and the goods. Chart.yaml is the label: name, version, description. values.yaml is the packing list of defaults, and it doubles as the documentation of what the chart lets you change, so read it first on any chart you did not write. Everything under templates/ goes through Go's template engine and has to come out the other side as valid Kubernetes YAML, correct indentation included. _helpers.tpl holds named snippets you reuse, such as a label block or a naming rule, so the chart stays readable as it grows. NOTES.txt gets printed to whoever installs the chart. templates/tests/ holds pods that run only when you ask for them with helm test. The charts/ directory arrives empty, and it is where subcharts land once you declare dependencies in Chart.yaml and run helm dependency update. You can also add a values.schema.json, a JSON Schema file that validates values before rendering, so a typo fails on your laptop instead of in the cluster. Helm 4 throws in one extra template, httproute.yaml, for the Gateway API.

payments-api/Chart.yaml
apiVersion: v2
name: payments-api
description: The payments API service
type: application
version: 0.1.0 # the CHART's version. this is what you pin
appVersion: "1.4.2" # a label for the software inside. informational only

Two version fields live in that file, and confusing them costs you an afternoon. version is the chart's own version, and it is the number you pin when you install. appVersion is a label describing the software inside, and it is what the APP VERSION column of helm list reports. Change your image tag in values without touching appVersion and that column keeps announcing the old number while a completely different image runs. Read the manifest, not the column. When the chart itself is ready to ship, helm package payments-api produces payments-api-0.1.0.tgz. You can push that to an OCI (Open Container Initiative) registry, the same kind of registry that stores container images, with helm push payments-api-0.1.0.tgz oci://registry.acme.internal/charts, then install straight from there by version.

Templates and Values: One Chart, Many Environments

A template is ordinary manifest YAML with holes punched in it. {{ .Values.replicaCount }} pulls from whatever values are in effect. {{ .Chart.Name }} and {{ .Release.Name }} come from the chart metadata and from the release you are installing right now. include pulls in a named helper from _helpers.tpl. A {{- with }} block is a guard: if that value is empty, the whole block vanishes from the output instead of leaving a dangling key. And nindent 12 indents the block piped into it by twelve spaces, which matters more than it looks. YAML is whitespace-sensitive, so a misindented render produces a manifest the API server rejects with a spectacularly unhelpful error.

payments-api/templates/deployment.yaml (excerpt)
spec:
replicas: {{ .Values.replicaCount }}
template:
spec:
serviceAccountName: {{ include "payments-api.serviceAccountName" . }}
{{- with .Values.podSecurityContext }}
securityContext:
{{- toYaml . | nindent 8 }}
{{- end }}
containers:
- name: {{ .Chart.Name }}
{{- with .Values.securityContext }}
securityContext:
{{- toYaml . | nindent 12 }}
{{- end }}
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
{{- with .Values.resources }}
resources:
{{- toYaml . | nindent 12 }}
{{- end }}
values-prod.yaml
replicaCount: 5
image:
repository: registry.acme.internal/payments-api
tag: "1.4.2" # pin an immutable tag, never "latest"
pullPolicy: IfNotPresent
podSecurityContext: # pod level
runAsNonRoot: true
runAsUser: 10001
seccompProfile:
type: RuntimeDefault # seccomp limits which syscalls the process may make
securityContext: # container level
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests: { cpu: 200m, memory: 256Mi }
limits: { cpu: "1", memory: 512Mi }

Values stack up like coats of paint, and the last coat wins. The chart's own values.yaml is the floor. Each -f file paints over it, left to right. --set beats every file. When a setting is not what you expected, that order is usually the reason. helm get values payments -n payments prints the overrides Helm recorded for the live release, and adding -a prints the full computed set including chart defaults, which is the one you want at 3am. Keep the values file as the reviewable artifact in Git and reserve --set for things CI injects, like an image tag.

terminal
$ helm lint ./payments-api -f values-prod.yaml
$ helm template payments ./payments-api -f values-prod.yaml \
--set replicaCount=6 --show-only templates/deployment.yaml | head -18
output
==> Linting ./payments-api
[INFO] Chart.yaml: icon is recommended
1 chart(s) linted, 0 chart(s) failed
---
# Source: payments-api/templates/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-payments-api
labels:
helm.sh/chart: payments-api-0.1.0
app.kubernetes.io/name: payments-api
app.kubernetes.io/instance: payments
app.kubernetes.io/version: "1.4.2"
app.kubernetes.io/managed-by: Helm
spec:
replicas: 6
selector:
matchLabels:
app.kubernetes.io/name: payments-api
app.kubernetes.io/instance: payments

Read the YAML Before the Cluster Does

helm template is print preview. It runs the whole rendering step on your machine and writes the finished manifests to stdout without touching the cluster, so you can read exactly what would be created before anything is. Add --validate and it also asks the API server whether those objects are legal, which does need cluster access. The first question to ask of any chart you did not write: what does this thing create?

terminal
$ helm template audit ingress-nginx/ingress-nginx --version 4.15.1 > /tmp/rendered.yaml
$ grep -c '^---' /tmp/rendered.yaml
$ grep -c '^kind:' /tmp/rendered.yaml
$ grep '^kind:' /tmp/rendered.yaml | sort | uniq -c | sort -rn
output
19
18
2 kind: ServiceAccount
2 kind: Service
2 kind: RoleBinding
2 kind: Role
2 kind: Job
2 kind: ClusterRoleBinding
2 kind: ClusterRole
1 kind: ValidatingWebhookConfiguration
1 kind: IngressClass
1 kind: Deployment
1 kind: ConfigMap

Notice the two counts disagree, and trust the second one. Helm emits a --- separator for every template it processes, including templates that render nothing. Here the PodDisruptionBudget template is switched off by default, so it produces a document containing only a comment. Nineteen separators, eighteen actual objects. Counting kind: is the honest count. Of those eighteen, only a handful are the workload. Two ClusterRoles, two ClusterRoleBindings, and a ValidatingWebhookConfiguration are cluster-wide power: permissions that apply in every namespace, plus a webhook that gets to inspect and reject other people's objects as they are created. For an ingress controller that is normal, and it is still worth knowing on purpose rather than by accident. The habit generalizes. Grep the rendered file for the things that grant reach.

terminal
$ helm template audit acme/node-agent --version 2.3.0 > /tmp/node-agent.yaml
$ grep -nEi 'hostpath|hostnetwork|hostpid|privileged|clusterrolebinding|"\*"|path: /' \
/tmp/node-agent.yaml
output
38:kind: ClusterRoleBinding
55: - apiGroups: ["*"]
56: resources: ["*"]
57: verbs: ["*"]
141: hostNetwork: true
142: hostPID: true
149: privileged: true
168: - mountPath: /host
172: hostPath:
173: path: /

Every one of those lines has a legitimate use. A log shipper or a security agent genuinely needs to see the host. The question is whether you decided to grant it. Wildcards for apiGroups, resources and verbs, bound cluster-wide, mean that pod's ServiceAccount token can read every Secret in every namespace, which is where your cloud credentials live. A hostPath volume with path: / mounted at /host hands the container the node's entire filesystem, including /etc/kubernetes and the kubelet's client certificate. hostPID plus privileged turns escaping to the node into a short walk. Put together, compromising that one pod is compromising the cluster.

So you do three things. Check whether the chart lets you turn the reach off in values, since many expose knobs like rbac.create, hostNetwork, or a narrower role. Run the rendered file past your policy engine before it reaches the cluster. And make the cluster enforce the rule anyway, using Pod Security admission or an admission policy engine. Admission control is the bouncer on the door: every object presented to the API server gets checked before it is let in, so no chart can quietly exceed what you allow. helm install --dry-run=server renders the manifests and submits them for that check without creating anything, which is the cheapest way to find out where you stand. Piping helm template into kubectl apply --dry-run=server does the same job for a chart you have not installed yet.

terminal
$ helm template audit acme/node-agent --version 2.3.0 | kubectl apply --dry-run=server -f -
output
serviceaccount/node-agent created (server dry run)
clusterrole.rbac.authorization.k8s.io/node-agent created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/node-agent created (server dry run)
Error from server: error when creating "STDIN": admission webhook "validate.kyverno.svc-fail" denied the request:
resource DaemonSet/default/node-agent was blocked due to the following policies
disallow-host-path:
autogen-host-path: 'validation error: HostPath volumes are forbidden. The field
spec.template.spec.volumes[*].hostPath must be unset. rule autogen-host-path
failed at path /spec/template/spec/volumes/0/hostPath/'
A chart is code, and it runs with your credentials
helm install renders someone else's templates and applies the result as you, so a chart creates anything your RBAC allows. Charts can also carry hooks, which are manifests annotated with helm.sh/hook (pre-install, post-upgrade, and friends) that Helm applies and waits for before the rest of the release goes out. Think of a hook as a note stapled to the outside of the parcel saying run me first. Usually it is a database migration Job. It can equally be a Job that pulls an image you have never heard of and runs it inside your network. helm template prints hook manifests too, and helm get hooks shows them for an installed release, so read the output for that annotation. Always pin with --version, because without it you install whatever the repo serves today, and two runs of the same command can differ. Where the publisher signs charts, helm pull --verify checks the provenance file against your keyring before anything is installed.
From chart to release, and back
1Chart, pinned version
templates/ + values.yaml + Chart.yaml
2Your values
-f values-prod.yaml, then --set
3Render locally
helm template: plain YAML, no cluster
4Read it
kinds, RBAC, hostPath, hooks
5Apply
helm upgrade --install --atomic
6Revision recorded
Secret sh.helm.release.v1.NAME.vN
7Rollback
re-applies a stored revision's manifest

Releases, Revisions, and the Ledger Helm Keeps

helm upgrade --install is the form to use in automation. Install if the release is absent, upgrade if it is present, one command either way. --atomic is the flag that saves weekends. It waits for the resources to become ready, and if anything fails or the timeout expires it puts the release back the way it was instead of leaving you half deployed. On a first install there is nothing to go back to, so it uninstalls the wreckage instead. --timeout defaults to 5m0s, which is short for anything pulling a large image. --create-namespace saves you a separate kubectl command. One naming note if you are on Helm 4: --atomic still works but is deprecated in favour of --rollback-on-failure.

terminal
$ helm upgrade --install payments ./payments-api \
--namespace payments --create-namespace \
-f values-prod.yaml \
--atomic --timeout 5m
output
Release "payments" does not exist. Installing it now.
NAME: payments
LAST DEPLOYED: Mon Jul 20 14:22:07 2026
NAMESPACE: payments
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
1. Get the application URL by running these commands:
export POD_NAME=$(kubectl get pods --namespace payments -l "app.kubernetes.io/name=payments-api,app.kubernetes.io/instance=payments" -o jsonpath="{.items[0].metadata.name}")
export CONTAINER_PORT=$(kubectl get pod --namespace payments $POD_NAME -o jsonpath="{.spec.containers[0].ports[0].containerPort}")
echo "Visit http://127.0.0.1:8080 to use your application"
kubectl --namespace payments port-forward $POD_NAME 8080:$CONTAINER_PORT

Ship a new build and the revision counter moves. Watch what the APP VERSION column does when the image changes and Chart.yaml does not.

terminal
$ helm upgrade payments ./payments-api -n payments \
-f values-prod.yaml --set image.tag=1.4.3 --atomic | head -7
$ helm list -n payments
output
Release "payments" has been upgraded. Happy Helming!
NAME: payments
LAST DEPLOYED: Mon Jul 20 15:04:51 2026
NAMESPACE: payments
STATUS: deployed
REVISION: 2
TEST SUITE: None
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
payments payments 2 2026-07-20 15:04:51.883421 +0530 IST deployed payments-api-0.1.0 1.4.2

Revision 2 is running image tag 1.4.3 and the table still says 1.4.2, because that column reads Chart.yaml and nothing else. So where does the history itself live? In the namespace, as Secrets. Each revision is one Secret named sh.helm.release.v1.RELEASE.vREVISION, of type helm.sh/release.v1, holding gzipped JSON of the whole release: the rendered manifest, the values you supplied, the status, the timestamps. Two consequences for defenders. Anyone who can read Secrets in that namespace can read every rendered manifest and every value you passed, so a password stuffed into a values file is now readable by everyone with secret-read on the namespace. And the ledger has a finite number of pages. --history-max defaults to 10, after which the oldest revisions are pruned and you can no longer roll back to them.

terminal
$ kubectl get secret -n payments -l owner=helm
$ kubectl get secret sh.helm.release.v1.payments.v2 -n payments \
-o jsonpath='{.data.release}' | base64 -d | base64 -d | gzip -d | head -c 190
output
NAME TYPE DATA AGE
sh.helm.release.v1.payments.v1 helm.sh/release.v1 1 54m
sh.helm.release.v1.payments.v2 helm.sh/release.v1 1 12m
{"name":"payments","info":{"first_deployed":"2026-07-20T14:22:07.412+05:30","last_deployed":"2026-07-20T15:04:51.883+05:30","deleted":"","description":"Upgrade complete","status":"deployed"}

That double base64 -d is not a typo. Kubernetes base64-encodes the Secret field, and Helm base64-encodes its own gzipped payload inside it, so you peel two layers before gzip sees anything.

Rollback, and What It Does Not Undo

Rolling back takes one command and a revision number. Helm reads that revision's stored manifest, applies it, and records the result as a new revision, so the history stays append-only and the rollback itself is visible to the next person who reads it.

terminal
$ helm rollback payments 1 -n payments --wait
$ helm history payments -n payments
output
Rollback was a success! Happy Helming!
REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION
1 Mon Jul 20 14:22:07 2026 superseded payments-api-0.1.0 1.4.2 Install complete
2 Mon Jul 20 15:04:51 2026 superseded payments-api-0.1.0 1.4.2 Upgrade complete
3 Mon Jul 20 15:19:33 2026 deployed payments-api-0.1.0 1.4.2 Rollback to 1
Rollback restores manifests, not the world
helm rollback re-applies the YAML a revision stored. Anything outside that YAML stays exactly where it is. A database migration that already ran is still applied. Data in a PersistentVolumeClaim is untouched. And if your image tag is mutable (latest, or a tag someone re-pushed), the old manifest and the new one name the same string, so you roll back to the identical broken bits. Pin tags you never move, or reference the image by digest, the cryptographic fingerprint of the exact image bytes. Two more sharp edges. helm upgrade --force replaces resources outright instead of patching them, which recreates pods and can fail on immutable fields such as a Service's cluster IP, so reach for it only when you know why (Helm 4 renames it to --force-replace). And helm uninstall removes every object the release owns, including any PersistentVolumeClaim the chart templated, unless it carries the annotation helm.sh/resource-policy: keep. PVCs created by a StatefulSet's volumeClaimTemplates are the exception: the StatefulSet controller made those, not Helm, so they survive and you clean them up by hand.

Proving the Cluster Still Matches the Chart

The filing cabinet only helps if what hangs on the wall still matches the copy in the drawer. helm get manifest prints exactly what the current revision applied. Pipe that into kubectl diff and you are comparing the chart's intent against the live objects field by field, with the API server doing the merge so its own defaults do not show up as noise. Hooks are stored separately and are not part of this output, which is one more reason to read helm get hooks as well.

terminal
$ helm get manifest payments -n payments | kubectl diff -f - -n payments
output
diff -u -N /tmp/LIVE-2181234567/apps.v1.Deployment.payments.payments-payments-api /tmp/MERGED-3312904455/apps.v1.Deployment.payments.payments-payments-api
--- /tmp/LIVE-2181234567/apps.v1.Deployment.payments.payments-payments-api 2026-07-20 15:41:02.118374611 +0530
+++ /tmp/MERGED-3312904455/apps.v1.Deployment.payments.payments-payments-api 2026-07-20 15:41:02.122374611 +0530
@@ -24,7 +24,7 @@
generation: 3
spec:
progressDeadlineSeconds: 600
- replicas: 12
+ replicas: 5
revisionHistoryLimit: 10
selector:
matchLabels:
@@ -63,7 +63,7 @@
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
- runAsNonRoot: false
+ runAsNonRoot: true
runAsUser: 10001
seccompProfile:
type: RuntimeDefault

Two changes here, and nobody mentioned either one. Someone scaled the Deployment to twelve replicas by hand, which is a capacity conversation. Someone also flipped runAsNonRoot to false, which is a security conversation: those containers may now start as user ID 0, which is root, and root inside a container is one bad mount or one kernel bug away from root on the node. Neither change appears in helm history, because neither went through Helm. kubectl diff exits 1 when it finds a difference and 0 when it does not, so it works as a check in a scheduled job rather than a thing you have to remember to run.

Quick check
01A bad build reaches production. You run helm rollback payments 2 -n payments, Helm reports success, the pods restart, and the broken behaviour is still there. What is the most likely reason?
Incorrect — a revision stores the fully rendered manifest, templates and values already baked together, and rollback re-applies that whole manifest.
Correct — the YAML went back, the bytes behind the tag did not.
Incorrect — --force replaces resources instead of patching them, which is riskier, and it is never required for a normal rollback. Helm 4 renames it to --force-replace.
Incorrect — Helm keeps up to --history-max revisions (10 by default) as Secrets, and asking for one that has been pruned makes rollback fail loudly rather than quietly substitute another.
02In a chart's Chart.yaml, what is the difference between the version and appVersion fields, and which one do you pin when installing?
Incorrect — this reverses them; version is the chart version you pin, appVersion is the label.
Correct — you pin version, while appVersion only labels the software and can keep showing an old number after you change the image tag.
Incorrect — they routinely differ and Helm packages them happily, which is exactly the gotcha to watch.
Incorrect — neither field refers to a Kubernetes API version or a Helm CLI version.
03You render a third-party chart with helm template and grep the output. Bound in a ClusterRoleBinding you find a role with apiGroups: ["*"], resources: ["*"], verbs: ["*"]. What is the concrete risk if you install this chart?
Correct — a cluster-wide wildcard role lets that ServiceAccount read all Secrets everywhere, so one compromised pod becomes a cluster compromise.
Incorrect — a ClusterRoleBinding applies the role's rules across all namespaces, which reaches namespaced Secrets.
Incorrect — helm template only previews; helm install applies exactly these objects with your credentials.
Incorrect — wildcard rules are valid role-based access control that Kubernetes accepts, which is what makes them dangerous.

Give every release you own a weekly two-command check: helm get manifest RELEASE -n NS | kubectl diff -f - -n NS, then compare the chart version in helm list against the version pinned in Git. A Deployment that quietly grew a hostPath mount, gained a second container, or lost runAsNonRoot shows up as a four-line diff on a Tuesday, which is a much smaller thing to deal with than the same finding in an incident review.

Try this

Run helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a chart is code, and it runs with your credentials. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related