Signing, provenance & OCI

Trust the charts you deploy.

A Helm chart is a box of instructions that tells Kubernetes what to run, and Helm carries out those instructions using your credentials. Whoever wrote the chart gets to create Deployments, mount Secrets, and hand out permissions, all under your name. So the question that matters is not whether the chart looks tidy. It is whether you know who produced these exact bytes, and whether anything has changed since they did.

Signing answers both halves of that question. It works like a wax seal on a letter. The sender presses a signet ring into hot wax, and anyone who knows the ring can tell the letter is genuine and was not opened along the way. Break the seal and it shows. Helm gives you two ways to seal a chart. The older one is built on GPG (GNU Privacy Guard, a long-standing command-line tool for signing and encrypting files). The newer one borrows the same machinery people already use to sign container images.

What a signature actually proves

A seal makes two claims at once, and both matter. The first is origin: who sealed this. The second is integrity: nothing has been altered since. Put together, these two claims are called provenance, the trustworthy record of where something came from. GPG proves them with a key pair. You hold a private key that only you can seal with, and you publish a matching public key that anyone can use to check the seal but never to forge one.

Here is the mechanism. helm package --sign reads your chart directory and computes a SHA-256 hash of the packaged file. Think of that hash (SHA stands for Secure Hash Algorithm) as a short fingerprint of the exact bytes: change a single byte and the fingerprint changes completely. Helm writes that fingerprint into a small text record next to the chart's metadata, then signs the whole record with your private key. You get two files out: the chart tarball, and a .prov file (short for provenance) sitting beside it.

# GnuPG 2.1+ keeps no secring.gpg, so export the secret key into the legacy format Helm reads
gpg --export-secret-keys > ~/.gnupg/secring.gpg
 
helm package --sign --key 'Acme Ops' --keyring ~/.gnupg/secring.gpg ./payments-api

The .prov file is plain text you can open and read. It holds the chart's metadata plus the file's fingerprint, wrapped in a clear-signed PGP block (PGP, or Pretty Good Privacy, is the standard GPG implements). The signature covers everything between the header and footer, so you cannot change the hash without breaking the seal.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
 
apiVersion: v2
name: payments-api
version: 1.4.0
description: Payments API service
type: application
 
...
files:
  payments-api-1.4.0.tgz: sha256:5f9e0c8a1d3b7e2f0a4c9d1e6b8f2a3c4d5e6f7081920a3b4c5d6e7f8091a2b3
-----BEGIN PGP SIGNATURE-----
 
wsBcBAEBCgAQBQJl9Qk...snip...Rk9Qk=
=Ab3d
-----END PGP SIGNATURE-----

On the other end, helm verify checks the seal. It needs the signer's public key in your keyring. If the key is there and the file is untouched, it tells you who signed the chart and confirms the hash matches.

helm verify payments-api-1.4.0.tgz

Now watch what a defender wants to happen when something is wrong. Say the tarball was altered in transit, or a compromised mirror served you a doctored copy. The bytes no longer match the hash the publisher sealed, so verification refuses to pass.

echo "junk" >> payments-api-1.4.0.tgz     # stand in for tampering in transit
helm verify payments-api-1.4.0.tgz

That is the whole point. The seal turns a silent swap into a loud failure. In practice you do not run verify by hand every time. You push the check into the deploy step with helm install --verify (and helm upgrade --verify), which refuses to proceed unless the .prov file is present and valid. A missing .prov fails the same way an altered chart does, so an attacker cannot strip the signature off to slip past the check.

helm install payments ./payments-api-1.4.0.tgz --verify

Charts move into OCI registries

For years, charts lived in HTTP chart repositories: a web server, an index.yaml file listing versions, and tarballs to download. That works, but it means chart trust and image trust are two separate worlds with two separate sets of tools. The shift is to store charts in an OCI registry (OCI stands for Open Container Initiative, the standard format and API that container registries speak). The registry that already holds your images can hold your charts too, in the same aisles, behind the same locks and access controls.

OCI support became stable in Helm 3.8. Before that you had to set the environment variable HELM_EXPERIMENTAL_OCI=1 to turn it on. You log in to the registry, then push a packaged chart to an oci:// address. The chart's name from Chart.yaml becomes the last segment of the path.

helm registry login registry.internal -u ci
helm push payments-api-1.4.0.tgz oci://registry.internal/charts

Notice the Digest line. The tag 1.4.0 is a friendly label, but the digest is the true, immutable name for those exact bytes. Two charts with the same tag can carry different digests. The same digest is always the same chart. Hold on to that difference, because it is where the newer signing tools earn their keep. Pulling and installing look the same as before, with an oci:// URL in place of the file path: helm install payments oci://registry.internal/charts/payments-api --version 1.4.0.

Cosign and the public ledger

Once a chart is an OCI artifact, you can sign it with Cosign, the same signer used for container images. Cosign behaves like a notary who not only stamps your document but also writes a copy of the stamp into a public ledger anyone can read. A forged stamp gets caught, because it was never recorded in the ledger.

The modern way to run it is keyless signing, which a defender likes because there is no long-lived private key sitting on a build machine waiting to be stolen. Instead you prove who you are with OIDC (OpenID Connect, the "sign in with" protocol behind your identity provider). A service called Fulcio issues you a certificate that lives for only a few minutes. Rekor, a public append-only transparency log, permanently records that this identity signed this digest at this time. Sign the digest, not the tag.

cosign sign registry.internal/charts/payments-api@sha256:9b2c8f1e0a4d6b3c5e7f90812a3b4c5d6e7f8091a2b3c4d5e6f7089012a3be4f

Verifying is the mirror image. You state, ahead of time, which identity you are willing to trust and which issuer vouched for it. Cosign then checks that the signature over this digest came from exactly that identity, and that Rekor holds the matching record. A signature from anyone else, or over any other bytes, does not pass.

cosign verify \
  --certificate-identity=https://gitlab.com/acme/charts//.gitlab-ci.yml@refs/heads/main \
  --certificate-oidc-issuer=https://gitlab.com \
  registry.internal/charts/payments-api@sha256:9b2c8f1e0a4d6b3c5e7f90812a3b4c5d6e7f8091a2b3c4d5e6f7089012a3be4f

Where the check happens

There are two natural gates, and they check different things. The first sits in CI (continuous integration, the automated pipeline that builds and tests your code before it ships): verify the chart's signature there, and fail the pipeline if it does not match. The second sits at admission, inside the cluster. An admission controller (a webhook the Kubernetes API server calls before it accepts a resource) can reject workloads whose container images are not signed by an identity you trust. Tools like Kyverno and the sigstore policy-controller run there and enforce that rule across the whole cluster.

Keep the two gates straight, because it is easy to blur them. Helm pulls and renders the chart into plain manifests before those manifests ever reach the API server, so the chart's own signature is checked at pull time or in CI, not by the admission controller. The admission controller checks the image references inside the rendered manifests. Chart trust and image trust are separate seals on separate objects. Do both, and you have covered the package your team installs and the containers it launches.

One honest limit. A valid signature proves origin and integrity, not good behavior. A correctly signed chart from a source you trust can still mount a host path, request cluster-admin, or pull an image you would rather not run. So checking the seal is the first step, not the last. Render the manifests with helm template (or install with --dry-run) and read what the chart will actually create before it touches the cluster, especially for anything unfamiliar or freshly bumped.

So the working habit is short. Pull by version, read off the digest, verify the seal against an identity you picked ahead of time, then hand that digest to Helm to render and install. Run helm pull oci://registry.internal/charts/payments-api --version 1.4.0, note the sha256 it resolves to, verify it, and deploy the digest rather than the tag. An unverified chart is code you have not read, running with your permissions. The two minutes it takes to check the seal is far cheaper than the incident review if you skip it.

Try this

Run gpg --export-secret-keys > ~/.gnupg/secring.gpg on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: there is no secring.gpg anymore. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related