Sign, verify & allow registries

Keyless signing and verification at admission.

Your scanner cleared the image on Tuesday. No critical CVEs (Common Vulnerabilities and Exposures), so it shipped. On Friday someone with push access repushed a backdoored build to the same latest tag, and every node that pulled it did so without a word of complaint. That's the hole. Scanning tells you an image was clean at the moment you looked. It says nothing about whether the thing running right now is the same thing you scanned.

Three steps close that gap, and they only work as a set. Build makes the image. Scan and sign produce evidence about it. Verify at admission is the single step that can refuse to run something. Most teams do the first two and quietly skip the third, then act surprised when a signed image and an unsigned one schedule exactly alike. A signature nobody checks stops nothing. Anyone can still deploy an image that was never signed at all.

Sign the digest, not the tag

A signature is a tamper-evident seal, the kind that tells you whether someone opened the box in transit. The old way to make one is a private key you keep in a vault and guard forever, which turns into the most valuable single thing an attacker can lift from your pipeline. Cosign keyless signing throws that key away. Your continuous integration (CI) job proves who it is to Sigstore using OpenID Connect (OIDC), the same 'log in with Google' handshake you already know. A certificate authority called Fulcio hands back a cert that lives for a few minutes and then expires. Cosign signs with it, then writes a record to Rekor, a public transparency log that behaves like a notary's ledger. Anyone can later prove the entry was there, and nobody can quietly edit it. Nothing durable stays on disk to steal.

One detail does a lot of work here. Sign the digest, not the tag. A tag like latest is a sticky note you can peel off and slap on any image. The digest is a SHA-256 (Secure Hash Algorithm, 256-bit) hash of the actual image content, so it names exactly one build and changes the instant a single byte moves. Your pipeline signs the digest it just built and scanned, never the mutable tag.

$ cosign sign --yes "$CI_REGISTRY_IMAGE@$(cat image.digest)"
Generating ephemeral keys...
Retrieving signed certificate from Fulcio...
Successfully verified SCT...
tlog entry created with index: 148372911
Pushing signature to: registry.internal/payments-api
#
# a short-lived Fulcio cert was issued to the CI OIDC identity,
# the signature was pushed, and a tamper-evident entry landed in Rekor.
# no long-lived key was ever written to disk.

Before you hand this off to an admission controller that runs it on every pod, do the check once by hand so you know exactly what it asserts. State the identity you expect. A signature that exists but was made by the wrong subject has to fail just as hard as no signature at all, otherwise you've built a lock that opens for anyone with a key.

$ cosign verify \
    --certificate-identity-regexp ".*@acme.internal" \
    --certificate-oidc-issuer "https://gitlab.acme.internal" \
    registry.internal/payments-api@sha256:9f2a...
 
Verification for registry.internal/payments-api@sha256:9f2a... --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
[{"critical":{"identity":{"docker-reference":"registry.internal/payments-api"}...}}]
 
# same command, run against an unsigned digest
$ cosign verify ... registry.internal/payments-api@sha256:dead... ; echo "exit=$?"
Error: no matching signatures
exit=1                     # non-zero exit = reject

Check it at the door

An admission controller is the bouncer at the cluster door. Every request to create a pod passes through it before anything schedules, and the bouncer is allowed to say no. Kyverno is a policy engine that plugs into that spot. Give it two rules, and you need both. One pins the registry so images can only come from registry.internal. The other verifies the signature and the identity behind it. Drop the registry pin and a signature check on its own becomes a trap: an attacker signs their own malicious image with their own valid identity and sails right through. Drop the signature check and anyone who can push to your registry name runs whatever they like. The question you actually want answered is narrow. Is this image signed by us, and did it come from our registry?

Why enforce this at admission instead of just in the pipeline? Because the pipeline isn't the only road into your cluster. A tired engineer with kubectl access can apply a manifest straight to production. A GitOps controller can be pointed at a fork. A compromised CI runner can push past its own gates. Admission is the one chokepoint every pod crosses no matter how it got there, which is why the enforcing check belongs there and nowhere else.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: verify-and-restrict }
spec:
  validationFailureAction: Enforce
  rules:
  - name: only-internal-registry
    match: { any: [{ resources: { kinds: [Pod] } }] }
    validate:
      message: "images must come from registry.internal"
      pattern: { spec: { containers: [{ image: "registry.internal/*" }] } }
  - name: verify-signature
    match: { any: [{ resources: { kinds: [Pod] } }] }
    verifyImages:
    - imageReferences: ["registry.internal/*"]
      attestors:
      - entries:
        - keyless: { issuer: "https://gitlab.acme.internal", subject: "*@acme.internal" }

There's a quieter thing Kyverno does once a signature checks out. By default it rewrites the pod's image reference from the tag to the exact digest it just verified (that's the mutateDigest option, on out of the box). Skip that and you keep a subtle hole: Kyverno verifies payments-api:v3, admits the pod, and a second later the kubelet pulls payments-api:v3 after someone quietly repointed the tag at a different image. The thing you checked and the thing that runs are no longer the same thing. That's a classic time-of-check-to-time-of-use (TOCTOU) gap, and pinning the admitted pod to the digest closes it. Same rule as before, enforced by the cluster this time: the digest is the truth, the tag is a suggestion.

Apply it, then try to break it. The only honest test of an admission policy is a pod that should be rejected actually getting rejected, so throw two bad pods at it: one from a public registry, one from your own registry that was never signed.

$ kubectl apply -f verify-images.yaml
clusterpolicy.kyverno.io/verify-and-restrict created
 
# a public image — should trip the registry rule
$ kubectl run rogue --image=docker.io/library/nginx:latest
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
resource Pod/default/rogue was blocked due to the following policies
 
verify-and-restrict:
  only-internal-registry: 'validation error: images must come from registry.internal.
    rule only-internal-registry failed at path /spec/containers/0/image/'
 
# an internal image that was never signed — should trip the signature rule
$ kubectl run unsigned --image=registry.internal/payments-api:pr-42
Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:
resource Pod/default/unsigned was blocked due to the following policies
 
verify-and-restrict:
  verify-signature: 'failed to verify image registry.internal/payments-api:pr-42:
    no matching signatures found.'

Enforce, not Audit

Kyverno has two moods. Audit writes a policy report and lets the pod run anyway. Enforce blocks it. A policy left on Audit feels safe because the dashboard fills up with violations, but unsigned images keep scheduling the entire time. Flip it to Enforce and confirm a bad pod is genuinely turned away, the way the runs above show. Roll it out namespace by namespace if you're nervous about breaking a deploy, but don't ever mistake a red dashboard for a closed door.

Audit mode is for finding gaps. Leaving production in Audit forever is how unsigned images keep shipping.

Key management matters as much as the signature. Protect cosign keys or use keyless issuers with identity bindings you understand.

Verify in CI and again at admit. CI-only checks cannot stop a human kubectl apply from a laptop.

Signature verification policies must name the public keys or identity issuers you trust. A verify that accepts any signature from any key on the internet is a false sense of safety. Keep the trust root in the same change review as the admission deployment. Write the verification command next to the control in the same pull request, and keep the sample output so the next on-call person can tell pass from fail without guessing.

Try this

Sign an image digest with cosign and verify it. Then show an unsigned digest failing an admission check in a lab policy.

$ DIGEST=$(crane digest registry.internal/payments-api:1.4.2)
$ cosign sign --key cosign.key registry.internal/payments-api@$DIGEST
Pushing signature to: registry.internal/payments-api
 
$ cosign verify --key cosign.pub registry.internal/payments-api@$DIGEST
Verification for registry.internal/payments-api@$DIGEST --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key
 
$ kubectl -n payments apply -f unsigned-deploy.yaml
Error from server (Deny): admission webhook "cosign-gate" denied the request: no valid signature

Takeaway

Sign digests, verify at admission, and run enforce mode only after the pipeline always signs. Tags lie; signatures on digests do not.

Related