Verification at admission

The last gate: unsigned means unscheduled.

A signed-for parcel does nothing if the courier hands it over without looking at the signature. Every step before this one produced paperwork: signed commits, provenance (a signed record of how and where the artifact was built), a software bill of materials (SBOM, a list of everything inside the artifact), scan results, cryptographic signatures. Admission is the doorway where someone finally reads that paperwork and decides whether the parcel comes in. In Kubernetes, that doorway has a name: admission control. Get it right and an image that lacks the evidence you demand never runs. It is refused before it is ever scheduled onto a machine.

The door is the admission webhook

Kubernetes runs on a front desk called the API server (API, application programming interface, the endpoints other programs call). Nothing happens in the cluster without a request passing through it: create this pod, scale that deployment, mount this secret. A pod is the smallest thing Kubernetes runs, one or more containers that live and die together. Before the API server writes a new pod into its database (etcd, the key-value store that holds cluster state) and lets the scheduler place it on a node, one of the machines that actually runs your containers, it can phone a set of outside checkers and ask, should I allow this? Those checkers are admission webhooks. A webhook is an HTTP call (HTTP, the protocol web requests ride on) the API server makes out to another service. A policy engine like Kyverno or the Sigstore policy-controller registers itself as one of those checkers.

Same building, one more character. The webhook is the bouncer the front desk calls before it stamps anyone's entry pass. The bouncer looks at the container image the pod wants to run, pulls the signatures and attestations attached to that image in the registry (the server that stores your container images), checks them against your rules, and answers admit or deny. An attestation is a signed statement about the image, for example, this was built by pipeline X from commit Y. Deny means the pod is rejected at creation. It never reaches a node, and the person who ran kubectl (the Kubernetes command-line tool) sees the error right away.

A signature alone proves nothing

Here is the mistake that turns a signature check into theater. You verify that an image is signed, full stop, and you feel safe. But anyone can sign anything. An attacker who pushes a malicious image to your registry can sign it with their own key or their own identity, and a check that only asks is there a signature waves it straight through.

A real check asks four questions, and every one has to pass. Is the image signed? Was it signed by the identity you actually expect, your continuous integration system (CI, the automated pipeline that builds and ships your code) proving who it is through OpenID Connect (OIDC, a standard way for one service to prove its identity to another), backed by the issuer that vouched for it? Are the attestations you require present and valid, say provenance from your builder and a passing scan? Does the image live in a registry you trust? Drop any one of these and you leave a hole.

You can run the same check by hand that the cluster runs at admission. cosign is the command-line tool from the Sigstore project for signing and verifying artifacts. Notice the identity flags, they are the whole point.

cosign verify \
  --certificate-identity-regexp '^https://gitlab.acme.internal/acme/.+$' \
  --certificate-oidc-issuer 'https://gitlab.acme.internal' \
  registry.acme.internal/acme/checkout:1.4.2

Verification for registry.acme.internal/acme/checkout:1.4.2 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
 
[{"critical":{"identity":{"docker-reference":"registry.acme.internal/acme/checkout"},
"image":{"docker-manifest-digest":"sha256:9f2a4c...c1"},"type":"cosign container image signature"},
"optional":{"Issuer":"https://gitlab.acme.internal",
"Subject":"https://gitlab.acme.internal/acme/checkout//.gitlab-ci.yml@refs/heads/main"}}]

Now the attacker's copy. Same repository path, same tag scheme, but signed by whoever pushed it instead of by your pipeline. The signature exists. It does not belong to the identity you named.

cosign verify \
  --certificate-identity-regexp '^https://gitlab.acme.internal/acme/.+$' \
  --certificate-oidc-issuer 'https://gitlab.acme.internal' \
  registry.acme.internal/acme/checkout:1.4.2-patched

Error: no matching signatures: none of the expected identities matched what was in the certificate
main.go:74: error during command execution: no matching signatures:
none of the expected identities matched what was in the certificate

A signature check on its own would have passed that image. The identity check catches it. The same logic extends to the build record: verify the provenance attestation, not only that one exists, and confirm it was signed by the same pipeline identity.

cosign verify-attestation \
  --type slsaprovenance1 \
  --certificate-identity-regexp '^https://gitlab.acme.internal/acme/.+$' \
  --certificate-oidc-issuer 'https://gitlab.acme.internal' \
  registry.acme.internal/acme/checkout:1.4.2

Verification for registry.acme.internal/acme/checkout:1.4.2 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
{"payloadType":"application/vnd.in-toto+json","payload":"eyJfdHlwZSI6...","signatures":[{"keyid":"","sig":"MEUCIQ..."}]}

What keyless actually means

You may have noticed there was no public key in those commands. That is keyless signing, and it deserves a plain explanation, because it is doing a lot of quiet work. The old way: you generate a signing key, guard it forever, and pray it never leaks. A leaked key is a skeleton key to your whole supply chain. Keyless throws the long-lived key away.

When your pipeline signs an image, it proves its identity to a certificate authority (CA, the trusted party that issues certificates) called Fulcio, part of Sigstore. Think of showing a passport to a notary. Fulcio hands back a certificate good for only a few minutes, stamped with that identity, the GitLab pipeline URL you saw in the Subject field. The signature and that short-lived certificate get recorded in a public, append-only log called Rekor, a notary's ledger anyone can read but nobody can quietly edit. Verification runs the trip in reverse. The certificate chains up to a Fulcio root you trust, the identity on it matches the one you demanded, and the entry is present in Rekor. No key for you to lose, and the signer's identity is baked into the proof.

The policy that enforces it

The by-hand checks prove the evidence is good. The policy makes the cluster run those checks on every pod, automatically, with no human in the loop. Here is a Kyverno ClusterPolicy that demands both a valid signature from your pipeline and a matching provenance attestation before any image from your registry is allowed to run.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: require-signed-provenance
spec:
  validationFailureAction: Enforce     # Enforce = deny (Audit would only warn)
  webhookTimeoutSeconds: 30
  failurePolicy: Fail                   # webhook unreachable = request denied
  rules:
    - name: verify-signature-and-provenance
      match:
        any:
          - resources:
              kinds: [Pod]
      verifyImages:
        - imageReferences:
            - "registry.acme.internal/*"
          required: true                # a matching image MUST be verified, never skipped
          mutateDigest: true            # pin the tag to the verified digest
          verifyDigest: true
          attestors:
            - entries:
                - keyless:
                    issuer: "https://gitlab.acme.internal"
                    subject: "https://gitlab.acme.internal/acme/*"
                    rekor:
                      url: https://rekor.acme.internal
          attestations:
            - type: https://slsa.dev/provenance/v1
              attestors:
                - entries:
                    - keyless:
                        issuer: "https://gitlab.acme.internal"
                        subject: "https://gitlab.acme.internal/acme/*"
              conditions:
                - all:
                    - key: "{{ runDetails.builder.id }}"
                      operator: Equals
                      value: "https://gitlab.acme.internal/acme/checkout//.gitlab-ci.yml@refs/heads/main"

Read it top to bottom. validationFailureAction: Enforce means deny, not warn. failurePolicy: Fail means that if Kyverno itself is unreachable, the API server refuses the pod rather than letting it slide (the trap below). imageReferences scopes the rule to your registry. required: true means a matching image has to actually be verified, not silently skipped. mutateDigest: true rewrites the tag to the exact digest (the sha256 content address) that was verified, so nobody can swap an image's contents behind a tag after the check. The keyless block is the identity gate: only signatures from your GitLab issuer and subject path count. The attestations block demands a SLSA (Supply-chain Levels for Software Artifacts, a framework for rating how trustworthy a build is) provenance document of the right type, and the conditions go one level deeper to check what the provenance says: that the build ran on your builder and nowhere else.

Prove the gate is closed

A policy you cannot see working is a policy you do not have. Apply it and confirm it is live and enforcing.

kubectl apply -f /policies/require-signed-provenance.yaml
kubectl get clusterpolicy require-signed-provenance

clusterpolicy.kyverno.io/require-signed-provenance created
NAME                        ADMISSION   BACKGROUND   READY   AGE   MESSAGE
require-signed-provenance   true        true         True    12s   Ready

Now try to run the attacker's image, the one that failed the by-hand check. Enforce should stop it dead at admission.

kubectl run rogue --image=registry.acme.internal/acme/checkout:1.4.2-patched

Error from server: admission webhook "mutate.kyverno.svc-fail" denied the request:
 
resource Pod/default/rogue was blocked due to the following policies
 
require-signed-provenance:
  verify-signature-and-provenance: |
    failed to verify image registry.acme.internal/acme/checkout:1.4.2-patched:
    .attestors[0].entries[0].keyless: no matching signatures:
    none of the expected identities matched what was in the certificate

The pod object was never created. Now the real image, signed by your pipeline. It should be admitted, and mutateDigest should pin it to its digest so the tag can never be pointed at different bytes later.

kubectl run checkout --image=registry.acme.internal/acme/checkout:1.4.2
kubectl get pod checkout -o jsonpath='{.spec.containers[0].image}'

pod/checkout created
registry.acme.internal/acme/checkout:1.4.2@sha256:9f2a4c...c1

Fail-open is the quiet bypass

There is a failure mode that makes an enforcing policy silently stop enforcing, and attackers know it. Admission webhooks have a setting called failurePolicy. If it is set to Ignore and the policy engine is down (crashed, overloaded, mid-upgrade), the API server shrugs and admits the pod without checking. Your gate is wide open and the dashboard still says Enforce. Setting failurePolicy: Fail means an outage blocks unverified pods instead of nodding them through. The cost: if Kyverno is truly broken, new pods in covered namespaces stop scheduling, which is the safe direction for a security control but needs alerting so you find out in minutes, not days.

Close one more gap while you are here. This rule only governs images whose reference matches registry.acme.internal/*. An image pulled from docker.io is not covered by it at all, so an attacker points at a public image and sidesteps the whole check. Pair this policy with a second rule that denies any registry except yours, otherwise the identity gate guards a door with an open window beside it.

Stage it: Audit before Enforce

Flipping a cluster-wide deny on day one is how these projects get rolled back by lunch. Every third-party image you never signed, every helper pod in a system namespace, every vendor base image, all of it starts failing at once, and the fastest fix under pressure is to delete your policy. Run in Audit first. Same rules, but instead of denying, Kyverno records what it would have blocked and admits with a warning.

kubectl patch clusterpolicy require-signed-provenance \
  --type merge -p '{"spec":{"validationFailureAction":"Audit"}}'
kubectl run rogue --image=registry.acme.internal/acme/checkout:1.4.2-patched

clusterpolicy.kyverno.io/require-signed-provenance patched
Warning: require-signed-provenance: verify-signature-and-provenance: failed to verify image
registry.acme.internal/acme/checkout:1.4.2-patched: .attestors[0].entries[0].keyless: no matching signatures
pod/rogue created

That warning, plus the pod being created anyway, is exactly what Audit is for. Total the failures across the cluster with kubectl get policyreport -A, and each fail is your homework: verify it, sign it, or write a scoped exception for the namespaces and vendor images that genuinely cannot be signed. When the fails hit zero in a namespace, switch that policy back to Enforce and re-run the rogue image to confirm it is denied again. A gate you introduced gradually is a gate that stays up.

Keep one image around that must always fail verification, an unsigned or wrong-identity build, and run kubectl run against it on a schedule. The day that pod is admitted instead of denied is the day your gate came off its hinges, and you want to hear it from your own canary, not from an incident review.

Try this

Run kubectl apply -f /policies/require-signed-provenance.yaml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: fail-open erases your enforcement. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related