Registry trust & allowed sources

Pin who you pull from, not just what.

A warehouse receiving dock runs two different checks on every delivery. One is the seal on the crate: is it intact, and does the paperwork match what you ordered? The other happens at the gate: is this truck run by a carrier on your approved-vendor list, or a van that rolled up off the street? A genuine, unbroken seal on a crate delivered by a company you never signed a contract with is still a delivery you turn away. Container images have exactly these two checks, and most teams only ever build the first one.

Verifying a signature answers one question: is this the exact image our pipeline built and signed? Restricting the registry (the server that stores and serves container images) answers a different one: did this image come from a place we chose to trust? This lesson is about the second check, the gate, the approved-vendor list. In the jargon it is an allowed-sources or allowed-registries policy, and it is the blunt, boring control that a lot of signature-heavy setups quietly skip.

Two locks, two different jobs

A signature, checked properly, tells you two things: the bytes have not changed since it was signed, and the signer is an identity you decided to trust, your CI's key or its keyless identity (Cosign is the common tool for signing container images, and you point it at a specific key or OIDC identity you accept). Pin that identity and a stranger's key is rejected outright, which is the whole point. So why isn't signing the end of the story? Because a real cluster runs a pile of images your pipeline never signed: the upstream base image, the public database you depend on, the ingress controller from some vendor. You cannot demand "signed by our CI" on an image your CI never touched. The only question left for those is where they came from, and that is the question a signature can never answer. Source restriction can.

So for an image your own pipeline builds, the full rule has three parts joined by AND: signed by our CI identity (CI is continuous integration, the automated pipeline that builds and tests your code), AND pulled from our registry, AND carrying the provenance we expect (provenance is a signed record of how and where the image was built). Three separate assertions, because an attacker only needs the one you left out. And here is the quieter trap, even for images you do sign: if your signature policy only asks "is this signed?" instead of "is this signed by us?", a shortcut plenty of teams take, then a stranger's own valid signature sails right through it. Source restriction does not care about that shortcut. A registry you never approved is refused either way.

The runtime will pull from anywhere

Here is the uncomfortable default. The kubelet (the agent on each Kubernetes node that starts containers) and the container runtime under it will pull whatever registry a manifest names, from anywhere on the internet, without asking anyone. There is no built-in allowlist. Change one line in a deployment from registry.acme.internal/api to docker.io/evilcorp/api, and the node fetches the attacker's image on the next rollout as obediently as it fetches yours.

The cheaper version of the same attack does not even need to touch your manifests. It is a typo you make for them. Someone pulls ngnix instead of nginx, or a public namespace that used to be legit gets taken over after the maintainer's account lapses. The name looks right, the pull works, and now a registry you never vetted is running in production. A pull from anywhere is a trust decision, and if you did not make it on purpose, an attacker made it for you.

Pin the source on the host

The first place to draw the line is the host runtime itself, before Kubernetes is even in the picture. On a current Linux that uses systemd (the service manager that boots and supervises everything on modern distributions like Ubuntu 22.04 or Debian 12) running Podman or CRI-O (both are container tools that share the same registry configuration), the file /etc/containers/registries.conf decides which registries are reachable and which are refused outright.

# Bare names like "nginx" resolve here and nowhere else.
unqualified-search-registries = ["registry.acme.internal"]
 
# Public registries are refused on this host. Public dependencies come
# through the proxy project on our own registry, which caches and scans them.
[[registry]]
prefix = "docker.io"
blocked = true
 
[[registry]]
prefix = "quay.io"
blocked = true
 
[[registry]]
prefix = "ghcr.io"
blocked = true

Now a pull straight from Docker Hub does not fail three layers deep with a vague error. It is refused at the door, and the message tells you exactly why and which file made the call. Verify the change worked by trying a pull that should now be dead:

podman pull docker.io/library/nginx:1.27

Trying to pull docker.io/library/nginx:1.27...
Error: initializing source docker://nginx:1.27: registry docker.io is blocked in /etc/containers/registries.conf or /etc/containers/registries.conf.d/*.conf

That failure is the point. On a stock box that pull would have downloaded happily. The fact that it now stops, and names the config file that stopped it, is your proof the block is live.

Pin the source at the cluster gate

Host config is per-node and easy to drift. The stronger, cluster-wide place to enforce the allowlist is admission control: the checkpoint the Kubernetes API server runs on every create-and-update request before it is ever saved. Think of it as the guard who reads the guest list at the door instead of trusting each room to check for itself. Kyverno (a policy engine for Kubernetes that validates or mutates resources as they are admitted) is one common way to write that rule. OPA Gatekeeper (Gatekeeper is a Kubernetes admission controller built on Open Policy Agent, a general-purpose policy engine) and the built-in ValidatingAdmissionPolicy (VAP for short), which uses CEL, the Common Expression Language, do the same job.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
spec:
  background: true
  rules:
    - name: registries-allowlist
      match:
        any:
          - resources:
              kinds: [Pod]
      validate:
        failureAction: Enforce
        message: >-
          Images may only be pulled from registry.acme.internal.
        pattern:
          spec:
            # =(...) is a conditional anchor: check it only if it exists.
            =(ephemeralContainers):
              - image: "registry.acme.internal/*"
            =(initContainers):
              - image: "registry.acme.internal/*"
            containers:
              - image: "registry.acme.internal/*"

Notice that the rule covers initContainers and ephemeralContainers, not only the main containers. That is deliberate. A policy that checks only containers leaves an obvious side door: an attacker uses an initContainer or an injected debug (ephemeral) container from a foreign registry, and your allowlist waves it through. Cover all three.

kubectl apply -f restrict-image-registries.yaml

clusterpolicy.kyverno.io/restrict-image-registries created

With the policy enforcing, a pod that names a disallowed registry never gets created. The rejection comes back to whoever tried it, and it lands in the API server's audit log where a defender can see the attempt:

kubectl run probe --image=docker.io/library/nginx:1.27

Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
 
resource Pod/default/probe was blocked due to the following policies
 
restrict-image-registries:
  registries-allowlist: 'validation error: Images may only be pulled from
    registry.acme.internal. rule registries-allowlist failed at path
    /spec/containers/0/image/'

Stopping new bad pods is half the job. The other half is finding the ones already running from registries nobody approved, which is why the policy sets background: true. Kyverno scans the workloads already on the cluster and writes the result for each one into its own PolicyReport, named after that resource. Any report with a number in the FAIL column is a pod on your fix list:

kubectl get policyreport -A

NAMESPACE   NAME                                   PASS   FAIL   WARN   ERROR   SKIP   AGE
default     e8f3a1b2-4c7d-4a9e-9f10-2b6c1d0a7e44   1      0      0      0       0      2d
default     a1c2e3f4-5b6d-4c7e-8f90-1a2b3c4d5e6f   1      0      0      0       0      2d
legacy      3c9d7e4f-8a21-4b06-bd3e-9f2a4c115d8b   0      1      0      0       0      2d
legacy      7b1a2c3d-4e5f-4a6b-8c9d-0e1f2a3b4c5d   0      1      0      0       0      2d
legacy      9f2a4c11-5d8b-4e06-9d3e-3c1d7e4f8a21   0      1      0      0       0      2d
legacy      1e3f5a7b-9c1d-4a6b-8c0d-5a7b9c1d3e2f   0      1      0      0       0      2d

Every row is one workload. In default, each report comes back a clean PASS. In legacy, four of them show a 1 in the FAIL column. Those are four pods running right now, pulling from a registry that is not on your list. That is the inventory you want before an incident, not during one. Open each failing report to get the pod name, then decide, one at a time, whether it moves to your registry or gets switched off.

Pin the digest, not the tag

Locking the registry down still leaves one soft spot. A tag like :1.4.2 is a sticky note, not a fingerprint. Whoever can push to that repository can repoint the tag at different bytes tomorrow, and your allowlist, which only cares about the hostname, is perfectly happy. The fix is to pin the digest: the sha256 content hash that names the exact bytes and changes if a single one does. Grab it with a tool like crane:

crane digest registry.acme.internal/api:1.4.2

sha256:9b1ede9c6a3d2f77a0b3c4e5f6071829abf4c1d2e3a4b5c6d7e8f9a0b1c2d3e4

Then reference it in the manifest as registry.acme.internal/api@sha256:9b1ede9c... instead of the tag. Now the source is pinned and so are the bytes. An attacker who repoints the tag changes nothing you actually run, because your workloads ask for the hash.

Make the registry itself the choke point

Your internal registry is not a passive shelf you point at. It is infrastructure to run and to defend, and it can enforce policy of its own. Stand up a private registry (Harbor and Artifactory are the usual choices) as the single source for production images. Configure a proxy project, also called a pull-through cache, so the public dependencies you genuinely need flow through your registry, get scanned and cached there, and an upstream outage or a poisoned public image never reaches a node unfiltered.

On top of that, make the registry gate promotion. Scan on push and refuse to serve images with CRITICAL vulnerabilities. Require a valid signature before an image can be tagged for the prod repository. Now the same rule, approved sources only, is enforced in three places that back each other up: the host refuses foreign registries, the cluster gate denies them at admission, and the registry itself only hands out images that were scanned, signed, and promoted on purpose.

So write the allowlist down as a list of hostnames you can say out loud. Every entry is a supplier you chose, for a reason you can name. If a registry is on the list and nobody can explain why it's there, that is the one to pull off it, before an attacker notices it first.

Try this

Run podman pull docker.io/library/nginx:1.27 on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: an allowlist that's too tight takes the cluster down. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related