>_SecOpsLog_
CoursesKubernetes security & hardeningProtect node metadata & verify binaries

Protect node metadata & verify binaries

Block the cloud metadata endpoint; checksum platform binaries.

Advanced10 min · lesson 5 of 24

In 2019 an attacker walked off with roughly 100 million Capital One records. The whole chain started at one address: 169.254.169.254, the cloud instance metadata service. A server-side request forgery bug (SSRF, the trick of getting a server to make a request on your behalf) let the attacker aim the company's own server at that endpoint and ask it for its cloud credentials. The endpoint answered. No password. No TLS. Plain HTTP. Those credentials unlocked the S3 buckets, and the buckets held the data. Now the part that should keep you up at night: a Kubernetes node exposes that exact same endpoint, and by default every pod running on the node can reach it too. That includes the pod you didn't write, the one pulled from a public image with a vulnerable dependency buried three layers down.

Think of the node as an office with a safe bolted into the corner. The metadata service is a phone on the wall that reads out the safe's combination to anyone who dials it, no questions asked, no log of who called. Get a shell in any pod on that node and you get to pick up the phone. Ask it for iam/security-credentials and back comes the node's cloud role: permission to read buckets, assume other roles, spin up instances, and push further into the account with each one. That last part is what makes it so dangerous. One role often lets you assume another, and the attacker follows that trail deeper until they own the whole account. This is the most common way someone climbs from a single compromised pod up into the cloud itself. It turns up on the CKS exam (Certified Kubernetes Security Specialist) and in real audits because so many clusters leave the phone plugged in. You unplug it with a single NetworkPolicy.

The cluster-to-cloud kill chain
1Pod compromisedRCE, SSRF, or a poisoned…2curl169.254.169.254plain HTTP, no auth3Node IAMcredentialsthe node's role handed back as…4Own the cloudaccountread S3, assume roles, spin up…
The egress policy cuts the arrow between the first two boxes. Everything after that point hangs on one HTTP request reaching the endpoint, so if that request never lands, the rest of the chain has nowhere left to go. You don't have to defend all four steps. You break the one that feeds the others.

A NetworkPolicy behaves like a bouncer with a guest list, not a blacklist. There's no line you can write that says 'deny this one.' A rule only ever names who's allowed through. So to block a single address, you turn the problem inside out: you allow egress to the entire internet (0.0.0.0/0), then carve the metadata IP back out with an except clause. The moment a pod is selected by an egress policy, its outbound traffic flips to default-deny, and from then on only what you list gets out. Apply this to every namespace that runs real workloads, not just the one you're worried about today. And the pods that genuinely do need cloud permissions shouldn't be siphoning them off the node at all. Give each one a scoped, short-lived token through IRSA (IAM Roles for Service Accounts, on AWS) or Workload Identity (on GKE), tied to the pod's own ServiceAccount, so it gets exactly the access it needs and nothing the node happens to be carrying.

deny-metadata.yaml
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: deny-cloud-metadata, namespace: payments }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except:
        - 169.254.169.254/32     # metadata endpoint: carved out
    ports: []                    # empty = all ports to everything else
terminal
# apply the policy to the workload namespace
$ kubectl apply -f deny-metadata.yaml -n payments
networkpolicy.networking.k8s.io/deny-cloud-metadata created
 
# a pod in the DEFAULT namespace (no policy there): the node role still leaks
$ kubectl exec -it web -n default -- curl -s \
    http://169.254.169.254/latest/meta-data/iam/security-credentials/
node-instance-role          # <- an attacker enumerates and assumes this
 
# a pod IN payments, selected by podSelector {}: the request times out
$ kubectl exec -it api -n payments -- curl -m 3 http://169.254.169.254/ ; echo exit=$?
exit=28                     # curl exit 28 = operation timed out

Verify the platform binaries

A tampered kubelet is a full node compromise wearing a trusted name. The same goes for a swapped kubectl or a doctored kubeadm. A download that came over a flaky mirror, a proxy that quietly sat in the middle of the connection, an artifact somebody replaced inside a shared build cache: none of that leaves a mark you can spot by opening the file. So you check its fingerprint instead. Every Kubernetes release publishes a SHA-256 checksum, served to you over TLS. You hash the bytes you actually downloaded and compare them against that published number. A match tells you nothing altered the file between the release server and your disk. Any other result tells you not to run it. This matters most on air-gapped or offline installs, where a binary passes through hands and USB drives and internal mirrors long before it ever reaches a node.

terminal
# download the binary and its published checksum, then compare
$ curl -LO "https://dl.k8s.io/release/v1.31.0/bin/linux/amd64/kubectl"
$ curl -LO "https://dl.k8s.io/release/v1.31.0/bin/linux/amd64/kubectl.sha256"
$ echo "$(cat kubectl.sha256)  kubectl" | sha256sum --check
kubectl: OK
 
# what a swapped binary looks like. this is the whole reason you check
$ echo "$(cat kubectl.sha256)  kubectl.tampered" | sha256sum --check
kubectl.tampered: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

A checksum only proves the bytes match a number posted on the same web page. It says nothing about who produced those bytes. An attacker who can rewrite the download page can rewrite the checksum sitting right beside it just as easily, and now your careful comparison passes against a lie. Signatures close that gap. Since v1.24 the Kubernetes release team signs every artifact with cosign, using keyless signing from Sigstore, which means the signature is bound to the build's own verified identity rather than to some private key sitting on a server. Every signature is also written to a public transparency log that anyone can inspect after the fact. Be clear about what is doing the work here, because it is not that the certificate ships next to the binary: whoever could swap the checksum could swap a .sig and a .cert beside it just as easily. What holds is that the certificate was issued by Fulcio, Sigstore's certificate authority, and chains back to Sigstore's own root, that you name the identity you expect yourself with --certificate-identity instead of believing whatever the downloaded certificate claims to be, and that the transparency log entry lets you confirm all of it without trusting the site you downloaded from.

terminal
# fetch the signature and certificate the release publishes, then verify identity
$ curl -LO "https://dl.k8s.io/release/v1.31.0/bin/linux/amd64/kubectl.sig"
$ curl -LO "https://dl.k8s.io/release/v1.31.0/bin/linux/amd64/kubectl.cert"
$ cosign verify-blob kubectl \
    --signature kubectl.sig \
    --certificate kubectl.cert \
    --certificate-identity [email protected] \
    --certificate-oidc-issuer https://accounts.google.com
Verified OK

AWS ships a second version of that service, IMDSv2 (Instance Metadata Service version 2), and the difference is one extra step. Instead of answering any GET that arrives, it makes the caller send a PUT to /latest/api/token first and carry the token it hands back on every request after that. A server-side request forgery bug that can only make a server fetch a URL cannot issue that PUT, so v2 closes the exact hole that emptied Capital One. It does nothing against a shell in a pod, which can send the PUT itself. That half belongs to the NetworkPolicy, which is why you turn on both.

Binary verification belongs in the bootstrap script, not in a wiki page nobody opens. If someone can swap kubectl or kubelet on disk, every later check is theater. Run both checks before a node's first join: the published digest to catch a mangled download, and cosign verify-blob to catch a rewritten download page.

The same address does the same job on the other clouds, so the single except line above is not an AWS-only trick: Azure Instance Metadata Service and the Google Cloud metadata server both answer on 169.254.169.254 as well. What changes between clouds is the handshake, not the address. Azure only replies to a request carrying a Metadata: true header, and Google only to one carrying Metadata-Flavor: Google, which is enough to stop a plain SSRF bug that cannot set its own headers, though not a shell inside a pod. Two addresses do sit outside the pattern. On GKE the server that hands pods their Workload Identity token listens separately on 169.254.169.252, so leave that one reachable rather than carving it out, and on Alibaba Cloud the metadata service lives at 100.100.100.200, which needs an except entry of its own. So the except list is not copy-paste between clouds. Look up which address your provider answers on before you tell anyone the hole is closed.

Try this

Block pod egress to the link-local metadata IP, then prove in one run that a probe cannot fetch it and that DNS still resolves. Finish by checking the kubectl on disk against its published digest.

terminal
$ kubectl -n payments apply -f - <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: block-metadata }
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
  - to:
    - ipBlock:
        cidr: 0.0.0.0/0
        except: [169.254.169.254/32]
    - namespaceSelector:
        matchLabels: { kubernetes.io/metadata.name: kube-system }
EOF
networkpolicy.networking.k8s.io/block-metadata created
 
$ kubectl -n payments run meta --rm -it --image=curlimages/curl:8.7.1 --restart=Never -- sh -c \
  'curl -s --max-time 2 http://169.254.169.254/latest/meta-data/ || echo METADATA_BLOCKED;
   nslookup kubernetes.default.svc.cluster.local >/dev/null && echo DNS_STILL_WORKS'
METADATA_BLOCKED
DNS_STILL_WORKS
 
$ # also verify platform binaries before you trust a fresh node image
$ # the published digest is per-version, so look up what is on disk first
$ kubectl version --client -o yaml | grep gitVersion
  gitVersion: v1.31.0
$ curl -sLO "https://dl.k8s.io/release/v1.31.0/bin/linux/amd64/kubectl.sha256"
$ echo "$(cat kubectl.sha256)  /usr/bin/kubectl" | sha256sum --check
/usr/bin/kubectl: OK

Takeaway

Cloud metadata is a credential vending machine for the node role. Carve 169.254.169.254 out of an allow-all egress rule in every workload namespace, curl it from a pod to prove the policy actually took, and verify kube binaries with cosign rather than only the digest printed beside them.

Quick check
01You reuse the payments policy on a GKE cluster and add 169.254.169.252/32 to the except list beside 169.254.169.254/32, figuring both are link-local metadata addresses. What breaks?
Incorrect — except takes a list and accepts more than one entry without complaint. The apply prints created either way, which is exactly why you finish with a curl instead of trusting that message.
Incorrect — CoreDNS is an ordinary in-cluster service sitting on the pod network, nowhere near 169.254.0.0/16, so it stays inside the allowed 0.0.0.0/0 range. Proving DNS still resolves is half the point of the probe.
Correct — GKE runs the Workload Identity token server separately on 169.254.169.252. Carve that out and you cut off the scoped, short-lived token meant to replace stealing the node role, so leave it reachable and block only .254.
Incorrect — A NetworkPolicy governs pod traffic. The node's own network namespace is not selected by podSelector, so the kubelet keeps reaching the metadata service whatever you put in except.
02Someone rewrites the download page: a new binary, plus a new kubectl.sha256 that matches it. Your sha256sum --check still prints kubectl: OK. Which check catches the swap?
Incorrect — TLS protects the bytes on the way to you, and the lesson does want that digest fetched over HTTPS. It says nothing about a page rewritten at the source, where the number you received is genuinely the one on offer.
Correct — You state the identity you expect, [email protected], rather than believing whatever the downloaded certificate claims. Fulcio issued that certificate, it chains back to Sigstore's root, and the transparency log lets you confirm all of it afterwards.
Incorrect — Whoever swapped the checksum can drop a .sig and a .cert beside it just as easily. Presence proves nothing; the issuer chain and the identity you supply yourself are what do the work.
Incorrect — Mirrors usually copy from the same upstream page, so both can carry the same rewritten number. Even a match only tells you two files agree, never who produced the bytes.
03Every pod in payments now times out with exit=28 on 169.254.169.254, except one DaemonSet pod that still gets back node-instance-role. What do you check in its spec first?
Incorrect — A plugin that ignores egress rules applies the object cleanly and blocks nothing, but that failure hits every pod in the namespace at once. Your others return exit=28, so enforcement is clearly working here.
Incorrect — podSelector: {} is an empty selector, which picks every pod in the namespace. No label change moves a pod out of its reach.
Incorrect — IRSA and Workload Identity give a pod its own scoped, short-lived token, which is the right long-term fix. Neither one stops that pod from opening a connection to 169.254.169.254.
Correct — That is the giveaway. A hostNetwork pod sits in the node's own network namespace, so pod NetworkPolicy never sees its traffic and it dials the metadata endpoint directly. Back the policy with IMDSv2 and a hop limit of 1 for the pods that do have a network namespace of their own.
Two ways the policy silently does nothing
First, a NetworkPolicy is only enforced if your CNI (Container Network Interface) plugin actually implements egress filtering. Some plugins don't. The object applies cleanly, throws no error, and blocks precisely nothing, which is the whole reason you run the curl test after applying instead of trusting that it took effect. Second, a pod with hostNetwork: true is never selected by the policy at all, because it lives in the node's network namespace rather than its own, and the AWS control you would reach for next does not close that gap either. A metadata hop limit of 1 does not make the endpoint refuse anything: the service stamps that number into the time-to-live of its reply, and an ordinary pod sits one routing hop further out, so the reply expires on the way back and never arrives. A hostNetwork pod has no extra hop to cross, so the reply reaches it intact. Set IMDSv2 and the hop limit anyway, then handle hostNetwork on its own terms by deciding through admission policy who is allowed to ask for it.
Ingress with TLSHardening the API server & kubelet

Related

What Kubernetes isKubernetes fundamentalsCluster architectureKubernetes administrationThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep