CoursesAdvanced cloud securityContainer & serverless workload security

Container & serverless workload security

Least-privilege roles, runtime controls, and image provenance.

Advanced35 min · lesson 14 of 15

A running container is a room inside a building you own. Three things decide how bad it gets when someone picks the lock: the keycard the occupant is carrying, what you let through the door in the first place, and whether a guard is watching once they are inside. Translate that to a workload. The keycard is its cloud identity (the IAM role it can assume, where IAM means identity and access management). The door is the image (the exact code and libraries you shipped). The guard is the runtime (what the process may do on the host once it runs). A container or function that gets compromised becomes whatever its role allows, reaches whatever the network permits, and runs whatever made it into the image. Hardening a workload means shrinking all three at once, so a foothold lands somewhere with almost nothing to stand on.

The blast radius has three planes

Blast radius is the set of resources a compromised workload can touch before anything stops it. It is the sum of three planes. The identity plane is the role the code assumes. The supply-chain plane is the code and dependencies baked into the image. The runtime plane is what the process can do on the host once it starts. An attacker walks along whichever plane you left widest, so the widest one is the only one that matters. Here is the part people get wrong: most real container break-ins are not exotic kernel bugs. They are an ordinary application flaw that reaches an over-scoped role, or a poisoned dependency that sailed into production because nothing checked the image at the door. Treat each plane as its own control you can tighten without touching the other two. The rest of this lesson does exactly that on AWS, Google Cloud and Azure side by side, because the idea is identical on all three and only the spelling changes.

Per-workload identity, not node identity

The oldest mistake is handing the whole node a broad role and letting every pod borrow it. The node is the virtual machine your pods run on (an EC2 instance, short for Elastic Compute Cloud, in Amazon EKS, Elastic Kubernetes Service; a node in Google Kubernetes Engine, GKE; a scale-set instance in Azure Kubernetes Service, AKS). Give that machine one powerful role and every pod on it can reach the on-box metadata endpoint and grab those credentials. It is a building where every room opens with the same master key: pick one lock and you hold them all. The fix is a separate keycard per workload. Each Kubernetes ServiceAccount (the pod's in-cluster identity) is exchanged for a short-lived cloud credential scoped to exactly one job. AWS spells this two ways, and the difference matters. The older one, IRSA (IAM Roles for Service Accounts), leans on the cluster's public OIDC issuer (OpenID Connect, a standard for proving identity with short-lived signed tokens): the pod presents a signed token, AWS STS (Security Token Service, its short-lived-credential vending machine) validates it against that issuer, and returns credentials. The newer one, EKS Pod Identity, drops OIDC entirely; an agent on each node talks to the EKS Auth service, and the role's trust policy names the principal pods.eks.amazonaws.com. GKE calls its version Workload Identity Federation and AKS calls its Workload Identity, and both run the OIDC dance against the cluster's issuer. No static key ever touches the pod on any of them.

terminal
# AWS - EKS Pod Identity: map ONE ServiceAccount to ONE role (successor to IRSA)
aws eks create-pod-identity-association \
--cluster-name prod --namespace reporting \
--service-account report-svc \
--role-arn arn:aws:iam::222222222222:role/report-svc
output
{
"association": {
"clusterName": "prod",
"namespace": "reporting",
"serviceAccount": "report-svc",
"roleArn": "arn:aws:iam::222222222222:role/report-svc",
"associationId": "a-1a2b3c4d5e6f7g8h9",
"associationArn": "arn:aws:eks:us-east-1:222222222222:podidentityassociation/prod/a-1a2b3c4d5e6f7g8h9",
"createdAt": "2026-07-22T10:14:07.521000+00:00",
"modifiedAt": "2026-07-22T10:14:07.521000+00:00"
}
}
# NOTE: the role's trust policy must allow the service principal pods.eks.amazonaws.com
terminal
# GCP - Workload Identity Federation for GKE: bind the K8s SA to a Google SA, then annotate it
gcloud iam service-accounts add-iam-policy-binding \
--role roles/iam.workloadIdentityUser \
--member "serviceAccount:my-project.svc.id.goog[reporting/report-svc]"
kubectl annotate serviceaccount report-svc --namespace reporting \
iam.gke.io/gcp-service-account=report-svc@my-project.iam.gserviceaccount.com
output
Updated IAM policy for serviceAccount [[email protected]].
bindings:
- members:
- serviceAccount:my-project.svc.id.goog[reporting/report-svc]
role: roles/iam.workloadIdentityUser
etag: BwYX9k2mF3o=
version: 1
serviceaccount/report-svc annotated
# (Newer path: skip the Google SA and grant roles straight to the principal://... identity.)
terminal
# Azure - AKS Workload Identity: federate the K8s SA subject to a managed identity
az identity federated-credential create \
--name report-svc --identity-name report-mi --resource-group rg-prod \
--issuer "$(az aks show -g rg-prod -n prod --query oidcIssuerProfile.issuerUrl -o tsv)" \
--subject system:serviceaccount:reporting:report-svc \
--audiences api://AzureADTokenExchange
output
{
"audiences": [
"api://AzureADTokenExchange"
],
"id": "/subscriptions/<sub>/resourceGroups/rg-prod/providers/Microsoft.ManagedIdentity/userAssignedIdentities/report-mi/federatedIdentityCredentials/report-svc",
"issuer": "https://eastus.oic.prod-aks.azure.com/<tenant>/<guid>/",
"name": "report-svc",
"subject": "system:serviceaccount:reporting:report-svc"
}

Every one of those tokens and credentials is short-lived and rotated for you, so a leaked one goes stale within hours and is useless anywhere except that cloud's own token service. The trade-off is real and worth saying out loud: you now manage one role per workload instead of one role per node. More objects to track. But that is precisely the granularity you want, because scoping down or revoking one workload never touches another. AWS lets you attach up to 5,000 Pod Identity associations per cluster (a fixed limit, not one you can raise on request), so the ceiling is rarely your problem; discipline is. A single shared 'app' role that ten services quietly reuse is how least privilege rots without anyone noticing. Wire the mapping into your Helm chart or Terraform module so every new service is born with its own role, and a missing role fails the deploy instead of falling back to something broad.

Scoped identity is theatre if the node's metadata endpoint is still reachable
Per-workload identity only governs your application's credentials. It does nothing about the node's own role. If a compromised pod can still reach the metadata endpoint (IMDS, the Instance Metadata Service, the link-local address every cloud VM exposes at 169.254.169.254), it can ask for the node's credentials directly and walk around everything you set up. Close that door on every cloud. On EKS, set the metadata hop limit to 1 so packets from ordinary pods, which sit one network hop further out, cannot reach it: aws ec2 modify-instance-metadata-options --instance-id i-0abc --http-put-response-hop-limit 1 --http-tokens required. One gap to know: a pod running with hostNetwork: true shares the node's own network stack and reaches IMDS regardless, so keep untrusted workloads off host networking. On GKE, enabling Workload Identity is the fix, because the GKE metadata server hands pods their scoped identity and hides the node's raw metadata. On AKS, block egress to 169.254.169.254 with a NetworkPolicy, since workload identity does not need it. And never leave a wide role on the node to begin with.

Provenance: run only what you built

Provenance is the paper trail that answers one question: did this exact image come out of my pipeline, unchanged? Two controls together earn you a yes you can trust. Scanning rejects images carrying known, fixable flaws before they ship. Signing plus admission enforcement makes sure only images your pipeline built and signed can ever start. First rule, and it is the one people skip: pin images by their digest (the cryptographic hash of the exact bytes, written sha256:...), never a moving tag like :latest. A tag is a sticky note on a box; anyone can peel it off and press it onto a different box. Pin the box by its contents and a swap becomes impossible. Between the moment you scan a tag and the moment you deploy it, an attacker who can push to your registry can repoint that tag at a poisoned layer, so you would scan one image and run another.

terminal
# Fail the build on fixable HIGH/CRITICAL vulnerabilities (any registry, any cloud)
trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 \
registry.internal/report@sha256:9f2c4b1e...
output
2026-07-22T10:15:02Z INFO [vuln] Vulnerability scanning is enabled
2026-07-22T10:15:03Z INFO Detected OS family="debian" version="12.5"
2026-07-22T10:15:03Z INFO [debian] Detecting vulnerabilities... pkg_num=142
report@sha256:9f2c4b1e... (debian 12.5)
=======================================
Total: 1 (HIGH: 1, CRITICAL: 0)
┌─────────┬───────────────┬──────────┬────────┬───────────────────┬──────────────────┬───────────────────┐
│ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │
├─────────┼───────────────┼──────────┼────────┼───────────────────┼──────────────────┼───────────────────┤
│ libssl3 │ CVE-2024-6119 │ HIGH │ fixed │ 3.0.14-1~deb12u1 │ 3.0.14-1~deb12u2 │ openssl: DoS in │
│ │ │ │ │ │ │ X.509 name checks │
└─────────┴───────────────┴──────────┴────────┴───────────────────┴──────────────────┴───────────────────┘
exit status 1
terminal
# Keyless sign: no private key exists to steal; the signature binds to your CI (continuous integration) pipeline's OIDC identity (cosign v2)
cosign sign --yes registry.internal/report@sha256:9f2c4b1e...
output
Generating ephemeral keys...
Retrieving signed certificate from Fulcio...
Successfully verified SCT...
tlog entry created with index: 148372913
Pushing signature to: registry.internal/report
terminal
# Verify at the door: reject anything not signed by OUR pipeline identity
cosign verify \
--certificate-identity=https://github.com/secops/ci/.github/workflows/build.yml@refs/heads/main \
--certificate-oidc-issuer=https://token.actions.githubusercontent.com \
registry.internal/report@sha256:9f2c4b1e...
output
Verification for registry.internal/report@sha256:9f2c4b1e... --
The following checks were performed on each of these signatures:
- The cosign claims were validated
- Existence of the claims in the transparency log was verified offline
- The code-signing certificate was verified using trusted certificate authority certificates

Signing is worthless if nothing checks the signature. That check belongs at admission, the instant the cluster decides whether to let a pod run. The clean part: the signing side is identical everywhere, because cosign (from the Sigstore project) produces the same signature whether the image lands on AWS, Google Cloud or Azure. That signature is backed by Sigstore's certificate authority (Fulcio) and recorded in its public transparency log (Rekor), the tlog entry printed above. It can also carry an attestation: an SBOM (software bill of materials, the itemised list of everything inside the image) and SLSA provenance (Supply-chain Levels for Software Artifacts, a signed record of how and where the image was built), so the cluster can check who signed the image and, from the attestation, how it was built. The enforcement side has choices. A policy engine like Kyverno or the Sigstore policy-controller runs on any Kubernetes cluster and blocks pods whose images fail verification, so one policy file guards EKS, GKE and AKS unchanged. Each cloud also ships a native option, and they are not at the same maturity. Google's Binary Authorization is generally available and can hard-deny a deploy that lacks a required attestation. Azure's AKS Image Integrity (built on Ratify plus Azure Policy) is still in preview and today supports only the Audit effect, meaning it records a violation but will not block it, so if you need AKS to actually refuse an unsigned image right now, run Kyverno or policy-controller there too. AWS has no built-in admission signature check at all, which makes the agnostic policy engine the answer on EKS. Whatever you pick, gate the registry the same way, so an unsigned image cannot even be pulled.

verify-images.yaml
# verify-images.yaml - the SAME policy runs unchanged on EKS, GKE and AKS (Kyverno)
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: require-signed-images
spec:
validationFailureAction: Enforce # Enforce = block the pod; Audit = log only
rules:
- name: verify-report-svc
match:
any:
- resources:
kinds: [Pod]
namespaces: [reporting]
verifyImages:
- imageReferences:
- "registry.internal/report*"
required: true # a matching image MUST carry a valid signature
mutateDigest: true # rewrite the tag to the verified digest
attestors:
- entries:
- keyless:
issuer: "https://token.actions.githubusercontent.com"
subject: "https://github.com/secops/ci/.github/workflows/build.yml@refs/heads/main"
rekor:
url: "https://rekor.sigstore.dev"

Harden and watch the runtime

A signed image with a tight role should still run in the smallest possible room. Strip the runtime to bare walls. Drop every Linux capability (the fine-grained superpowers a process can hold, like binding low ports or loading kernel modules). Run as a non-root user. Mount the root filesystem read-only. Set a seccomp profile (secure computing mode, a kernel filter that whitelists which system calls a process may make) so the kernel refuses anything the app never needed. Each of these turns a code-execution bug into a dead end. No writable filesystem to drop a payload onto. No privilege to escalate to root. No capability left to open a raw socket. The manifest below is the shape you want, and the same securityContext works on every managed Kubernetes, because it is plain Kubernetes rather than a cloud feature.

deployment.yaml
# deployment.yaml - the room a foothold lands in: non-root, read-only, zero capabilities
apiVersion: apps/v1
kind: Deployment
metadata: { name: report-svc, namespace: reporting }
spec:
replicas: 2
selector: { matchLabels: { app: report-svc } }
template:
metadata: { labels: { app: report-svc } }
spec:
serviceAccountName: report-svc # this workload's scoped identity
automountServiceAccountToken: false # drop the kube-API token; cloud identity uses a separate audience-scoped token
securityContext:
runAsNonRoot: true
runAsUser: 10001
seccompProfile: { type: RuntimeDefault }
containers:
- name: app
image: registry.internal/report@sha256:9f2c4b1e... # pinned by digest
securityContext:
readOnlyRootFilesystem: true
allowPrivilegeEscalation: false
capabilities: { drop: ["ALL"] }
volumeMounts:
- { name: tmp, mountPath: /tmp } # writable scratch, without a writable /
volumes:
- name: tmp
emptyDir: {}

Static hardening cannot see behaviour. It locks doors; it does not notice someone already inside doing something they should not. That is the job of runtime behavioural detection, a sensor watching the live process for moves that only appear after an exploit lands: a shell spawning inside a container that has never once run a shell, an outbound connection to an address the app has never called, a binary executing that was never in the image. Falco is the open-source standard and reads kernel events through eBPF (extended Berkeley Packet Filter, a safe way to run tiny observer programs inside the Linux kernel). The managed equivalents feed the same idea into each cloud's alert pipeline: Amazon GuardDuty Runtime Monitoring, Google Cloud Container Threat Detection (part of Security Command Center), and Microsoft Defender for Containers. Run one per cluster and route it where a human will actually see the alert. A detection nobody reads is decoration.

Read-only root breaks apps that expected to write
readOnlyRootFilesystem: true will crash any app that writes to disk on startup (temp files, caches, pid files) with a 'read-only file system' error, and it looks like the image broke when the hardening is what changed. The fix is not to remove the flag. Find every path the app writes to and mount a small emptyDir volume there, as with /tmp above, or a writable cache directory. You keep the hardening and the app keeps running. The same discipline applies to verification: cosign verify and Kyverno must check the image by its digest, because a signature is bound to specific bytes. Verify a tag and you have verified nothing that lasts.

Serverless: the execution role is the perimeter

A function has no host you can harden and no long-running process you can watch. It appears, runs for a few hundred milliseconds, and vanishes. So its blast radius collapses almost entirely onto three things: its execution role, its dependencies, and its triggers. An over-broad role on a function is the whole game. One vulnerable package, or one crafted event, becomes everything that role can reach. Three levers, then. Scope the execution role to the exact resources the function touches, with no wildcards. Pin and scan dependencies, because with no host to break into, the packages you import are the attack surface. And restrict which triggers may invoke it, so a stranger cannot fire it at all. The pattern holds across clouds: AWS Lambda gets a scoped execution role, Google Cloud Run and Cloud Functions run as their own per-service service account, and Azure Functions use a managed identity (an identity Azure creates and rotates for the resource, with no secret for you to store or leak). Never a shared, project-wide account behind all of them.

terminal
# AWS - scope a Lambda execution role to exactly one bucket prefix, no wildcards
aws iam put-role-policy --role-name report-fn --policy-name s3-read \
--policy-document '{"Version":"2012-10-17","Statement":[{"Effect":"Allow",
"Action":"s3:GetObject","Resource":"arn:aws:s3:::reports-prod/inbound/*"}]}'
# then scan the function's dependencies before publishing
npm audit --audit-level=high
output
# put-role-policy returns nothing on success
# npm audit:
found 0 vulnerabilities
terminal
# GCP - Cloud Run runs as its OWN service account, digest-pinned, no public invoker
gcloud run deploy report-fn \
--image us-central1-docker.pkg.dev/my-project/apps/report@sha256:9f2c4b1e... \
--service-account [email protected] \
--no-allow-unauthenticated
output
Deploying container to Cloud Run service [report-fn] in project [my-project] region [us-central1]
✓ Deploying new service... Done.
✓ Creating Revision...
✓ Routing traffic...
Done.
Service [report-fn] revision [report-fn-00007-abc] has been deployed
and is serving 100 percent of traffic.
Service URL: https://report-fn-abcdefghij-uc.a.run.app
terminal
# Azure - give the Function's managed identity a tightly scoped role (not a shared one)
az role assignment create \
--assignee-object-id "$(az functionapp identity show -g rg-prod -n report-fn --query principalId -o tsv)" \
--assignee-principal-type ServicePrincipal \
--role "Storage Blob Data Reader" \
--scope "/subscriptions/<sub>/resourceGroups/rg-prod/providers/Microsoft.Storage/storageAccounts/reportsprod"
output
{
"condition": null,
"principalId": "8f3c9a1e-6b2d-4a77-9c1e-2f0d7b3e5a10",
"principalType": "ServicePrincipal",
"roleDefinitionId": "/subscriptions/<sub>/providers/Microsoft.Authorization/roleDefinitions/2a2b9908-6ea1-4ae2-8e65-a410df84e7d1",
"roleDefinitionName": "Storage Blob Data Reader",
"scope": "/subscriptions/<sub>/resourceGroups/rg-prod/providers/Microsoft.Storage/storageAccounts/reportsprod"
}

Look at the two invoke guards in those commands. --no-allow-unauthenticated on Cloud Run means no anonymous caller reaches the service at all; only identities you grant the run.invoker role can fire it. On Lambda the equivalent is a tight resource policy plus event-source permissions, so only the specific queue, bucket, or API Gateway you intend can trigger the function, not the open internet. Lock the trigger and you shrink the set of things that can even start the code. Leave it open and the tightest role in the world still runs on an attacker's schedule.

The three planes of workload blast radius, and the serverless variant
Identity plane
Per-workload role
one ServiceAccount to one role, never the node role
Short-lived credentials
OIDC-federated or agent-vended, rotated, no static keys
Block the node metadata endpoint
stop pods stealing the broad node credentials
Supply-chain plane
Scan
reject fixable HIGH/CRITICAL before ship
Sign (cosign keyless)
bind the image to CI's OIDC identity
Verify at admission
digest-pinned and signed, or no schedule
Runtime plane
Hardened context
non-root, read-only fs, drop ALL caps, seccomp
Behavioural detection
Falco / GuardDuty / Container Threat Detection / Defender
Serverless variant (no host)
Execution role = perimeter
scope to the exact resources used
Dependencies + triggers
pin and scan packages, restrict invokers
Tighten each plane on its own; an attacker only needs the widest one.

Set all three planes on one service and you have a template. Set them on three hundred services across three clouds by hand and you have a spreadsheet that drifts the first busy week. The scoped role, the signature policy, and the hardened baseline only hold when the platform stamps them onto every new account and workload automatically, from the moment it is created. That is the work of guardrails as code and landing zones, and it is where the next lesson goes.

Quick check
01Your CI scans registry.internal/report:latest, it passes clean, and your deploy references that same :latest tag. Why is pinning to the sha256 digest still the safer choice?
Incorrect — a digest is about identity, not speed, and pull time has nothing to do with tampering.
Incorrect — tags and digests both work on any registry; portability is not the concern here.
Correct — pinning by digest binds you to the exact bytes you scanned and closes that swap window.
Incorrect — a digest is a hash, not a signature; you still need signing and verification on top.
02You add readOnlyRootFilesystem: true to a container and it now crashes on startup with 'read-only file system' while writing to /tmp. What is the right fix?
Correct — you keep the hardening and give the app exactly the writable path it needs.
Incorrect — that throws away the control and hands a foothold a writable filesystem to drop a binary onto.
Incorrect — the block is the read-only mount, not the user; running as root changes nothing and weakens you.
Incorrect — that restores a dangerous capability and defeats the point of dropping them all.
03You enabled EKS Pod Identity so each pod holds its own scoped role, yet a compromised pod is reading an S3 bucket its role does not allow. What is the most likely cause?
Incorrect — digest pinning governs which code runs, not which IAM permissions a running pod can reach.
Incorrect — admission verification decides whether an image may start, not what a running pod's identity can access.
Incorrect — seccomp filters syscalls to the kernel and has no bearing on cloud IAM reach.
Correct — per-workload identity does not remove the node role, and an un-blocked IMDS lets a foothold request it directly.

Try this

Run cosign sign --yes registry.internal/report@sha256:9f2c4b1e... on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: scoped identity is theatre if the node's metadata endpoint is still reachable. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related