Lateral movement

The pivot chain and default-deny east-west.

By default, every pod in a Kubernetes cluster can open a network connection to every other pod, in every namespace. No firewall, no questions asked. That one fact is what turns a single hacked container into a cluster-wide problem. Most real cluster breaches you read about don't end where they started. The first foothold is almost never the prize, it's the doorway. There are two ways an attacker travels once they're inside. Privilege escalation, which has its own lesson, is the climb upward toward more power. Lateral movement is the sideways kind: the same level of access you already have, used to reach machine after machine and service after service until one of them holds something worth stealing. Think of a hotel where every room has a connecting door to the next, and every one of those doors is unlocked. Get into a single room and you can walk the whole floor. Your job as a defender is to lock those doors, and to notice when someone starts trying the handles.

The flat network is one open floor

Kubernetes ships this way on purpose. A brand-new cluster runs a flat network so services can find each other without anyone writing firewall rules first, which is exactly what makes demos and getting-started guides painless. The cost shows up later, in production. An attacker who lands a shell in a low-value pod, say a public web frontend running an image with a known vulnerability, inherits that pod's full network reach. From inside it they can query the cluster's internal DNS (Domain Name System, the phone book that turns service names like payments-api into the numeric addresses machines actually use), resolve the names of services they were never meant to see, and connect straight to databases, payment services, and internal admin panels that were built to trust any caller coming from inside the cluster. Nothing about the frontend being 'just the frontend' slows that down at all.

# Attacker has code execution in the frontend pod and opens a shell.
$ kubectl exec -it frontend-6d9c8f7b4-2xk9p -n prod -- sh
 
# From inside the pod, they reach the internal payments API directly.
/app # curl -s http://payments-api.payments.svc.cluster.local:8080/v1/accounts/balance
{"account":"acc_8842","currency":"USD","balance":19230.55}

The connecting door is the network. The keycard is the pod's service-account token. A service account (SA) is the identity a pod uses when it talks to the Kubernetes API server. API stands for Application Programming Interface, but you can picture that server as the cluster's control switchboard, the one point every command has to pass through. Unless you tell it not to, Kubernetes mounts a token file for that identity inside every pod, always at the same fixed path. The token itself is a signed JSON Web Token (JWT), a tamper-proof string that proves who you are. An attacker sitting in the pod just reads the file and starts making API calls as that identity, no password required. Modern clusters make that token short-lived and tied to the specific pod, which helps, but short-lived still means valid right now, in the attacker's hand, for exactly the identity they've landed on. And if the SA was handed more permission than it actually uses, which a surprising number are, that token stops being a single room key and becomes a skeleton key into other namespaces.

# Read the token Kubernetes auto-mounted into the pod.
/app # TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
/app # API=https://kubernetes.default.svc
 
# Use it to list pods in a namespace the frontend has no business reading.
/app # curl -sk -H "Authorization: Bearer $TOKEN" \
        $API/api/v1/namespaces/staging/pods | jq -r '.items[].metadata.name'
staging-db-0
staging-db-1
payroll-worker-77c9f4b8d6-ktz2p

Lock the door: default-deny, then allow what's real

A NetworkPolicy is an allow-list, and its rules only ever add permission, never subtract it. So locking down a namespace takes two steps. First you apply a policy that selects every pod in the namespace and permits nothing, which flips that namespace from open to closed. Then you add narrow policies that re-open exactly the flows a real workload needs, and nothing beyond them. One detail trips people up over and over: policies are directional, and a policy only protects the pods it selects. To stop the frontend from reaching payments, you don't lock the frontend's outbound door, you lock the payments namespace's inbound door. The token door gets its own separate lock, over in RBAC (Role-Based Access Control, the rulebook that decides which identity is allowed to do what). Scope each service account down to what it genuinely needs, and set automountServiceAccountToken: false on any pod that never calls the API, so there's no keycard sitting there to steal in the first place. And whichever door you lock, test it the way an attacker would: run the connection yourself and confirm it dies.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: payments
spec:
  podSelector: {}            # selects every pod in the namespace
  policyTypes: ["Ingress"]   # deny all inbound; no rules below means nothing allowed
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-gateway-to-payments
  namespace: payments
spec:
  podSelector:
    matchLabels: { app: payments-api }
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: api-gateway }
          podSelector:
            matchLabels: { app: gateway }
      ports:
        - { protocol: TCP, port: 8080 }

$ kubectl apply -f payments-netpol.yaml
networkpolicy.networking.k8s.io/default-deny-ingress created
networkpolicy.networking.k8s.io/allow-gateway-to-payments created
 
# Network pivot, retried from the compromised frontend pod:
$ kubectl exec -n prod frontend-6d9c8f7b4-2xk9p -- \
    curl -s --max-time 5 http://payments-api.payments.svc.cluster.local:8080/v1/accounts/balance
curl: (28) Failed to connect to payments-api.payments.svc.cluster.local port 8080 after 5001 ms: Timeout was reached
 
# Token pivot, after scoping the frontend SA's RBAC to its own namespace:
$ curl -sk -H "Authorization: Bearer $TOKEN" $API/api/v1/namespaces/staging/pods
pods is forbidden: User "system:serviceaccount:prod:frontend"
  cannot list resource "pods" in API group "" in the namespace "staging"

Seeing the handle turn, not just holding the door

Blocking the pivot is only half the job. You also want to know an attempt happened at all, because an attacker who gets one door slammed shut just walks down the hall and tries the next one. Two signals cover this sideways, pod-to-pod traffic, which network people call east-west (as opposed to north-south, the traffic heading in and out of the cluster from the outside world). On the network side, a CNI that actually enforces policy can also log every packet it drops. Cilium's Hubble is the clearest example: it hands you the exact flow it denied, with the source pod, the destination service, and the verdict, all on one line. On the identity side, that token pivot leaves its own fingerprints in the API server's audit log. The frontend service account suddenly asking to list pods in staging, a namespace it has never once touched, is about as loud an anomaly as you will find. Neither signal alone tells the whole story, so you watch the network side and the identity side together. The next lesson turns both of them into real detections you can alert on.

$ hubble observe --namespace payments --verdict DROPPED --last 2
Jul 16 10:42:19.331: prod/frontend-6d9c8f7b4-2xk9p:47122 <> payments/payments-api-5f8c9d7b6-q4m2n:8080 Policy denied DROPPED (TCP Flags: SYN)
Jul 16 10:42:24.402: prod/frontend-6d9c8f7b4-2xk9p:47140 <> payments/payments-api-5f8c9d7b6-q4m2n:8080 Policy denied DROPPED (TCP Flags: SYN)

Try this

Work through “Seeing the handle turn, not just holding the door” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a NetworkPolicy your CNI ignores is a green light, not a wall. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related