>_SecOpsLog_
CoursesKubernetes administrationThe cluster network model

The cluster network model

The flat-network rules every CNI must honor.

Advanced10 min · lesson 31 of 65
In plain terms
Every pod gets its own phone number and can call any other pod directly — no switchboard, no number translation. It’s one big flat phone network across the whole cluster.

A Pod (the smallest thing Kubernetes runs, usually one container, sometimes a couple bundled together that share an address) gets its own IP address, and it can reach any other Pod in the cluster by that IP directly. Two Pods sitting on two different physical machines talk to each other the same way two laptops on the same office Wi-Fi do. No gateway in the middle rewriting addresses. No port-forwarding table anyone has to keep in sync. That one property is the ground floor everything else stands on, and most of what looks baffling about Services and DNS (the cluster's phone book, which turns names into addresses) gets simple the moment you trust it.

The four rules every plugin has to honor

Think of an old office where every desk has its own direct phone extension and anyone can dial anyone else straight through. No operator, no switchboard translating numbers in the middle. Kubernetes insists on that exact shape of network, and it comes down to four properties any plugin has to satisfy.

Every Pod gets an IP that's unique across the whole cluster, not just unique on its own machine. Any Pod can reach any other Pod at that IP with no NAT (Network Address Translation, the address-swapping your home router does so a dozen devices can share one public address). Between Pods, none of that happens. The IP a Pod sees on its own network card is the exact IP everyone else uses to reach it. And on each machine, the small agents running there like the kubelet (the per-node agent that starts your containers and keeps an eye on them) can reach the Pods on that same machine. That last rule is deliberately narrow: it covers a node reaching its own Pods, not every Pod in the cluster. Most plugins do let a node talk to Pods on other machines as well, but that's a habit of the plugins rather than something the model promises.

The thing that actually makes all this true is the CNI, the Container Network Interface: a plugin like Calico, Cilium, or Flannel. The model doesn't care how the plugin pulls it off, only that those four properties hold (that's the next lesson). What matters right here is the shape they produce. One flat network. Every Pod a single hop from every other Pod, no matter which machine it happened to land on.

list pods with their IP and node
kubectl get pods -o wide
output
NAME       READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
web-6d4f   1/1     Running   0          12m   10.244.1.7   node-a   <none>           <none>
web-8b21   1/1     Running   0          12m   10.244.2.4   node-b   <none>           <none>

Where those IPs come from

The two Pods above landed in different ranges, 10.244.1.x and 10.244.2.x, and that's on purpose. A cluster is set up with one big block of addresses set aside for Pods, the cluster CIDR (Classless Inter-Domain Routing, a compact way to write a whole range of IPs at once, like 10.244.0.0/16). In a default cluster, when a node joins, the controller manager (the cluster's background housekeeper, a process that quietly keeps reality lined up with what you asked for) slices a smaller chunk off that block and stamps it onto the Node's record. Every Pod scheduled onto that node draws its address from that node's chunk. Different nodes, different chunks, which is exactly why the two Pods sat in .1 and .2. Not every cluster works that way. Some plugins carve the per-node blocks themselves, Calico with its own IPAM (IP Address Management) and Cilium in cluster-pool mode among them, and there the Node's podCIDR field often comes back empty because the blocks live in the plugin's own records instead. The command below tells you which setup you're on.

the slice of the pod range each node owns
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
output
node-a	10.244.1.0/24
node-b	10.244.2.0/24

The kubelet doesn't actually hand out the address itself. When a Pod starts, the CNI plugin gets invoked (the container runtime, the lower-level software that launches the containers, is what calls it), and the plugin pulls a free IP from that node's chunk, builds the Pod's virtual network card, and programs the routes so the flat model holds up. That part is plumbing the plugin owns. What the model cares about is the result: a stable, cluster-unique IP the Pod keeps for its whole life, and gives up for good the second it dies.

No NAT, and why that matters more than it sounds

The no-NAT rule reads like plumbing trivia. It isn't small. NAT rewrites the source address on a packet, so the receiver sees the router's address instead of the real sender's, the way every call out of a big building can show up as 'front desk' on the caller ID. Kubernetes forbids that between Pods. When Pod A opens a connection to Pod B, B sees A's actual Pod IP as the source, the genuine sender.

That preserved identity pulls real weight. NetworkPolicy rules (the cluster's firewall, which you'll meet later) match on the source Pod IP. Service meshes and mutual TLS (mutual Transport Layer Security, where both ends prove who they are with certificates) pin a workload's identity to it. Your access logs are only worth reading because of it. Quietly break the no-NAT rule and all three start telling you comfortable lies.

a pod reports its own IP, then a probe pinned to node-a reaches it across nodes
kubectl exec web-8b21 -- hostname -i
kubectl run probe --rm -i --restart=Never --image=busybox:1.36 \
  --overrides='{"spec":{"nodeName":"node-a"}}' -- \
  wget -qO- -T 3 http://10.244.2.4:8080
output
10.244.2.4
served by web-8b21
pod "probe" deleted

web-8b21 reports its own address as 10.244.2.4, the very IP the pod list showed a moment ago. That's the same-IP property in plain sight: what a Pod calls itself is exactly what everyone else calls it. Then a throwaway probe Pod, pinned to node-a with nodeName so it can't land beside its target, reached that address on port 8080 and got a real answer back, with no Service and no DNS anywhere in the path. The pinning is the whole point: web-8b21 sits on node-b, so the request had to leave one machine and arrive at another. Let the scheduler drop the probe wherever it likes and it may well land on node-b, and then all you've shown is that two Pods on one machine can talk. Raw Pod IP, straight across the flat network.

A pod can't reach another pod: where's the break?
Pod A can't reach Pod B
work down the layers, raw IP first
fails on the same node too
CNI wiring on that node
the pod's virtual link isn't set up, or the CNI agent on that node is down
only fails across nodes
the overlay path
a firewall on VXLAN/IP-in-IP, or a wrong MTU
only fails via the Service IP
kube-proxy or endpoints
a layer sitting above the flat model
only fails by name
CoreDNS
name resolution, not reachability
The flat model is raw Pod-IP reachability. If Pod-to-Pod by IP already works, the fault lives in a layer above it, Service or DNS, not in the network model.

Reach is not the same as safety

There's a sharp edge folded into all this reach. Flat and open means open to everything. With no NetworkPolicy in place, one compromised Pod, say a leaked shell or a container quietly mining crypto for someone else, can reach every other Pod, every database, every internal API, because the model promises it can. Reachability is not a security boundary. The network starts fully connected, and you carve isolation out of it by adding NetworkPolicy, which the security section digs into. Reading 'well, they're in different namespaces' as if that meant isolation is one of the most common, and most expensive, mistakes people make on a cluster.

Same-node works, cross-node hangs: suspect the overlay, not your app
This one's brutal and common. Pods on the same machine talk fine. Ping across machines works. Even small HTTP requests work. But any larger response, a TLS handshake or a real API payload, hangs forever. Cross-node Pod traffic usually rides through an overlay: the CNI wraps each Pod packet inside another packet to ferry it between machines. Two usual culprits break that. First, a node firewall or cloud security group is silently dropping the overlay traffic (for example VXLAN on UDP port 8472, or IP-in-IP), which kills cross-node paths while same-node stays healthy. Second, the MTU is wrong. MTU (Maximum Transmission Unit) is just the biggest packet a link will carry, and the overlay adds roughly 50 extra bytes of headers to every packet. If the Pod's interface still claims it can send 1500-byte packets, a full-size one overflows the tunnel and gets dropped without a peep. Ping and the handshake use tiny packets and sail right through, which is exactly what makes this so maddening. Fix: allow the overlay ports in the security group, and set the CNI's MTU a bit below the node's.

HostPorts and hostNetwork punch holes in the model. Use them rarely and document why.

NetworkPolicy sits on top of the model. Without a supporting CNI, YAML is fiction.

When two pods cannot talk, decide first whether you are debugging CNI, Service, or DNS. Mixing layers wastes hours.

Try this

Get pods with wide output and note that every pod IP is reachable from another pod without NAT in a healthy CNI. Then compare a Service ClusterIP — that is a different virtual address.

terminal
$ kubectl get pods -o wide
 
$ kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.podCIDR}{"\n"}{end}'
 
$ kubectl exec web-8b21 -- hostname -i
$ kubectl run probe --rm -i --restart=Never --image=busybox:1.36 \
  --overrides='{"spec":{"nodeName":"node-a"}}' -- \
  wget -qO- -T 3 http://10.244.2.4:8080
 
$ kubectl get svc -o wide

Takeaway

The cluster network model is flat pod IPs, no NAT between pods, and agents on nodes. CNI plugins implement those rules.

Quick check
01A Pod on node-a can ping a Pod on node-b, and small HTTP requests between them succeed, but any response larger than a few kilobytes hangs and eventually times out. What's the most likely cause?
Correct — This is the classic MTU-mismatch fingerprint. Ping and the TCP handshake are small and succeed, but the first large payload overflows the tunnel MTU and vanishes, so the connection stalls. Lower the CNI's MTU below the node's.
Incorrect — A policy that dropped this traffic would kill the whole connection, including the ping and the handshake. Success that depends on payload size points at MTU or the overlay path, not policy.
Incorrect — DNS isn't in this path at all. The test connects by raw IP, and a name-resolution failure would stop the connection before any packet flowed, not halfway through a large response.
Incorrect — kube-proxy only matters for Service (ClusterIP) traffic. This is direct Pod-to-Pod by IP, which the flat model delivers without kube-proxy involved.
02Two Pods land in 10.244.1.x and 10.244.2.x because each node owns a different slice of the cluster's pod address block. In a default cluster, which component carves that per-node slice (the Node's podCIDR) out of the cluster CIDR?
Incorrect — The CNI does pull each Pod's IP from the node's slice, but where podCIDR is set, the controller manager set it before the plugin ever ran. Some plugins (Calico with its own IPAM, Cilium in cluster-pool mode) do carve per-node blocks themselves, and then they keep them in their own records rather than in podCIDR.
Correct — in a default cluster the controller manager slices a smaller chunk off the cluster CIDR and stamps it onto the Node's record as its podCIDR.
Incorrect — The kubelet starts containers but never defines the node's range or hands out addresses, as the lesson states directly.
Incorrect — kube-proxy handles ClusterIP-to-Pod rewriting, a layer above the flat network, and plays no part in assigning pod ranges.
03A container in the 'web' namespace is compromised. The cluster has no NetworkPolicy anywhere. Can that container open a connection to a database Pod running in a separate 'payments' namespace?
Correct — reachability is not a security boundary, so without a NetworkPolicy the flat model lets one compromised Pod reach every other Pod regardless of namespace.
Incorrect — This is the exact expensive mistake the lesson warns about; namespaces group objects but do not isolate the network.
Incorrect — Raw Pod-to-Pod reachability needs no Service; the flat model delivers packets by Pod IP directly, across namespaces.
Incorrect — The flat model makes every Pod one hop from every other regardless of node, so node placement has no bearing on reachability.
Multiple schedulers & profilesPod networking & CNI

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep