Pod networking & CNI

How a pod gets an IP, and what a CNI plugin does.

Spin up a fresh cluster with kubeadm, then run kubectl get nodes. Every node says NotReady. Create a pod and it sits in ContainerCreating and never moves. Nothing is actually broken. You just haven't hired the one thing Kubernetes refuses to do for itself: give your pods a network.

Kubernetes is deliberate about where its job ends. It defines the rules a cluster network must obey, then steps back and lets a plugin do the real wiring. Think of a new apartment building. The developer decides every unit gets its own phone line and any unit can dial any other directly, but the developer doesn't pull a single cable. They bring in an electrical contractor for that. In a cluster the contractor is the CNI plugin. CNI is short for Container Network Interface, and it's the piece that hands each Pod (Kubernetes' smallest unit, one or more containers sharing a single network identity) its own IP address and plugs it into the flat network that everything above it assumes is already there.

What the kubelet actually hands off

The kubelet is the agent Kubernetes runs on every node, and its job is to turn 'this pod belongs here' into a running container. Networking isn't part of that job. When a pod lands on a node, the kubelet asks the container runtime (containerd on most clusters) to build it, talking through the Container Runtime Interface, or CRI. The runtime first creates a tiny sandbox container whose only purpose is to hold the pod's network namespace, a private networking world with its own interfaces and routing table, walled off from the host. Then, before your application container ever starts, the runtime runs the CNI plugin binary and passes it the command ADD.

The plugin does three jobs and then leaves. It picks an IP for the pod out of the node's slice of the cluster network, work known as IPAM (IP Address Management). It builds a veth pair, a virtual ethernet cable with two ends, drops one end inside the pod's namespace as eth0, and leaves the other end standing on the host. And it writes routes so packets can find their way in and back out. When you delete the pod, the runtime calls the same plugin with DEL, which pulls the cable and returns the IP to the pool. All of this runs from a DaemonSet, one plugin pod per node, living in the kube-system namespace.

$ kubectl -n kube-system get pods -l k8s-app=calico-node -o wide

NAME                READY   STATUS    RESTARTS   NODE
calico-node-7hx2q   1/1     Running   0          node-1
calico-node-p9k4m   1/1     Running   0          node-2

You can watch both ends of the plugin's work from the pod's own point of view. The pod thinks it has a normal machine with a normal network card called eth0.

$ kubectl exec payments-api-x2f -- ip -br addr show eth0
$ kubectl exec payments-api-x2f -- ip route

eth0@if23    UP    10.244.1.7/32
default via 169.254.1.1 dev eth0
169.254.1.1 dev eth0 scope link

That eth0@if23 is a small clue worth reading. The @if23 means the pod's eth0 is one end of a veth pair whose other end sits on the host at interface index 23. That pairing is the cable. The default route points at 169.254.1.1, an address no real router owns; the host simply answers for it and forwards, which is how Calico gets a pod's traffic onto the node without a bridge.

When it breaks, and how to read it

A pod stuck in ContainerCreating that never clears is almost always the network. The pod exists, the scheduler placed it, but the sandbox can't be wired, so the kubelet keeps retrying. The events on the pod tell you who failed and why, and the message names the plugin directly.

$ kubectl describe pod checkout-9f2 | grep -A5 Events

Events:
  Type     Reason                  Message
  ----     ------                  -------
  Warning  FailedCreatePodSandBox  Failed to create pod sandbox: rpc error:
           code = Unknown desc = failed to setup network for sandbox
           "a1b2c3": plugin type="calico" failed (add): no IP addresses
           available in range set 10.244.2.0/24

There are two flavors of this. A single stuck pod usually means the plugin is up but couldn't finish, often because the node ran out of pod IPs. Every node showing NotReady means no plugin is running at all, so the kubelet reports the node's network as unavailable and the scheduler won't place work on it.

$ kubectl get nodes

NAME     STATUS     ROLES           AGE   VERSION
node-1   Ready      <none>          9d    v1.31.2
node-2   NotReady   <none>          9d    v1.31.2

The IP-exhaustion case is worth understanding because it surprises people. Each node gets a fixed slice of the pod network, often a /24, which is 256 addresses. Set max-pods too high, or leave addresses stranded by pods that never got cleaned up, and IPAM runs dry. New pods on that node then fail to get an IP even though the rest of the cluster has plenty of room. Widening the per-node block or lowering max-pods is the fix, not restarting the plugin.

The plugin you pick is the feature set you get

Here is the part people learn the hard way. NetworkPolicy, the object you write to say which pods may talk to which, is not enforced by Kubernetes. It's enforced by the CNI. Think of a NetworkPolicy as a sign on the door that reads 'residents only.' The sign changes nothing unless a guard is standing there reading it. Plain Flannel is a building with no guard: it accepts your policy, stores it in the API, and lets every packet through anyway. Calico and Cilium actually enforce it. The same split decides the rest of your options: pod-to-pod encryption with WireGuard, an eBPF datapath (eBPF, the extended Berkeley Packet Filter, runs small sandboxed programs inside the Linux kernel, which is how Cilium can take over Service routing from kube-proxy and drop it entirely), and traffic visibility through tools like Hubble. Choosing a CNI is choosing that whole list, not just whether packets move.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: payments
spec:
  podSelector: {}
  policyTypes:
  - Ingress

$ kubectl apply -f default-deny.yaml
$ kubectl -n payments run probe --image=busybox --rm -it --restart=Never \
    -- wget -qO- --timeout=3 http://payments-api

networkpolicy.networking.k8s.io/default-deny-ingress created
wget: download timed out
pod "probe" deleted

Now the policy bites, and the wget from another pod times out instead of returning a page. That gap between 'the object exists' and 'the object is enforced' is the whole reason to test. Apply a default-deny, then actually try to reach the pod. If traffic still flows, your plugin isn't enforcing policy, and no amount of correct YAML will change that.

IPAM exhaustion looks like scheduling success and start failure. Watch allocated ranges.

Upgrading CNI is a traffic event. Stage it like a data-plane change, not like a doc edit.

Overlay versus routed underlay changes MTU and troubleshooting tools. Know which you bought. Watch allocated ranges.

Try this

List CNI pods in kube-system, check node Ready conditions for network plugins, and inspect a pod sandbox IP. Confirm the plugin name your cluster actually runs.

$ kubectl -n kube-system get pods -l k8s-app=calico-node -o wide
 
$ kubectl exec payments-api-x2f -- ip -br addr show eth0
$ kubectl exec payments-api-x2f -- ip route
 
$ kubectl describe pod checkout-9f2 | grep -A5 Events
 
$ kubectl get nodes
 
$ kubectl apply -f default-deny.yaml
$ kubectl -n payments run probe --image=busybox --rm -it --restart=Never \
    -- wget -qO- --timeout=3 http://payments-api
 
networkpolicy.networking.k8s.io/default-deny-ingress created
wget: download timed out
pod "probe" deleted

Takeaway

CNI gives each pod its interface and routes. Plugin failure shows up as NotReady nodes or pods stuck Creating.

Related