Service networking & kube-proxy

From ClusterIP to an endpoint via iptables/IPVS.

Try to ping a Service's ClusterIP on a cluster running kube-proxy in iptables mode, the usual default, and nothing comes back. Open a plain TCP connection to that exact same address, on its port, and it answers on the first try. That contradiction is the whole lesson, and the thing pulling off the trick is a small program called kube-proxy.

A Service in Kubernetes is a stable front for a group of identical, throwaway pods (a pod is the smallest thing the cluster runs, usually one container plus its sidekicks). The pods behind it come and go and get fresh IP addresses each time they respawn. The Service's address never moves. And that address, when the Service type is ClusterIP, is completely made up. In iptables mode no network card owns it. No pod is listening on it directly. It's a number the cluster invented so you'd have something steady to talk to. Think of a company's main phone line. That number doesn't ring any one desk. A receptionist answers, patches you through to whoever is free, then steps off the line. kube-proxy is that receptionist. It runs on every node (a node is one worker machine), watches the cluster's control desk, the API server that every part of Kubernetes reads from and writes to, for Services and the pods behind them, and writes rules into the node's Linux kernel. The rules say one thing. Any packet aimed at this ClusterIP, rewrite its destination to the real address of one healthy pod.

That rewrite has a name, DNAT, short for destination network address translation, and where it happens is the part worth getting right (it's a favorite on the Certified Kubernetes Administrator exam, the CKA). kube-proxy never touches your traffic itself. It only installs the rules. The kernel does the actual packet surgery, at full network speed, which is why putting a Service in front of your pods adds almost no delay. So how does kube-proxy know which pods count as healthy? It reads EndpointSlices. An EndpointSlice is a little list object that a controller keeps in step with each Service's label selector, one entry per backing pod. Each entry is stamped ready or not-ready based on the pod's readiness probe, the health check a pod has to pass before it should get traffic. kube-proxy writes a rule only for the ready entries. Fail the probe and you drop out of rotation the moment the slice updates, even though your entry is still sitting in the slice marked not-ready.

$ kubectl get svc payments-api
NAME           TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
payments-api   ClusterIP   10.96.14.7    <none>        80/TCP    9d
 
$ kubectl get endpointslices -l kubernetes.io/service-name=payments-api
NAME                 ADDRESSTYPE   PORTS   ENDPOINTS
payments-api-7bd2k   IPv4          8080    10.244.1.7,10.244.2.4,10.244.1.9

You can read the rules kube-proxy wrote, right there on the node. In iptables mode (iptables is the classic Linux packet-filtering system), a packet to the ClusterIP hits a chain called KUBE-SERVICES, jumps to a per-Service chain, and lands on one endpoint chain picked at random. That last hop is where the load balancing actually lives. Three backends means the kernel rolls a weighted die: a one-in-three chance for the first, then a coin flip between the two that are left, and if both of those miss, the third. Work the math and each pod ends up with an even third of the traffic. kube-proxy also stamps every rule it writes with a comment naming the Service and the pod behind it, which is the only reason grepping for a Service name finds anything at all.

$ sudo iptables-save -t nat | grep payments-api
-A KUBE-SERVICES -d 10.96.14.7/32 -p tcp -m comment --comment "default/payments-api:http cluster IP" -m tcp --dport 80 -j KUBE-SVC-XKN7
-A KUBE-SVC-XKN7 -m comment --comment "default/payments-api:http -> 10.244.1.7:8080" -m statistic --mode random --probability 0.33333333 -j KUBE-SEP-AAA
-A KUBE-SVC-XKN7 -m comment --comment "default/payments-api:http -> 10.244.2.4:8080" -m statistic --mode random --probability 0.50000000 -j KUBE-SEP-BBB
-A KUBE-SVC-XKN7 -m comment --comment "default/payments-api:http -> 10.244.1.9:8080" -j KUBE-SEP-CCC
-A KUBE-SEP-AAA -p tcp -m comment --comment "default/payments-api:http" -m tcp -j DNAT --to-destination 10.244.1.7:8080

iptables, IPVS, and now nftables

kube-proxy has three ways to write those rules, and the difference starts to bite once a cluster gets big. iptables mode, the long-time default, builds one long chain and picks a backend with that random statistic match. It's fine for most clusters. The catch is that iptables is a list. Every Service you add makes the whole rule set longer, and a full re-sync of tens of thousands of rules can take seconds. Your traffic doesn't stop while that runs: the kernel keeps forwarding on the old rule set until the new one is swapped in whole. What arrives late is the news. For those seconds the node is still handing packets to pods that already left. IPVS mode (IP Virtual Server, a load balancer built into the Linux kernel) swaps the list for a hash table, with real scheduling algorithms like round-robin and least-connection, and lookup time that stays flat no matter how many Services you run. It also does one thing that surprises people: it binds every ClusterIP to a dummy interface on each node called kube-ipvs0, so on an IPVS cluster the node really does own the address and a ping to a ClusterIP usually gets a reply. Newer still is nftables mode, and nftables is the modern Linux packet-filtering framework that's slowly replacing iptables. It went stable in Kubernetes 1.33, so on a current cluster it is a production choice and not a preview. It keeps that flat, kernel-side speed without some of IPVS's rough edges, it leaves the ClusterIP owned by nothing the way iptables mode does, and it's where the project is heading on fresh clusters. By default the die roll happens per connection, so two requests from the same client can land on different pods. Set spec.sessionAffinity to ClientIP and kube-proxy pins each client to one backend for a while, which some stateful apps rely on. You pick the mode once, in kube-proxy's ConfigMap, which is just a bag of settings stored in the cluster.

$ kubectl -n kube-system get configmap kube-proxy -o jsonpath='{.data.config\.conf}' | grep mode
mode: "ipvs"
 
$ sudo ipvsadm -Ln | grep -A3 10.96.14.7:80
TCP  10.96.14.7:80 rr
  -> 10.244.1.7:8080    Masq    1      0          0
  -> 10.244.2.4:8080    Masq    1      0          0
  -> 10.244.1.9:8080    Masq    1      0          0

When a Service goes dark

Because kube-proxy only wires up ready endpoints, debugging a broken Service collapses into two questions. First: does the Service have any ready backends at all? The fastest check is kubectl get endpoints on the Service name. That command prints only the ready addresses, the exact set kube-proxy would route to, boiled down from those slices. If the list comes back empty, kube-proxy is off the hook. There's nothing for it to route to, and it says so out loud: with zero ready endpoints kube-proxy installs a REJECT rule for that ClusterIP, so the client gets connection refused the instant it tries instead of hanging. Learn that signal. An immediate refusal points at an empty endpoints list. A connection that hangs until it times out points somewhere else, at a missing rule or a NetworkPolicy. When the list is empty the fault is upstream, either a label selector that doesn't match your pods or pods stuck failing their readiness probe. Only when the endpoints are populated and the ClusterIP still won't answer do you turn to kube-proxy and the kernel rules, on the node where the client is actually running. Get that order backwards and you'll burn an hour restarting kube-proxy when one kubectl get endpoints would have shown you an empty list and pointed you the other way.

kube-proxy runs as a DaemonSet, which is Kubernetes-speak for one pod on every node, and each copy writes only its own node's rules. That's why the weirdest outage you'll meet is a Service that works from pods on most nodes but not from pods on one. A kube-proxy that crashed or fell behind on that single node leaves stale or missing rules there while every other node hums along. You check it the same way you'd check any other DaemonSet.

$ kubectl -n kube-system get ds kube-proxy
NAME         DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   AGE
kube-proxy   3         3         3       3            3           41d
 
$ kubectl -n kube-system logs ds/kube-proxy --tail=3
I0716 09:14:22.108  server_others.go:74] "Using ipvs Proxier"
I0716 09:14:22.640  proxier.go:1503] "SyncProxyRules complete" elapsed="21.7ms"
I0716 09:14:52.661  proxier.go:1503] "SyncProxyRules complete" elapsed="12.9ms"

IPVS and iptables fail differently under huge Service counts. Know your mode before you tune sysctls.

externalTrafficPolicy Local preserves client IPs and can also drop traffic on nodes without local pods.

If kube-proxy is down on one node, only traffic that hits that node breaks. Symptoms look random until you map node connectivity.

Try this

Make a node prove all of this on your own cluster. Pick any ClusterIP Service, write down its address and how many ready endpoints it has, then find that same address in a node's kernel rules and count the backends listed there. The question you are answering is whether the two counts match.

# from your workstation: the address, the ready backends, the mode
$ kubectl get svc <your-service>
$ kubectl get endpointslices -l kubernetes.io/service-name=<your-service>
$ kubectl -n kube-system logs ds/kube-proxy | grep -m1 -i Proxier
 
# on a node, typing in the ClusterIP you just read (10.96.14.7 here)
$ sudo iptables-save -t nat | grep '10.96.14.7/32'
$ sudo iptables-save -t nat | grep '<your-service>' | grep -c ' -j KUBE-SEP'
 
# IPVS clusters instead: the backend list, and the interface holding the address
$ sudo ipvsadm -Ln | grep -A5 '10.96.14.7:'
$ ip addr show kube-ipvs0 | grep 10.96.14.7
 
$ ping -c1 -W1 10.96.14.7

A healthy node gives you one KUBE-SERVICES rule for the address and one KUBE-SEP branch per ready endpoint, so the count matches what the API server told you. Fewer branches than endpoints, or no rules at all while the endpoints list is full, is the failure this lesson is about: that node's kube-proxy is behind or dead. Some empty results are not faults, though. On an IPVS cluster the iptables-save grep finds nothing while ipvsadm lists your Service, and the ping answers because kube-ipvs0 holds the address. On an iptables cluster it's the reverse: ip addr show kube-ipvs0 reports that the device does not exist, and the ping times out in silence while a TCP connection to the same address still works.

Takeaway

kube-proxy turns Service and EndpointSlice objects into node packet rules. With no ready endpoints it installs a REJECT rule, so clients get an instant connection refused rather than a silent hang.

Related