Network troubleshooting

DNS, kube-proxy, and CNI checks in order.

"The database is down." That's what the alert said. Ten services throwing unknown host: db.prod, pages firing, someone already spinning up an incident channel. But the database pods are up. A pod is the smallest unit Kubernetes runs, one or more containers that share a single network address, and these ones are healthy, answering connections from a debug pod sitting right next to them. Nothing is actually down. This is the shape of almost every Kubernetes network incident. The symptom screams "everything is broken," and the cause is one quiet layer.

Think of one pod reaching another as a letter moving through a giant building's mailroom. It has to clear five desks before it arrives. First it needs the address: DNS, the Domain Name System, is the cluster's phone book, and it turns the name db.prod into an IP address. Then someone has to be sitting at that desk, because a Service, the stable front door for a group of pods, needs ready pods behind it, called its endpoints. Then the building's internal routing has to carry the envelope, and that job is kube-proxy, which programs the kernel so a Service's virtual IP forwards to a real pod. Then security at the door has to wave it through, because a NetworkPolicy can block it. And under all of it, the corridors have to exist at all: the CNI, or Container Network Interface, is the plugin (Calico, Cilium, and friends) that hands each pod its IP address and moves packets between machines. Check the five desks in the order they actually fail, most common first, and stop at the first "no."

Start with DNS, because it lies to you

DNS breaks more often than anything else down here, and when it breaks the whole cluster looks like it caught fire, because suddenly every service can't find any other service. So you check it first. And you check it from inside an affected pod, not from your laptop and not by logging into the node, because the pod is the one whose view of the world is broken. Its own DNS config, its own network namespace. Only the pod sees what the pod sees.

kubectl exec -it deploy/web -n prod -- nslookup db.prod

;; connection timed out; no servers could be reached
command terminated with exit code 1

Read the failure, don't just log that it failed. A timeout means the query never reached CoreDNS, the DNS server that itself runs as a few pods in the kube-system namespace. Never reaching the server is a routing or firewall problem, not a naming problem. Compare that to NXDOMAIN, which means the query did reach CoreDNS and the name genuinely doesn't exist, usually a typo or the wrong namespace. Two different failures, two different fixes. A timeout that shows up right after someone shipped a new NetworkPolicy points almost straight at the cause.

The one-rule outage

A NetworkPolicy is a firewall rule for pods, and it's enforced by the CNI plugin, not by the API server. That surprises people. The classic trap is the "default-deny" pattern. You add a policy that blocks all egress, meaning outbound traffic, so your pods can only reach what you explicitly allow. Good instinct. But DNS lookups are egress too. If your allow-list forgets to permit port 53 to kube-system, every pod the policy selects goes deaf. It can't resolve a single name, so every dependency it has looks dead at once. One forgotten rule, a namespace-wide "outage."

kubectl describe networkpolicy default-deny-egress -n prod

Name:         default-deny-egress
Namespace:    prod
Spec:
  PodSelector:     <none> (Allowing the specific traffic to all pods in this namespace)
  Allowing egress traffic:
    <none> (Selected pods are isolated for egress connectivity)
  Policy Types: Egress

There it is. Egress set to <none> for every pod in the namespace, and no exception carved out for DNS. The fix doesn't touch the deny at all. NetworkPolicies are additive, so you leave the block in place and stack a second policy on top that allows port 53 to kube-system. This one finds kube-system by its built-in namespace label, the kubernetes.io/metadata.name label that Kubernetes stamps on every namespace automatically.

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: prod
spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

$ kubectl apply -f allow-dns-egress.yaml
networkpolicy.networking.k8s.io/allow-dns-egress created

kubectl exec -it deploy/web -n prod -- nslookup db.prod

Server:    10.96.0.10
Address:   10.96.0.10#53
 
Name:      db.prod.svc.cluster.local
Address:   10.96.140.22

When it isn't DNS

Say the name resolves fine and the app still can't connect. Next desk: is anyone actually answering? A Service points at pods through a label selector, a simple "match everything tagged app=db" filter. Only ready pods get listed as endpoints, the ones passing their readiness probe. That probe is the health check Kubernetes uses to decide whether a pod is allowed to receive traffic yet. No endpoints means no target, and an empty target reads like a network fault but isn't one.

kubectl get endpoints db -n prod

NAME   ENDPOINTS   AGE
db     <none>      15m

That <none> sends you to one of two places. Either the Service's selector doesn't match any pod's labels (a typo of app=db versus app=database will do it), or the pods are running but failing readiness, so Kubernetes is deliberately holding them out of rotation. Under the hood this list now lives in EndpointSlices, and kubectl get endpointslices -l kubernetes.io/service-name=db -n prod shows the same picture with more detail. If endpoints do exist, hit the ClusterIP and port directly. That exercises kube-proxy, the component that programs iptables or IPVS rules on every node so a virtual Service IP actually forwards to a live pod.

kubectl exec -it deploy/web -n prod -- nc -zv db.prod 5432

Connection to db.prod (10.96.140.22) 5432 port [tcp/postgresql] succeeded!

"Succeeded" means kube-proxy did its job and the packet reached a pod. If that same call hangs while endpoints clearly exist, confirm kube-proxy is even running on the nodes with kubectl get pods -n kube-system -l k8s-app=kube-proxy. Only when DNS resolves, endpoints exist, the ClusterIP answers, and no policy is blocking do you finally suspect the CNI. Its pods run as a DaemonSet, which just means Kubernetes keeps exactly one copy on every node. If Calico or Cilium is crash-looping on a node, every pod on that node loses networking at the same time, and that's a genuine outage instead of a one-rule slip. Walking the desks in order is the whole trick. It's what keeps you from reinstalling the CNI at 3 a.m. when the real problem was a forgotten port 53.

Working pod IP and failing ClusterIP points at Services or kube-proxy. Working ClusterIP and failing name points at DNS.

NetworkPolicy denies look like timeouts, not polite RSTs. Check policies early in locked namespaces.

Node-local problems affect only some clients. Map which worker your client pod landed on.

Try this

From a client pod, nslookup the Service, curl the ClusterIP, then curl a pod IP directly. The layer that fails first is where you dig.

$ kubectl exec -it deploy/web -n prod -- nslookup db.prod
 
$ kubectl describe networkpolicy default-deny-egress -n prod
 
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
$ metadata:
  name: allow-dns-egress
  namespace: prod
$ spec:
  podSelector: {}
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53
 
$ kubectl apply -f allow-dns-egress.yaml
networkpolicy.networking.k8s.io/allow-dns-egress created
 
$ kubectl exec -it deploy/web -n prod -- nslookup db.prod
 
$ kubectl get endpoints db -n prod

Takeaway

Network troubleshooting is DNS, then Service/endpoints/kube-proxy, then CNI. Order beats intuition.

Related