>_SecOpsLog_
CoursesKubernetes administrationDNS & CoreDNS

DNS & CoreDNS

Service and pod names, and how they resolve.

Intermediate10 min · lesson 35 of 65
In plain terms
CoreDNS is the cluster’s phone book. You look up a name (“payments”) and it hands back the number. Break the phone book and suddenly nobody can call anyone by name.

Half the cluster looks down at 3am. Payments can't reach the database, the web tier is throwing 'no such host,' and dashboards glow red across three namespaces that have nothing to do with each other. Nobody deployed. Nobody pushed. What actually broke is DNS, the Domain Name System, the same name-to-address lookup the public internet runs on, except this copy lives inside your cluster and everything quietly leans on it. There's no alert that says 'DNS is down.' Every symptom points somewhere else, which is exactly why it eats your night.

Every Pod (the smallest thing you deploy in Kubernetes, one or more containers that share a single network address) reaches the rest of the cluster by name, never by a hard-coded IP address. Your app opens a connection to db.data and trusts something to turn that name into a real address. That something is CoreDNS, the cluster's built-in name server. It works like the front-desk directory in an office building. You give a name, you get back a room number. The kubelet, the agent Kubernetes runs on every node, wires each Pod to CoreDNS on its own by writing a file called /etc/resolv.conf inside the container the moment the Pod starts. You set none of this per app. It just happens, on every Pod, in every namespace. That is the default though, not a fixed rule. A Pod running with hostNetwork: true gets the node's own resolver instead of CoreDNS unless you also set dnsPolicy: ClusterFirstWithHostNet, and a Pod with dnsPolicy: None plus a dnsConfig block uses whatever nameserver, search and ndots values you write there. When one node-level agent can't resolve a Service name while every other Pod can, that is the first thing to check.

terminal
$ kubectl exec -n shop payments-api-7c9d -- cat /etc/resolv.conf
search shop.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

Three lines do all the work. nameserver 10.96.0.10 is the ClusterIP of the CoreDNS Service, its stable cluster-internal address, so every question this Pod asks goes there. The search line is a list of suffixes the resolver tries when you hand it a short name, ordered most-specific first so same-namespace names win. And options ndots:5 sets the threshold: any name with fewer than five dots counts as a short name and gets run through the search list before it's tried as written. Five is the default, and it's deliberately generous so that both db.data and db.data.svc resolve to the same place. That same generosity is where a lot of DNS latency hides, which comes back to bite you later.

How a name becomes an IP

Services answer to one predictable pattern. A Service named db in the data namespace has the fully qualified name (FQDN) db.data.svc.cluster.local, its full postal address, spelled out end to end. From a Pod in the same namespace, plain db is enough, the way you'd just say 'Bob in Sales' when you're already in the building. From anywhere else, db.data works, because the resolver walks the search list and lands on db.data.svc.cluster.local along the way. The payoff is that you write app config against db.data, a name that never changes, while the Service's ClusterIP gets reassigned and Pods churn underneath it. The ndots:5 catch shows up right here. A short name gets tried against every search suffix first, so one lookup of db.data can fan out into several queries where all but one miss. On a hot path, writing the full name with a trailing dot (db.data.svc.cluster.local.) skips the guessing and cuts real load.

terminal
$ kubectl run probe --rm -it --image=busybox:1.28 --restart=Never -n shop -- \
    nslookup db.data
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
 
Name:      db.data.svc.cluster.local
Address 1: 10.96.140.7 db.data.svc.cluster.local
 
$ kubectl run probe --rm -it --image=busybox:1.28 --restart=Never -n data -- \
    nslookup db-headless.data
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
 
Name:      db-headless.data.svc.cluster.local
Address 1: 10.244.1.9 db-0.db-headless.data.svc.cluster.local
Address 2: 10.244.2.5 db-1.db-headless.data.svc.cluster.local

The first lookup returns a single address, the ClusterIP, and kube-proxy (the node component that spreads connections across the Pods behind a Service) balances the real traffic behind it. The second is a headless Service, created by setting clusterIP: None, and DNS hands back the Pod addresses directly instead of one shared front address. Think of the difference between a front desk that routes your call and a printed list of everyone's direct extensions. That direct-line behavior is how a StatefulSet (the controller for stateful, individually named Pods like the members of a database) gives db-0 and db-1 their own resolvable names. A primary-and-replica database needs exactly that, so a replica can reach the primary on purpose instead of being thrown at a random backend by a load balancer.

A name won't resolve: where to look
A name won't resolve
which of these is it?
CoreDNS Pods down or crash-looping?
Fix CoreDNS itself
read logs; check the Corefile and the upstream resolver
Only one namespace fails?
Suspect a NetworkPolicy
allow egress on port 53 to kube-system
Slow, but eventually works?
ndots and the search list
fully-qualify the hot names with a trailing dot
Only external names fail?
forward plugin / upstream
check the node's resolver and the forward line
Start by resolving a known Service from a throwaway Pod. If that hangs, the fault is below your app; walk this tree top to bottom.

Inside CoreDNS: the Corefile

CoreDNS is nothing exotic. It's an ordinary Deployment in the kube-system namespace, usually two replicas sitting behind a Service. Its whole behavior lives in a single file, the Corefile, stored as a ConfigMap (a plain configuration object the cluster keeps for you) that you can read and edit. The Corefile is a stack of plugins, each one a clerk in a mailroom who either handles a letter or slides it to the next desk, and two clerks do most of the work. The kubernetes plugin watches the API server (the control plane's front door) for Services and their Endpoints (the live list of Pod addresses behind each Service), then answers anything under cluster.local straight from that data, which is why a brand-new Service becomes resolvable within about a second of being created. The forward plugin takes everything else, github.com or your internal registry, and hands it to the node's upstream resolver. A cache plugin keeps recent answers for thirty seconds so repeat questions get served instantly. It skips caching cluster.local on purpose, since the kubernetes plugin already answers those from a live view, so a cached copy could only ever be staler than the source.

terminal
$ kubectl -n kube-system get deploy coredns
NAME      READY   UP-TO-DATE   AVAILABLE   AGE
coredns   2/2     2            2           58d
 
$ kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}'
.:53 {
    errors
    health {
       lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
       pods insecure
       fallthrough in-addr.arpa ip6.arpa
       ttl 30
    }
    prometheus :9153
    forward . /etc/resolv.conf {
       max_concurrent 1000
    }
    cache 30 {
       disable success cluster.local
       disable denial cluster.local
    }
    loop
    reload
    loadbalance
}

When DNS breaks, everything looks broken

Because every app finds its dependencies by name, CoreDNS is the one piece whose failure impersonates a hundred unrelated failures. Connection-refused and no-such-host errors spray across namespaces that never talk to each other, and it reads like a whole-cluster outage rather than one broken directory. So when a lot breaks at once and nothing shipped, DNS is a cheap thing to rule out first. Confirm the CoreDNS Pods are running, tail their logs for errors, then try to resolve a known Service from a disposable Pod. If that single lookup hangs while the CoreDNS Pods themselves look fine, you've found the blast radius, and it usually points at whatever sits between the Pod and CoreDNS. Watch the restart column while you're there. A CoreDNS Pod that keeps dying and restarting, often from the loop plugin catching a forwarding loop back to itself, shows up as a climbing restart count rather than a clean zero.

terminal
$ kubectl -n kube-system get pods -l k8s-app=kube-dns
NAME                       READY   STATUS    RESTARTS   AGE
coredns-7db6d8ff4d-4xk2p   1/1     Running   0          58d
coredns-7db6d8ff4d-q8n9v   1/1     Running   0          58d
 
$ kubectl run probe --rm -it --image=busybox:1.28 --restart=Never -n shop -- \
    nslookup db.data
Server:    10.96.0.10
Address 1: 10.96.0.10
 
nslookup: can't resolve 'db.data'
pod "probe" deleted
A default-deny egress policy that forgets port 53
Lock down a namespace with a default-deny egress NetworkPolicy (a firewall rule for Pods) and you also cut the path to CoreDNS unless you say otherwise. The nasty part is the delay. Existing connections and cached names keep working for a few minutes, then every Pod in that namespace starts failing name resolution at once, long after the policy landed, so it looks unrelated to the change nobody remembers making. Always pair a default-deny with an egress rule that allows UDP and TCP to port 53 toward the kube-system namespace. When a policy rollout triggers broad no-such-host errors on a delay, suspect the missing DNS exception before anything else.

ndots and search domains explain why some lookups append surprising suffixes. That is why FQDNs end with a dot in careful scripts.

Caching and autopath features change latency. The Corefile above turns caching off for cluster.local, so CoreDNS itself never hands back a stale in-cluster answer, but its replies carry a 30-second time to live (TTL), so an app that already resolved a name can keep using the old address for a few seconds after the Service is deleted.

NetworkPolicy must allow DNS egress or every allowlist becomes a denial of service on yourself.

Try this

Exec into a pod, cat resolv.conf, and nslookup a Service short name and a cross-namespace FQDN. Then break CoreDNS briefly in a lab and watch resolution fail.

terminal
$ kubectl run probe --image=busybox:1.28 --restart=Never -n shop -- sleep 3600
pod/probe created
 
$ kubectl exec -n shop probe -- cat /etc/resolv.conf
search shop.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5
 
$ kubectl exec -n shop probe -- nslookup payments-api
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
 
Name:      payments-api.shop.svc.cluster.local
Address 1: 10.96.88.21 payments-api.shop.svc.cluster.local
 
$ kubectl exec -n shop probe -- nslookup db.data.svc.cluster.local
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local
 
Name:      db.data.svc.cluster.local
Address 1: 10.96.140.7 db.data.svc.cluster.local
 
$ kubectl -n kube-system scale deployment coredns --replicas=0
deployment.apps/coredns scaled
 
$ kubectl exec -n shop probe -- nslookup db.data
Server:    10.96.0.10
Address 1: 10.96.0.10
 
nslookup: can't resolve 'db.data'
command terminated with exit code 1
 
$ kubectl -n kube-system scale deployment coredns --replicas=2
deployment.apps/coredns scaled
 
$ kubectl delete pod probe -n shop
pod "probe" deleted

Takeaway

CoreDNS turns Service and pod names into IPs. Most "network" outages are DNS. Check resolv.conf and CoreDNS pods before you rebuild CNI.

Quick check
01You apply a default-deny egress NetworkPolicy to the shop namespace. Everything works for a few minutes, then every Pod in shop starts failing with 'no such host,' while other namespaces resolve names fine. The CoreDNS Pods are all Running with zero restarts. What's the most likely cause?
Incorrect — the CoreDNS Pods are Running with no restarts and other namespaces still resolve, so the name server is healthy. Restarting it changes nothing.
Correct — the failure is scoped to shop, delayed until caches expired, and CoreDNS itself is fine. That's the signature of a default-deny that never allowed DNS out.
Incorrect — ndots adds latency and wasted queries, not total failure, and it wouldn't suddenly break one namespace the moment a NetworkPolicy landed.
Incorrect — ClusterIPs are stable for a Service's lifetime, and apps address Services by name precisely so IP changes never matter.
02A Pod's /etc/resolv.conf carries options ndots:5. Why can one lookup of a short name like db.data turn into several DNS queries, and how do you avoid it on a hot path?
Incorrect — ndots has nothing to do with caching; it controls when the search-suffix list is applied to a name.
Correct — db.data has under five dots, so the resolver appends each search domain in turn before trying it as written, and a trailing dot (db.data.svc.cluster.local.) skips that guessing.
Incorrect — The number is a dot-count threshold for applying the search list, not a retry counter.
Incorrect — There is a single nameserver line pointing at the CoreDNS Service; the fan-out comes from search suffixes, not multiple servers.
03In-cluster names like db.data resolve fine, but Pods can't resolve public names such as github.com. The CoreDNS Pods are Running with zero restarts. Which part of CoreDNS should you suspect?
Incorrect — That plugin is clearly healthy since it is answering your in-cluster names; the problem is only for names it hands off elsewhere.
Incorrect — Blocking port 53 would break every lookup, including the cluster names that are working, so it can't be scoped to external names only.
Correct — everything outside cluster.local is handed to the forward plugin, so an external-only failure points at that line or the upstream resolver behind it.
Incorrect — ndots adds latency and wasted queries, not a clean failure limited to external names while cluster names resolve.
Service networking & kube-proxyIngress

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep