>_SecOpsLog_
CoursesKubernetes administrationDaemonSets

DaemonSets

One pod per node, for agents and exporters.

Intermediate10 min · lesson 28 of 65
In plain terms
A DaemonSet is “one first-aid kit on every floor.” Every node gets exactly one copy, and any new floor that opens automatically gets one too.

You just added three worker nodes to the cluster. Nobody ran a deploy, nobody set a replica count, and within a minute a log shipper, a metrics agent, and the network plugin are all running on each new machine. That's a DaemonSet doing its one job. It keeps exactly one copy of a Pod (the smallest thing Kubernetes runs, one or more containers bundled together) on every node that qualifies, and it fixes the count on its own as nodes come and go.

Think of a building where every floor needs its own smoke detector. You don't decide the building gets twelve of them and scatter them around at random. You say one per floor, and when a new floor is built, it gets one too. A Deployment is the scatter approach: you pick a replica count and the scheduler drops those Pods wherever they fit. A DaemonSet is the one-per-floor rule. You never pick a number. The count is however many nodes you have, and it tracks them.

So the fit is anything that's about the node itself rather than about serving user traffic. A metrics exporter reading the node's CPU and disk. A log collector like Fluent Bit tailing every container's logs on that machine. A security agent like Falco watching system calls. The CNI plugin (Container Network Interface, the component that gives Pods their networking) has to sit on every node or Pods there never get an IP address. Run any of these as a Deployment and you'd get some arbitrary number of copies on random nodes, leaving most nodes uncovered. Blind spots, exactly where you wanted eyes.

One per node, and no replica field

Here's a real one: node-exporter, the Prometheus agent that reads hardware and kernel metrics off a node. Look for a replicas field. There isn't one. There's a label selector, a Pod template, and an update strategy, and that's the whole shape.

node-exporter.yaml (no replicas anywhere)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      tolerations:
      - operator: Exists
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.8.1
        ports:
        - containerPort: 9100
          hostPort: 9100

That tolerations block with operator: Exists means tolerate every taint, so no node repels this pod. Apply the manifest, then ask the API server (the cluster's control desk, the one thing every command talks to) what you got.

apply it, then read the summary
kubectl apply -f node-exporter.yaml
kubectl get ds node-exporter -n monitoring
DESIRED tracks node count; you never typed a number
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-exporter   4         4         4       4            4           <none>          38s

Read those columns like a checklist. DESIRED is how many nodes match (four here). CURRENT, READY, and AVAILABLE climb to meet it. UP-TO-DATE tells you how many run the current template, which is the number you watch during an upgrade. NODE SELECTOR is <none> because we didn't restrict it. Now confirm the placement is really one per node, not four on one node.

prove the spread with -o wide
kubectl get pods -n monitoring -o wide -l app=node-exporter
four pods, four different nodes, control-plane included
NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
node-exporter-4xk2p   1/1     Running   0          52s   10.0.0.10    cp-1       <none>           <none>
node-exporter-7dqzr   1/1     Running   0          52s   10.0.0.11    worker-1   <none>           <none>
node-exporter-b9m8s   1/1     Running   0          52s   10.0.0.12    worker-2   <none>           <none>
node-exporter-kf2wl   1/1     Running   0          52s   10.0.0.13    worker-3   <none>           <none>

Four pods on four nodes, and the IP is the node's own address because we set hostNetwork. The control-plane node got covered too, and that only happened because of the tolerations. That's where the interesting behavior lives.

What actually pins a pod to each node

For years the line was that the DaemonSet controller schedules its own pods. That stopped being true back in v1.12, and the newer story is worth knowing cold for the exam. The controller creates one Pod object per eligible node and stamps each one with a node affinity rule that names that exact node. Then it steps back and lets the normal kube-scheduler bind the pod, the same way it binds anything else. You can see the injected rule on any daemon pod.

look at what the controller stamped on the pod
kubectl get pod node-exporter-7dqzr -n monitoring -o yaml | grep -A9 affinity
a nodeAffinity that names exactly one node
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchFields:
          - key: metadata.name
            operator: In
            values:
            - worker-1

That matchFields on metadata.name is the fingerprint of a DaemonSet pod. Why does the change matter? Because the scheduler is the thing placing these pods now, they obey the same rules as everything else: resource requests, taints, priority, and preemption. If a node is genuinely out of room, the daemon pod sits Pending with an ordinary FailedScheduling event instead of getting force-fit onto a starved machine. It's more honest, and it surprises people who still expect daemon pods to muscle their way on.

The controller does hand you one convenience. It auto-adds tolerations for the taints Kubernetes stamps on a node itself: not-ready, unreachable when the control plane loses contact with the kubelet, low memory, low disk, low process IDs, and node.kubernetes.io/unschedulable, the taint kubectl cordon puts on a node. A pod using hostNetwork gets network-unavailable too. That's why a daemon pod keeps running on a node that's sick or cordoned, right when you most want its logs and metrics, and it's why cordoning a node does not stop a daemon pod from being placed there. What the controller does not do is tolerate taints you invented yourself. Hold that thought.

Rolling out a new version

A DaemonSet updates much like a Deployment, with one twist: there's one pod per node, so the rollout walks the cluster node by node. The default is RollingUpdate with maxUnavailable: 1, which means one node loses its daemon pod at a time while the replacement comes up. Bump the image and watch it march.

upgrade the image, then follow the rollout
kubectl set image ds/node-exporter node-exporter=prom/node-exporter:v1.8.2 -n monitoring
kubectl rollout status ds/node-exporter -n monitoring
one node at a time, because maxUnavailable is 1
daemonset.apps/node-exporter image updated
Waiting for daemon set "node-exporter" rollout to finish: 1 out of 4 new pods have been updated...
Waiting for daemon set "node-exporter" rollout to finish: 2 out of 4 new pods have been updated...
Waiting for daemon set "node-exporter" rollout to finish: 3 out of 4 new pods have been updated...
daemon set "node-exporter" successfully rolled out

maxUnavailable sets the pace. Push it to 25% on a large cluster and you replace pods a quarter of the nodes at a time, much faster, at the cost of that fraction of nodes briefly running no agent. Kubernetes also lets you set maxSurge instead, which brings the new pod up before the old one goes down on the same node, for true zero-gap coverage. That only works if the pod can tolerate two copies overlapping for a moment. A hostPort or hostNetwork agent like this one can't, because the port is already taken, so maxSurge would just wedge the new pod in Pending. The other strategy is OnDelete: nothing changes until you delete a pod by hand, for when you want to babysit a risky change node by node. Update history lives in ControllerRevision objects, not ReplicaSets, so a rollback is kubectl rollout undo ds/node-exporter.

Only some nodes

Sometimes one per node is too many. A GPU (graphics processing unit) monitoring agent only belongs on GPU nodes. Add a nodeSelector to the template and the eligible set shrinks to nodes carrying that label. Label one node, patch the DaemonSet, and the controller re-evaluates instantly.

narrow the DaemonSet to GPU nodes
kubectl label node worker-3 gpu=true
kubectl patch ds node-exporter -n monitoring --type merge \
  -p '{"spec":{"template":{"spec":{"nodeSelector":{"gpu":"true"}}}}}'
one node labeled, DaemonSet patched
node/worker-3 labeled
daemonset.apps/node-exporter patched
eligible set drops to the one matching node
kubectl get ds node-exporter -n monitoring
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-exporter   1         1         1       1            1           gpu=true        6m

DESIRED fell from 4 to 1. The three nodes that no longer match had their daemon pods removed, and only worker-3 keeps one. Flip the requirement (label more nodes, or drop the selector) and the controller adds or deletes pods to match, without you ever touching a count.

Your monitoring has a hole exactly where you can't see
The controller quietly tolerates the built-in node taints, so people assume daemon pods land everywhere. They don't. A control-plane node carries node-role.kubernetes.io/control-plane:NoSchedule, a taint the controller does not tolerate for you. Leave it out and your node-exporter runs on every worker and skips every control-plane node, so the machines running etcd (the cluster's key-value database) and the API server report no metrics at all. That's the worst place to go blind. Either add a toleration for that specific taint, or use tolerations with operator: Exists to accept all of them, then confirm with kubectl get pods -o wide that the pod count actually equals your node count.
How a brand-new node gets its daemon pod
1Node joinsa new machine registers with…2DaemonSetcontrollerwatching Nodes and its own…3Creates one podstamps it with a nodeAffinity…4kube-schedulerbinds the pod to that node,…5kubeletthe node's agent pulls the…
There's no replica count anywhere in this chain. The controller reacts to Node objects appearing and disappearing, so the pod count tracks the cluster on its own. And because the scheduler does the binding, a full or wrongly tainted node leaves the pod Pending instead of hiding the problem.

Two habits keep a DaemonSet rollout boring. First, pace it on purpose. maxUnavailable is really a count of how many nodes are blind at once, so 25% of a 200-node cluster means fifty machines with no log shipper for as long as the new pod takes to go Ready. Second, write the coverage down. If you are deliberately skipping control-plane nodes, say so in a comment on the manifest, because the next person paged at 3am cannot tell an intentional gap from a forgotten toleration.

One more thing is worth guarding: how much of the node the pod can touch. Agents like this one usually want hostNetwork, a hostPath mount of /proc or /var/log, and sometimes a privileged container, which means whoever gets inside a daemon pod effectively gets the node. Mount those host paths read-only wherever the agent will accept it, and give the pod's ServiceAccount only the API calls it actually makes, using RBAC (role-based access control, the rules that decide which API calls an identity is allowed to make).

Try this

Recreate the coverage gap on purpose in a lab namespace. Apply node-exporter.yaml with the tolerations block deleted, then compare DESIRED against your node count: the control-plane node is missing. Patch the toleration back in and watch coverage return. Then cordon a worker and delete its daemon pod. A replacement appears anyway, because cordon adds node.kubernetes.io/unschedulable and the controller already tolerates that one.

terminal
$ kubectl apply -f node-exporter.yaml    # tolerations block deleted
daemonset.apps/node-exporter created
 
$ kubectl get nodes --no-headers | wc -l
4
$ kubectl get ds node-exporter -n monitoring
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-exporter   3         3         3       3            3           <none>          25s
 
$ kubectl patch ds node-exporter -n monitoring --type merge \
  -p '{"spec":{"template":{"spec":{"tolerations":[{"operator":"Exists"}]}}}}'
daemonset.apps/node-exporter patched
$ kubectl get ds node-exporter -n monitoring
NAME            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
node-exporter   4         4         4       4            4           <none>          71s
 
$ kubectl cordon worker-2
node/worker-2 cordoned
$ kubectl delete pod node-exporter-6c9fx -n monitoring
pod "node-exporter-6c9fx" deleted
$ kubectl get pods -n monitoring -o wide -l app=node-exporter --field-selector spec.nodeName=worker-2
NAME                  READY   STATUS    RESTARTS   AGE   IP          NODE       NOMINATED NODE   READINESS GATES
node-exporter-tq7vn   1/1     Running   0          8s    10.0.0.12   worker-2   <none>           <none>
 
$ kubectl uncordon worker-2
node/worker-2 uncordoned

Takeaway

DaemonSets run one pod per matching node for agents and exporters. They ignore ordinary replica counts and follow node lifecycle.

Quick check
01A DaemonSet pod is Running on every node except one, where it stays Pending. kubectl describe pod on the stuck one shows a FailedScheduling event. What's the most likely cause?
Incorrect — No. That was the old behavior before v1.12. Today the kube-scheduler places daemon pods, and a crashed scheduler would stall pods on every node, not just one.
Correct — Since v1.12 the scheduler places DaemonSet pods, so they respect a node's free capacity, its taints, and priority. A single Pending daemon pod with a FailedScheduling event is almost always that one node being full or wearing an untolerated taint. Free up room or add the toleration.
Incorrect — No. A DaemonSet places one pod per eligible node regardless of what else runs there. There's no rule that excludes nodes for being occupied.
Incorrect — No. RollingUpdate replaces pods node by node under maxUnavailable, and updated pods reach Running, not Pending. A stuck Pending pod is a scheduling problem, not a deliberate canary.
02Your node-exporter DaemonSet uses hostNetwork and a hostPort. Why can't you switch its rollout to maxSurge for zero-gap coverage?
Correct — maxSurge needs the old and new pod to overlap briefly, which a host-port agent cannot do because the port is already taken.
Incorrect — No: DaemonSets do support maxSurge; the blocker here is the host port, not the field's existence.
Incorrect — No: skipping the CNI does not help; the port collision is what wedges the surged pod.
Incorrect — Backwards: maxSurge adds the new pod first; it is maxUnavailable that removes the old pod first.
03A node-exporter DaemonSet has no tolerations block. The cluster has 3 workers and 1 control-plane node. kubectl get ds shows DESIRED 3, CURRENT 3, all Running. Why isn't DESIRED 4, and what's the consequence?
Incorrect — No: they can run there if they tolerate the taint; skipping the control-plane node is a gap, not the design.
Incorrect — No: all three workers are covered and Running; the excluded fourth node is the control-plane node, not a slow worker.
Incorrect — No: nothing here runs it as a static pod; the node is excluded because of an untolerated taint.
Correct — the controller only auto-tolerates the built-in node taints, not the control-plane taint, so you go blind on exactly the nodes running etcd and the API server.
Requests, limits & QoSStatic pods

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep