>_SecOpsLog_
CoursesKubernetes administrationNode management

Node management

cordon, drain, and safely removing a node.

Intermediate10 min · lesson 56 of 65
In plain terms
Retiring a node is closing a supermarket checkout lane: put up the “lane closed” sign (cordon), send the current shoppers to other lanes (drain), then do the maintenance — and reopen when you’re done.

A kernel patch just landed and you need to reboot worker-2, one of the machines actually running your containers. Reboot it right now and every pod on that box dies at the same instant. A pod is one or more containers that Kubernetes schedules and runs together as a single unit. Kill the node out from under them and there's no heads-up to the app, no chance to finish the requests already in flight. If two copies of your payments service happened to land on that same machine, the service drops hard. Kubernetes gives you two commands so this never happens by accident: cordon and drain. Get them right and users never notice the node stepped out.

A node is a checkout lane at a busy supermarket. Closing the lane is three moves. Put up the 'lane closed' sign so no new shoppers queue there. Walk the people already in line over to other lanes. Then send the cashier home. cordon is the sign. kubectl cordon marks the node unschedulable, so the scheduler stops placing work there. The scheduler is the component that picks which node each new pod runs on. Everything already on the node keeps running, untouched. drain is walking the shoppers over. kubectl drain cordons the node for you first, then evicts the pods on it, and the controller that owns each pod (its Deployment or StatefulSet) starts a fresh copy on another node. When the machine comes back from maintenance, kubectl uncordon takes the sign down.

Retiring a node safely
1cordonno new pods land here2drainevict running pods gracefully3maintainpatch, reboot, or replace4uncordonnode back in rotation
drain honors PodDisruptionBudgets and skips DaemonSet pods. If evicting a pod would break a budget it refuses and retries instead of forcing it. That refusal is the safety net working, not a failure.
terminal
$ kubectl cordon worker-2
$ kubectl get nodes
terminal
node/worker-2 cordoned
NAME       STATUS                     ROLES           AGE   VERSION
cp-1       Ready                      control-plane   40d   v1.31.2
worker-1   Ready                      <none>          40d   v1.31.2
worker-2   Ready,SchedulingDisabled   <none>          40d   v1.31.2

What a drain actually does

A drain doesn't just delete the pods and move on. If it did, it would run straight through every safety rule you'd set. Instead it calls the Eviction API, a careful cousin of delete. The API server, which is the cluster's front door for every command, checks each eviction against your disruption budgets before it lets the pod go. For each pod that clears that check, the kubelet on the node (the kubelet is the agent Kubernetes runs on every machine) sends the container SIGTERM, the 'please wrap up now' signal. It waits out the pod's terminationGracePeriodSeconds, 30 seconds by default, so work in flight can finish, then stops the container. The controller that owned the pod notices a replica went missing and asks for a replacement, which lands on some other Ready node because the scheduler already knows worker-2 is off limits. Two kinds of pods get handled differently. DaemonSet pods, the ones meant to run exactly one copy per node like a log shipper or a metrics agent, get skipped, because evicting them makes no sense when the whole node is about to leave. And a pod holding data in an emptyDir volume, which is scratch space that lives and dies with the node, blocks the drain until you say out loud that you're fine losing it.

terminal
$ kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data
terminal
node/worker-2 already cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-4x9qz, monitoring/node-exporter-7tp2k
evicting pod default/payments-api-6b8d9c7f4d-2wq5r
evicting pod default/web-5f7c9d8b6-lm4xp
evicting pod kube-system/coredns-7db6d8ff4d-9kk2v
pod/web-5f7c9d8b6-lm4xp evicted
pod/coredns-7db6d8ff4d-9kk2v evicted
pod/payments-api-6b8d9c7f4d-2wq5r evicted
node/worker-2 drained

Check your work before you touch any hardware. That Warning line in the drain output, the one naming the DaemonSet pods it left behind, is expected. It isn't an error. List what's still pinned to the node and confirm the only things left are those per-node pods.

terminal
$ kubectl get pods -A -o wide --field-selector spec.nodeName=worker-2
terminal
NAMESPACE     NAME                  READY   STATUS    RESTARTS   AGE   IP          NODE       NOMINATED NODE   READINESS GATES
kube-system   kube-proxy-4x9qz      1/1     Running   0          40d   10.0.0.12   worker-2   <none>           <none>
monitoring    node-exporter-7tp2k   1/1     Running   0          40d   10.0.0.12   worker-2   <none>           <none>

Two pods left, both DaemonSet-managed, both fine to leave running. That's the all-clear. Patch the host, reboot it, wait for the node to report Ready again, then lift the sign so the scheduler can start using it once more.

terminal
$ kubectl uncordon worker-2
terminal
node/worker-2 uncordoned

PodDisruptionBudgets keep drains honest

Here's the exact failure that catches teams out. A service runs three replicas, and by bad luck two of them sit on the same node. Drain that node with no guardrail and you drop to one replica in a blink, or zero if a drain is already running somewhere else in the fleet. A PodDisruptionBudget, PDB for short, is that guardrail. It's a rule pinned to the workload that says 'at least this many of us stay up, whatever maintenance you're running.' You set one of two knobs. minAvailable is a floor: never fewer than this many healthy. maxUnavailable is a ceiling: never more than this many down at once. The Eviction API reads the budget on every single eviction and refuses to take the workload below the line. drain then waits and retries until a replacement pod is Running somewhere else and there's room to evict the next one.

pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: payments-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: payments-api
terminal
$ kubectl apply -f pdb.yaml
$ kubectl get pdb payments-api
terminal
poddisruptionbudget.policy/payments-api created
NAME           MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
payments-api   2               N/A               1                     5s

ALLOWED DISRUPTIONS is the column to watch. It's how many pods the Eviction API will let go right now. A value of 1 means: evict one, wait for its replacement to come up healthy, then release the next. Roll that across a whole cluster and you can drain every node in turn without the service ever dipping below the capacity it promised.

When a drain hangs

A drain that sits there repeating the same line is usually not broken. It's the budget doing its job. When ALLOWED DISRUPTIONS is stuck at 0, the workload is already sitting on its floor, and the drain will keep spinning until a healthy replacement turns up somewhere else. The message it prints tells you exactly why.

terminal
$ kubectl drain worker-4 --ignore-daemonsets
terminal
node/worker-4 cordoned
evicting pod default/payments-api-6b8d9c7f4d-8kk2v
error when evicting pods/"payments-api-6b8d9c7f4d-8kk2v" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod default/payments-api-6b8d9c7f4d-8kk2v
error when evicting pods/"payments-api-6b8d9c7f4d-8kk2v" -n "default" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

When you see that, keep your hands off --force and --disable-eviction. Both skip the Eviction API and hard-delete the pods, which is precisely the outage the budget exists to prevent. The real fix is to give the workload somewhere to go. Work out why replacements aren't arriving. Maybe a new pod is stuck Pending because no other node has spare room. Maybe the PDB floor equals the replica count, so there was never any slack to begin with. Add a node, scale the app up, or loosen the budget, and the drain finishes on its own.

Bare pods and a too-tight budget both trap you
Only pods owned by a controller (a Deployment, StatefulSet, or ReplicaSet) get rescheduled after eviction. A pod you created directly, with no owner above it, just gets deleted and never comes back, so drain refuses to touch it unless you pass --force and accept it's gone for good. The tighter trap is a budget with no headroom. Set minAvailable equal to the replica count, say minAvailable: 3 on a three-replica app, and ALLOWED DISRUPTIONS sits at 0. The very first eviction is already illegal, so the drain deadlocks with nothing to wait for. Leave slack, or use maxUnavailable: 1, which keeps scaling sensibly as you add replicas.

PodDisruptionBudgets can block drains. That is safety. Coordinate with app owners instead of --force habits.

DaemonSets may be ignored on drain depending on flags. Know what you leave behind on the node.

Always uncordon or delete the node object when done. Forgotten cordons shrink the cluster silently. PodDisruptionBudgets can block drains. That is safety.

Try this

Cordon a worker, drain it with a grace period, watch pods reschedule, then uncordon. Use a namespace you can disrupt.

terminal
$ kubectl cordon worker-2
$ kubectl get nodes
 
$ kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data
 
$ node/worker-2 already cordoned
$ Warning: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-4x9qz, monitoring/node-exporter-7tp2k
$ evicting pod default/payments-api-6b8d9c7f4d-2wq5r
$ evicting pod default/web-5f7c9d8b6-lm4xp
$ evicting pod kube-system/coredns-7db6d8ff4d-9kk2v
$ pod/web-5f7c9d8b6-lm4xp evicted
$ pod/coredns-7db6d8ff4d-9kk2v evicted
$ pod/payments-api-6b8d9c7f4d-2wq5r evicted
$ node/worker-2 drained
 
$ kubectl get pods -A -o wide --field-selector spec.nodeName=worker-2
 
NAMESPACE     NAME                  READY   STATUS    RESTARTS   AGE   IP          NODE       NOMINATED NODE   READINESS GATES
$ kube-system   kube-proxy-4x9qz      1/1     Running   0          40d   10.0.0.12   worker-2   <none>           <none>
$ monitoring    node-exporter-7tp2k   1/1     Running   0          40d   10.0.0.12   worker-2   <none>           <none>
 
$ kubectl uncordon worker-2

Takeaway

Cordon stops new pods; drain evicts politely. That is how you remove a node without a surprise outage.

Quick check
01A kubectl drain on worker-4 keeps printing 'Cannot evict pod as it would violate the pod's disruption budget' and never finishes. The app has 3 replicas and a PDB with minAvailable: 3. What's the correct fix?
Incorrect — that bypasses the Eviction API and hard-deletes the pods, causing the exact outage the budget exists to prevent. Never the move for a healthy budget.
Correct — minAvailable: 3 on a three-replica app leaves ALLOWED DISRUPTIONS at 0, so no eviction is ever legal. Add headroom and the drain completes on its own.
Incorrect — that works mechanically but throws away the safety net for this and every future disruption. Fix the slack, don't remove the guardrail.
Incorrect — it won't. With the floor equal to the replica count there is no cluster state where an eviction is allowed. It's a deadlock, not a delay.
02You run kubectl cordon worker-2 and its STATUS becomes 'Ready,SchedulingDisabled'. What happens to the Pods already running on worker-2?
Correct — cordon marks the node unschedulable, but evicting the existing Pods is drain's job, not cordon's.
Incorrect — that is what drain does; cordon on its own moves nothing.
Incorrect — graceful termination happens during a drain or eviction, not on a bare cordon.
Incorrect — cordon does not touch Service endpoints or networking; the Pods keep serving traffic normally.
03worker-3 hosts a Pod you created directly with kubectl run, with no Deployment or other controller above it. You run kubectl drain worker-3. What happens to that Pod?
Incorrect — only controller-owned Pods get recreated; a bare Pod has nothing above it to bring it back.
Incorrect — a bare Pod is not rescheduled at all, and emptyDir data never survives a Pod moving nodes.
Incorrect — Kubernetes has no live Pod migration; eviction deletes the Pod and, only for owned Pods, recreates it elsewhere.
Correct — an unmanaged Pod has no controller to recreate it, so drain blocks unless you explicitly accept losing it.
Debugging running podsOS upgrades & maintenance

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep