Taints & tolerations

Repelling pods from nodes, and opting back in.

You spin up a node with four GPUs, the specialized chips that train machine-learning models and the priciest hardware in most clusters. It costs more per hour than the rest of the cluster put together. Within a day a batch of ordinary web pods has drifted onto it and is chewing through the compute you were saving for real work. Nothing stopped them. By default the scheduler, the control-plane component that picks a node for every new Pod (the smallest thing Kubernetes runs, one or more containers scheduled together), treats every node as fair game. 'Expensive' and 'special' mean nothing to it. Taints are how you tell it otherwise.

Think of a taint as a 'wet paint, keep off' sign you hang on a bench. Most people read it and sit somewhere else. A toleration is the one painter allowed to sit there anyway. The node carries the sign. The pod carries the permission slip. That split is where most of the confusion lives, so hold onto it as we go.

What the three effects actually do

A taint has three parts: a key, an optional value, and an effect, written key=value:effect (say workload=gpu:NoSchedule). The effect is what changes behavior, and there are exactly three of them. NoSchedule is a hard filter: when a new pod comes through, the scheduler drops any node carrying a NoSchedule taint the pod doesn't tolerate, but pods already running there are left alone. PreferNoSchedule is the soft cousin, a scoring penalty the scheduler will happily ignore if no better node exists. NoExecute is the strict one: it blocks new pods and evicts running pods that don't tolerate it. One rule people forget: a node can carry several taints at once, and a pod has to tolerate every NoSchedule and NoExecute taint on that node before it's even a candidate. Miss one and the node is out.

kubectl taint nodes worker-2 workload=gpu:NoSchedule

node/worker-2 tainted

kubectl describe node worker-2 | grep -A1 Taints

Taints:             workload=gpu:NoSchedule
Unschedulable:      false

A toleration is a permission slip, not a magnet

A toleration goes on the pod and mirrors the taint it answers to: the same key, an operator, usually a value, and the effect it forgives. The operator is either Equal, where key, value and effect all must match, or Exists, which matches on the key alone whatever the value. Watch the Exists form closely. A toleration with operator Exists and no key at all tolerates every taint in the cluster, which quietly lets that pod schedule onto your control-plane nodes and your half-dead ones too. People paste that in to 'make scheduling work' and regret it later.

Now the part that trips everyone up. A toleration only removes repulsion. It never pulls a pod toward the tainted node. A pod that tolerates the GPU taint is allowed on the GPU node, and it's equally allowed on every plain node, so it'll often land somewhere ordinary and leave the expensive hardware idle. To actually pin a workload to special nodes you need both halves at once: taint the nodes so nothing else approaches, and add node affinity or a nodeSelector so your pods are drawn in. Affinity matches node labels, which are a completely separate system from taints, so in practice the node ends up both labeled workload=gpu and tainted workload=gpu:NoSchedule. Same string, two different mechanisms doing two different jobs.

kubectl label nodes worker-2 workload=gpu

node/worker-2 labeled

apiVersion: v1
kind: Pod
metadata:
  name: gpu-train
spec:
  tolerations:
  - key: workload
    operator: Equal
    value: gpu
    effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: workload
            operator: In
            values: ["gpu"]
  containers:
  - name: trainer
    image: nvcr.io/nvidia/pytorch:24.03-py3
    command: ["sleep", "3600"]

kubectl apply -f gpu-train.yaml
kubectl get pod gpu-train -o wide

pod/gpu-train created
NAME        READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
gpu-train   1/1     Running   0          11s   10.244.2.9   worker-2   <none>           <none>

Who enforces this: the scheduler and the taint manager

None of this is magic, and it helps to know who does what. Two components split the work. The scheduler runs a plugin called TaintToleration: in its filter phase it throws out nodes with an untolerated NoSchedule or NoExecute taint, and in scoring it docks points from PreferNoSchedule nodes. That covers placing new pods. Evicting pods that are already running is a different job, owned by the taint manager inside kube-controller-manager (the control-plane process that runs the background reconcile loops). When a node stops reporting healthy, the node controller stamps it with built-in taints like node.kubernetes.io/not-ready or node.kubernetes.io/unreachable at NoExecute, and the taint manager drains the pods that don't tolerate them.

Kubernetes uses this pattern everywhere. Control-plane nodes ship pre-tainted with node-role.kubernetes.io/control-plane:NoSchedule, which is the reason your app pods never land there. DaemonSet pods (one copy per node, for things like the network plugin) get the standard node-condition taints tolerated for them, so a node under memory pressure or briefly unreachable doesn't drop its per-node agents. And every pod you create gets two tolerations added for you automatically. Go look for them.

kubectl get pod gpu-train -o jsonpath='{range .spec.tolerations[*]}{.key}{" -> "}{.effect}{" / "}{.tolerationSeconds}{"\n"}{end}'

workload -> NoSchedule / 
node.kubernetes.io/not-ready -> NoExecute / 300
node.kubernetes.io/unreachable -> NoExecute / 300

When it breaks: reading the Pending event

When a pod won't schedule and you suspect a taint, don't guess at it. Describe the pod and read the event the scheduler leaves behind.

kubectl describe pod web-portal-6d5f9

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  18s   default-scheduler  0/3 nodes are available: 1 node(s) had untolerated taint {workload: gpu}, 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Read that literally. One node rejected the pod for an untolerated taint (that's the GPU box), and two rejected it on affinity. The taint line names the exact key that blocked you, so you can tell whether you're missing a toleration or you've tainted a node you never meant to touch. To see a node's taints straight, run kubectl describe node and check the Taints field. And to lift a taint you added, repeat the command with a trailing dash: kubectl taint nodes worker-2 workload=gpu:NoSchedule- and it answers node/worker-2 untainted.

There is a working habit buried in all this. If a placement rule matters, write it somewhere the scheduler can read it. 'Please don't put anything on node-7' in a team chat is not a rule, it's a hope, and the next person to deploy has never seen the message. A taint is the same instruction in a form the cluster enforces. Just check which effect you're setting before you apply one to a node that is already busy: NoSchedule leaves the pods already there alone, and NoExecute does not.

Try this

Label and taint a worker, try to schedule a pod that is pinned to it but carries no toleration, then apply one that does and watch it land. Remove the taint and the label when you finish.

$ kubectl label nodes worker-2 workload=gpu
$ kubectl taint nodes worker-2 workload=gpu:NoSchedule
 
$ kubectl describe node worker-2 | grep -A1 Taints
 
# pinned to the node by nodeSelector, but with no toleration: this one should sit Pending
$ kubectl run plain-web --image=nginx --overrides='{"spec":{"nodeSelector":{"workload":"gpu"}}}'
$ kubectl get pod plain-web
$ kubectl describe pod plain-web | grep -A5 Events
 
# gpu-train.yaml carries the matching toleration, so this one lands
$ kubectl apply -f gpu-train.yaml
$ kubectl get pod gpu-train -o wide
 
$ kubectl get pod gpu-train -o jsonpath='{range .spec.tolerations[*]}{.key}{" -> "}{.effect}{" / "}{.tolerationSeconds}{"\n"}{end}'
 
# clean up
$ kubectl delete pod plain-web gpu-train
$ kubectl taint nodes worker-2 workload=gpu:NoSchedule-
$ kubectl label nodes worker-2 workload-

Takeaway

Taints repel; tolerations opt in. Control-plane taints exist so random Deployments do not sit on etcd nodes.

Related