>_SecOpsLog_
CoursesKubernetes administrationNode affinity & selectors

Node affinity & selectors

Attracting pods to the right nodes.

Intermediate12 min · lesson 24 of 65
In plain terms
If a taint says “keep off,” affinity says “come sit here.” It’s a magnet that pulls the right pods toward nodes with the right label — say, the ones with fast SSDs.

Your cluster is not a pile of identical machines. One node has a fast local SSD, the solid-state kind of disk. Another carries a GPU, a graphics chip that heavy jobs like machine learning lean on. A third lives in a different availability zone (a separate datacenter location) with room to spare. The scheduler, the control-plane program that picks a node for every new Pod (the smallest thing Kubernetes runs, usually a single container), does check that a node has the CPU and memory the Pod asked for and is not fenced off. Past that, nothing in an ordinary Pod spec says the SSD machine is any different from the other two, so the scheduler treats them as interchangeable and drops your Pod wherever it fits. Node affinity is how you overrule that and say which machines a Pod actually belongs on.

Think of the signs on a hospital's wards: Cardiology, ICU, has-an-MRI. A referral slip that reads 'must go to a ward with an MRI' routes the patient without naming a specific room. Node labels work the same way. You tag nodes with key/value pairs like disktype=ssd, and a Pod carries a slip that says which tags its node must have. The simplest slip is nodeSelector: a plain map of labels the node is required to have. If the node has them, the Pod can land there. If not, it can't. That's the entire feature.

Before any Pod can ask for a label, that label has to exist on a node. You add it with kubectl label (kubectl is the command-line tool you use to drive the cluster), then read it back to be sure it stuck. Half the affinity problems you'll ever debug come down to a label that was never applied, or one spelled differently than the Pod expects.

label-and-verify.sh
kubectl label nodes worker-2 disktype=ssd
kubectl get nodes -L disktype
output.txt
node/worker-2 labeled
NAME       STATUS   ROLES    AGE   VERSION    DISKTYPE
worker-1   Ready    <none>   40d   v1.31.2
worker-2   Ready    <none>   40d   v1.31.2    ssd
worker-3   Ready    <none>   40d   v1.31.2

Now a Pod that will only ever run on an SSD node. nodeSelector goes right in the Pod spec as a map, and every key you list is mandatory. There's no ranking and no fallback here. Either a node has every label you named, or the Pod won't go there.

fast-store.yaml
apiVersion: v1
kind: Pod
metadata:
  name: fast-store
spec:
  nodeSelector:
    disktype: ssd
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
apply-and-check.txt
$ kubectl apply -f fast-store.yaml
pod/fast-store created
$ kubectl get pod fast-store -o wide
NAME         READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
fast-store   1/1     Running   0          6s    10.244.2.7   worker-2   <none>           <none>

It landed on worker-2, the one node carrying disktype=ssd. But nodeSelector can only do exact-match equality. The moment the requirement gets fuzzier than a single fixed value, say 'SSD or NVMe' (NVMe is an even faster class of disk), or 'prefer this rack but don't insist on it', you've outgrown it. Node affinity is the same idea with a real grammar.

Required and preferred

Node affinity comes in two strengths, and the choice between them is a decision you'll make over and over. requiredDuringSchedulingIgnoredDuringExecution is a hard filter. The scheduler throws out every node that fails the rule before it even scores the survivors, exactly like nodeSelector, but with operators: In, NotIn, Exists, DoesNotExist, plus Gt and Lt for numeric comparisons. preferredDuringSchedulingIgnoredDuringExecution is a soft nudge. Each rule carries a weight from 1 to 100, and during the scoring phase the scheduler adds that weight to the nodes that match. A preferred rule never removes a node from the running, so it can never cause a Pod to go unplaced. It only tips the ranking.

Use required for a genuine constraint: this database must sit on the nodes with the local SSDs, or must stay in the region where its data lives. Labels are the right tool when the thing you care about is a fact about the machine that a human wrote down. A job that needs a GPU is a different case. It asks for one as a resource, nvidia.com/gpu: 1 in the container’s limits, and the scheduler already keeps it off every node that has no GPU to hand out. You bring labels into that only when you care which model of GPU it lands on. Use preferred for anything you’d like but can live without: pack Pods into the zone with spare capacity, but run anywhere if that zone is full. Required rules can leave a Pod Pending forever. Preferred rules never can. That single sentence decides the field for you most of the time.

One structural detail trips up almost everyone, so learn it before you need it. Under required affinity, nodeSelectorTerms is a list, and the terms are OR'd together: a node passes if it satisfies any one term. Inside a single term, the matchExpressions are AND'd: the node must satisfy all of them. And inside one expression, the values are OR'd. So 'SSD and in zone-a, both mandatory' means two matchExpressions under one term, not two separate terms.

fast-store-affinity.yaml
apiVersion: v1
kind: Pod
metadata:
  name: fast-store-affinity
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In
            values: ["ssd", "nvme"]
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 60
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["us-east-1a"]
  containers:
  - name: app
    image: registry.k8s.io/pause:3.10
verify-placement.txt
$ kubectl apply -f fast-store-affinity.yaml
pod/fast-store-affinity created
$ kubectl describe pod fast-store-affinity | grep -A4 Events
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  8s    default-scheduler  Successfully assigned default/fast-store-affinity to worker-2

The Scheduled event is your proof the rule was honored: it names the node the scheduler bound the Pod to. If you ever doubt where a Pod ended up, that one line is the source of truth, not your memory of the YAML. Here the required rule threw out every node without disktype=ssd or disktype=nvme, which in this cluster leaves worker-2 and nothing else, so the weight-60 preference had no decision left to make. Label a second worker disktype=nvme and the preference becomes the tie-breaker, pulling the Pod toward whichever of the two sits in us-east-1a.

When it will not schedule

The failure mode for required affinity is quiet, which is exactly why it catches people. The Pod is accepted, the API server (the cluster's front desk, which records every request) writes it down, and then it just sits in Pending because no node passed the filter. There are no container logs to read, because no container ever started. The answer lives in the Pod's events, every time.

debug-pending.sh
kubectl get pod gpu-trainer
kubectl describe pod gpu-trainer | tail -5
output.txt
NAME          READY   STATUS    RESTARTS   AGE
gpu-trainer   0/1     Pending   0          40s
 
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  38s   default-scheduler  0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

That message names the cause outright: all three nodes were filtered out by the Pod’s affinity or selector. Now you check reality against intent. kubectl get nodes --show-labels dumps every label on every node, and nine times out of ten you’ll spot it there. The label is gpu=True on the Pod but gpu=true on the node, or the one machine that had the label got drained last week and nobody relabeled its replacement. Fix the label and the Pod schedules on the scheduler’s next pass, usually within a second or two. Relaxing the rule to preferred works too, but not on the Pod already sitting there: affinity is one of the fields Kubernetes will not let you change after a Pod is created, so you edit the manifest (or the Deployment template the Pod came from) and let a fresh Pod replace it.

Affinity decides where a Pod lands, not where it stays
Both rule names end in IgnoredDuringExecution, and that suffix is literal. The scheduler checks affinity once, at placement time, and never again. Relabel a node afterward, or fix a typo in its labels, and Pods already running there don't move. A Pod will happily keep running on a node that no longer matches its own required rule. If you genuinely need a Pod off a node, you evict it with kubectl drain, a delete, or a NoExecute taint, not by editing labels. The RequiredDuringExecution mode that would re-check live nodes was planned years ago and still hasn't shipped as of v1.31, so nothing continuously enforces node affinity.
Picking the right steering tool
You need this Pod on specific nodes
how firm is the requirement?
exact key=value, nothing fancy
nodeSelector
a plain map, hard filter, no operators; missing label means Pending
hard rule, richer matching
required...IgnoredDuringExecution
operators In/NotIn/Exists/Gt/Lt; no matching node means Pending
nice to have, must never block
preferred...IgnoredDuringExecution
weight 1 to 100 nudges the score; runs anywhere if needed
All three match on node labels, so the label must exist first (kubectl get nodes --show-labels). The only real question is how firm the requirement is, because that decides whether a missing label means Pending or just a lower score.

In practice you start at the top of that diagram and only move down when something forces you to. nodeSelector until one fixed value stops describing what you need, then required affinity for a set of acceptable values, then a preferred rule layered on for the part you would merely like. Writing fifteen lines of affinity where disktype: ssd would have done is a real cost, because the next person has to read it.

And a required rule that matches nothing leaves the Pod Pending for as long as you let it. Nothing in the cluster is broken. It is a line of YAML asking for a machine that does not exist, which is why nobody gets paged and the Pod can sit there for days.

One habit is worth forming while you are still choosing label names. Label a node for what it has, like disktype=ssd or gpu-model=a100, so that any machine with that hardware can wear the same label. Do not invent a label per workload, like runs=billing-api, and hang a single Pod off it. That is hand-picking the machine with extra steps, and the day that node goes away the Pod has nowhere else to go.

Try this

Run this on a cluster you can throw away. Label one worker disktype=ssd, apply fast-store-affinity.yaml, and confirm the Pod’s Scheduled event names that worker. Then break it on purpose, because that is where the learning is: delete the Pod, change the required value to something no node carries, apply it again, and watch it sit in Pending with FailedScheduling saying exactly why. You delete and re-apply rather than edit, because a Pod’s affinity is fixed once the Pod exists. For a second run, label another worker disktype=nvme and watch the weight-60 preference choose between two qualifying nodes.

terminal
$ kubectl label nodes worker-2 disktype=ssd
node/worker-2 labeled
$ kubectl get nodes -L disktype
NAME       STATUS   ROLES    AGE   VERSION    DISKTYPE
worker-1   Ready    <none>   40d   v1.31.2
worker-2   Ready    <none>   40d   v1.31.2    ssd
worker-3   Ready    <none>   40d   v1.31.2
 
$ kubectl apply -f fast-store-affinity.yaml
pod/fast-store-affinity created
$ kubectl get pod fast-store-affinity -o wide
NAME                  READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
fast-store-affinity   1/1     Running   0          6s    10.244.2.7   worker-2   <none>           <none>
 
# now break it on purpose: no node carries disktype=unobtainium
$ kubectl delete pod fast-store-affinity
pod "fast-store-affinity" deleted
# edit fast-store-affinity.yaml, set values: ["unobtainium"]
$ kubectl apply -f fast-store-affinity.yaml
pod/fast-store-affinity created
$ kubectl get pod fast-store-affinity
NAME                  READY   STATUS    RESTARTS   AGE
fast-store-affinity   0/1     Pending   0          15s
$ kubectl describe pod fast-store-affinity | tail -5
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  14s   default-scheduler  0/3 nodes are available: 3 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.

Takeaway

Node affinity attracts pods to labeled nodes. required means hard; preferred means soft scoring.

Quick check
01You want a Pod to run only on nodes that are both disktype=ssd AND in zone us-east-1a, as hard requirements. You write two entries under nodeSelectorTerms, one term matching disktype and a separate term matching the zone. Where can the Pod actually land?
Correct — The nodeSelectorTerms list is OR'd: a node satisfies the rule if it matches any single term. So an SSD node in the wrong zone, or a spinning-disk node that happens to be in us-east-1a, both pass. To force AND, put both matchExpressions inside one term.
Incorrect — This is what you intended, but not what the YAML says. Conditions are AND'd only when they sit as multiple matchExpressions under the same term. Two separate terms are treated as alternatives, not a combined requirement.
Incorrect — No conflict is detected. Kubernetes reads the two terms as two acceptable options, so the Pod schedules readily, just not where you meant it to.
02What is the practical difference between requiredDuringSchedulingIgnoredDuringExecution and preferredDuringSchedulingIgnoredDuringExecution node affinity?
Correct — required removes non-matching nodes, while preferred just nudges the ranking and can never strand a Pod.
Incorrect — both are IgnoredDuringExecution, so both are evaluated only at placement and never re-checked.
Incorrect — it's the reverse — required can strand a Pod in Pending, and preferred never removes a node from consideration.
Incorrect — both required and preferred support the operators In, NotIn, Exists, DoesNotExist, Gt, and Lt.
03A Pod was scheduled onto worker-2 by a required node affinity rule for disktype=ssd. Later someone removes the disktype=ssd label from worker-2, so it no longer matches what the Pod required. What happens to the already-running Pod?
Incorrect — affinity is evaluated only at scheduling time, so a running Pod is never re-checked or evicted for it.
Correct — the IgnoredDuringExecution suffix is literal — the scheduler checks affinity once at binding and never again.
Incorrect — nothing live-migrates a Pod for affinity; without a re-check it simply stays where it is.
Incorrect — a running Pod isn't reverted to Pending; the scheduler doesn't look at it again after binding.
Taints & tolerationsPod affinity & anti-affinity

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep