>_SecOpsLog_
CoursesKubernetes administrationPod affinity & anti-affinity

Pod affinity & anti-affinity

Placing pods relative to other pods.

Advanced10 min · lesson 25 of 65
In plain terms
Pod affinity places pods relative to each other: keep the cache right next to the app it serves (together), or spread the copies across different racks so one failure can’t take them all out (apart).

Two of your three web replicas landed on the same node. Nobody noticed until that node drained for a kernel patch at 2am, and two-thirds of your frontend went dark at once. The scheduler didn't do anything wrong. You never told it to keep those copies apart. That's the job pod affinity and anti-affinity do. They place a Pod (Kubernetes' smallest deployable unit, one or more containers that share an IP address) relative to other pods, not relative to node labels. Two flavors. Affinity pulls a pod toward where certain pods already run. Anti-affinity pushes it away. One buys you performance. The other buys you survival.

Attract, repel, and what "same" means

Think about seating at a big wedding. The planner works with two kinds of rules. "Put the band next to the dance floor" is a togetherness rule. "Never seat the two feuding uncles at the same table" is a keep-apart rule. Kubernetes has both. Pod affinity is the band-and-dance-floor rule: schedule this pod where pods matching a label already run, like a cache sitting close to the app that keeps hitting it. Pod anti-affinity is the feuding-uncles rule: keep this pod off any node that already runs a matching pod, so a service's replicas fan out instead of piling onto one machine. The word doing the quiet heavy lifting is topologyKey, and it answers a single question: same what? A table at the wedding could mean one physical table, or the whole west wing of the hall. In a cluster, kubernetes.io/hostname means "the same node," and topology.kubernetes.io/zone means "the same availability zone" (a separate failure domain in your cloud region, with its own power and network). topologyKey has to be a label that actually exists on your nodes. The scheduler groups nodes by that label's value, then applies your rule inside each group. Pick the wrong domain and you'll spread across nodes when you meant to survive a whole zone going down, or the reverse.

web-deploy.yaml (one web pod per node)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: web
              topologyKey: kubernetes.io/hostname
      containers:
        - name: web
          image: nginx:1.27
apply it, then confirm the spread
$ kubectl apply -f web-deploy.yaml
deployment.apps/web created
 
$ kubectl get pods -l app=web -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
web-6f9c8b7d5f-4kq2p   1/1     Running   0          15s   10.244.1.23   node-a   <none>           <none>
web-6f9c8b7d5f-lm8xn   1/1     Running   0          15s   10.244.2.31   node-b   <none>           <none>
web-6f9c8b7d5f-t7v6c   1/1     Running   0          15s   10.244.3.19   node-c   <none>           <none>

How the scheduler actually reads the rule

When you submit a pod, the scheduler makes two passes over every node. The first pass filters: it drops nodes that can't work, meaning not enough CPU or memory, taints the pod won't tolerate, and yes, anti-affinity violations. The second pass scores the survivors and ranks them, and preferred rules add their points here. Hard anti-affinity lives in that first pass. For each candidate node, the scheduler looks at the pods already scheduled in that node's topology group, and if any of them match your labelSelector, the node is out. This is why the feature costs more than node affinity. Node affinity compares a node against fixed labels. Pod affinity compares a node against the live placement of every other matching pod, so the work grows with how many pods you're checking against. On big clusters the docs warn against these once you get past a few hundred nodes, because the extra processing slows scheduling down, so keep the rules for workloads that truly need them. The long field name is trying to tell you something too. requiredDuringSchedulingIgnoredDuringExecution means the rule is enforced when the pod is placed and ignored once it's running. If a matching pod shows up in the same group later, or someone relabels things, Kubernetes won't evict your already-running pod to fix it. It's a scheduling-time promise, nothing more. One more detail people miss: labelSelector only matches pods in the same namespace by default. To reach across namespaces you set namespaceSelector, or list the namespaces explicitly.

Here's the failure everyone trips over once. Scale that deployment to four replicas on a three-node cluster, and the fourth pod has nowhere legal to land.

scale past the node count and it strands
$ kubectl scale deployment web --replicas=4
deployment.apps/web scaled
 
$ kubectl get pods -l app=web
NAME                   READY   STATUS    RESTARTS   AGE
web-6f9c8b7d5f-4kq2p   1/1     Running   0          3m
web-6f9c8b7d5f-lm8xn   1/1     Running   0          3m
web-6f9c8b7d5f-t7v6c   1/1     Running   0          3m
web-6f9c8b7d5f-zp4rd   0/1     Pending   0          22s
 
$ kubectl describe pod web-6f9c8b7d5f-zp4rd | sed -n '/Events/,$p'
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  24s   default-scheduler  0/3 nodes are available: 3 node(s)
    didn't match pod anti-affinity rules. preemption: 0/3 nodes are available: 3 No
    preemption victims found for incoming pod.

Read that event closely, because it's doing your debugging for you. "0/3 nodes are available: 3 node(s) didn't match pod anti-affinity rules" tells you the filter pass rejected every node for one reason, and the reason is your own rule. Not resources. Not taints. When a pod is stuck Pending, kubectl describe pod always names the filter that killed it, and here it points straight back at the hostname anti-affinity. The second you see "didn't match pod anti-affinity rules," count your schedulable nodes against your replica count. If replicas beat nodes and the rule is hard, that's the whole story, and no amount of waiting will fix it.

Soft rules, and pulling pods together

The fix isn't to give up on spreading. You just tell the scheduler you'd strongly prefer it instead of demanding it. The preferred form turns the rule from a filter into a score. Each term carries a weight from 1 to 100, and the scheduler adds that weight to the nodes that satisfy the term, then still places the pod somewhere even when nothing satisfies it. So four replicas on three nodes spread as evenly as they can, and the fourth doubles up on the least-loaded node instead of sitting Pending. You get good spread on a normal day and you keep running on a bad one. For pure availability spreading, a lot of teams now reach for topology spread constraints (the next lesson), which were built for exactly this, but preferred anti-affinity is the older, widely used tool and still works fine.

switch the anti-affinity to soft
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    app: web
                topologyKey: kubernetes.io/hostname
the 4th pod now runs instead of pending
$ kubectl apply -f web-deploy.yaml
deployment.apps/web configured
 
$ kubectl get pods -l app=web -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
web-7c4d9f6b88-2wq5r   1/1     Running   0          40s   10.244.1.24   node-a   <none>           <none>
web-7c4d9f6b88-6nkt8   1/1     Running   0          40s   10.244.2.33   node-b   <none>           <none>
web-7c4d9f6b88-h9zpl   1/1     Running   0          40s   10.244.3.21   node-c   <none>           <none>
web-7c4d9f6b88-x4m7d   1/1     Running   0          38s   10.244.1.25   node-a   <none>           <none>

The attract side, podAffinity, uses the same shape with the selector pointed at the pods you want to sit near. Say you run a small cache next to each web pod so lookups stay on the same machine and skip a network hop. You require the cache onto nodes already running app: web, over the hostname domain, and it snaps into place beside them.

web-cache.yaml (co-locate a cache with web)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-cache
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-cache
  template:
    metadata:
      labels:
        app: web-cache
    spec:
      affinity:
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchLabels:
                  app: web
              topologyKey: kubernetes.io/hostname
      containers:
        - name: cache
          image: redis:7
cache lands on the same nodes as web
$ kubectl apply -f web-cache.yaml
deployment.apps/web-cache created
 
$ kubectl get pods -o wide -l 'app in (web,web-cache)' --sort-by=.spec.nodeName
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
web-7c4d9f6b88-2wq5r        1/1     Running   0          5m    10.244.1.24   node-a   <none>           <none>
web-cache-5d8f7c9b4-qm2rt   1/1     Running   0          25s   10.244.1.28   node-a   <none>           <none>
web-7c4d9f6b88-6nkt8        1/1     Running   0          5m    10.244.2.33   node-b   <none>           <none>
web-cache-5d8f7c9b4-lx9vp   1/1     Running   0          25s   10.244.2.37   node-b   <none>           <none>
web-7c4d9f6b88-h9zpl        1/1     Running   0          5m    10.244.3.21   node-c   <none>           <none>
web-cache-5d8f7c9b4-w7kd2   1/1     Running   0          25s   10.244.3.24   node-c   <none>           <none>
Which placement rule do you actually want?
Placing a pod relative to other pods
pick the intent first, then hard vs soft
together
podAffinity
cache next to its app; same shape, selector points at the pods to join
apart, must never share
required anti-affinity
one per domain; needs schedulable nodes >= replicas or the surplus stays Pending
apart, best effort
preferred anti-affinity or topology spread
weight 1-100; spreads when it can, degrades gracefully when it can't
Intent decides the field; your risk tolerance decides required vs preferred. Reach for hard only when co-location is truly forbidden.
A hard anti-affinity rule caps your replicas at your node count
requiredDuringScheduling anti-affinity on kubernetes.io/hostname means exactly one pod per node, full stop. Run more replicas than you have schedulable nodes and the surplus sits Pending forever, and it tends to strike silently the next time a HorizontalPodAutoscaler scales you up, or a node goes NotReady and shrinks the pool. Two quieter versions of the same trap: if the topologyKey label is missing from some nodes, the scheduler can't group them and they drop out of consideration; and the labelSelector normally has to match the pod's own label (app: web selecting app: web), so a typo there makes the rule match nothing and silently spread zero pods. For availability, prefer the soft form or topology spread constraints so a full domain degrades instead of stalling a rollout.

Hard anti-affinity with too few domains leaves pods Pending. Soft preferred rules often fit real capacity better.

Co-locating chatty pods can cut latency and also create correlated failure. Pick consciously.

Label the targets you affinity toward. Orphan labels make rules silently match nothing.

Try this

Run a datastore pod with a role label, then schedule an app with pod affinity to that label on the same topology key. Add anti-affinity for the app replicas.

terminal
$ kubectl apply -f web-deploy.yaml
deployment.apps/web created
 
$ kubectl get pods -l app=web -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
web-6f9c8b7d5f-4kq2p   1/1     Running   0          15s   10.244.1.23   node-a   <none>           <none>
web-6f9c8b7d5f-lm8xn   1/1     Running   0          15s   10.244.2.31   node-b   <none>           <none>
web-6f9c8b7d5f-t7v6c   1/1     Running   0          15s   10.244.3.19   node-c   <none>           <none>
 
$ kubectl scale deployment web --replicas=4
deployment.apps/web scaled
 
$ kubectl get pods -l app=web
NAME                   READY   STATUS    RESTARTS   AGE
web-6f9c8b7d5f-4kq2p   1/1     Running   0          3m
web-6f9c8b7d5f-lm8xn   1/1     Running   0          3m
web-6f9c8b7d5f-t7v6c   1/1     Running   0          3m
web-6f9c8b7d5f-zp4rd   0/1     Pending   0          22s
 
$ kubectl describe pod web-6f9c8b7d5f-zp4rd | sed -n '/Events/,$p'
Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  24s   default-scheduler  0/3 nodes are available: 3 node(s)
    didn't match pod anti-affinity rules. preemption: 0/3 nodes are available: 3 No
    preemption victims found for incoming pod.
 
$ kubectl apply -f web-deploy.yaml
deployment.apps/web configured
 
$ kubectl get pods -l app=web -o wide
NAME                   READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
web-7c4d9f6b88-2wq5r   1/1     Running   0          40s   10.244.1.24   node-a   <none>           <none>
web-7c4d9f6b88-6nkt8   1/1     Running   0          40s   10.244.2.33   node-b   <none>           <none>
web-7c4d9f6b88-h9zpl   1/1     Running   0          40s   10.244.3.21   node-c   <none>           <none>
web-7c4d9f6b88-x4m7d   1/1     Running   0          38s   10.244.1.25   node-a   <none>           <none>
 
$ kubectl apply -f web-cache.yaml
deployment.apps/web-cache created
 
$ kubectl get pods -o wide -l 'app in (web,web-cache)' --sort-by=.spec.nodeName
NAME                        READY   STATUS    RESTARTS   AGE   IP            NODE     NOMINATED NODE   READINESS GATES
web-7c4d9f6b88-2wq5r        1/1     Running   0          5m    10.244.1.24   node-a   <none>           <none>
web-cache-5d8f7c9b4-qm2rt   1/1     Running   0          25s   10.244.1.28   node-a   <none>           <none>
web-7c4d9f6b88-6nkt8        1/1     Running   0          5m    10.244.2.33   node-b   <none>           <none>
web-cache-5d8f7c9b4-lx9vp   1/1     Running   0          25s   10.244.2.37   node-b   <none>           <none>
web-7c4d9f6b88-h9zpl        1/1     Running   0          5m    10.244.3.21   node-c   <none>           <none>
web-cache-5d8f7c9b4-w7kd2   1/1     Running   0          25s   10.244.3.24   node-c   <none>           <none>

Takeaway

Pod affinity places relative to other pods; anti-affinity spreads or isolates. Topology keys decide the domain — node, zone, or region.

Quick check
01A StatefulSet runs 5 replicas with required (hard) pod anti-affinity on kubernetes.io/hostname. The cluster has 4 worker nodes. Four pods are Running and the fifth is stuck Pending. What's happening, and what's the right fix?
Correct — One-per-node across 4 nodes places 4 pods; the 5th is unschedulable by design. kubectl describe pod names it: "didn't match pod anti-affinity rules." Fix with capacity or a soft rule.
Incorrect — describe pod shows a FailedScheduling anti-affinity message, not a scheduler outage. Restarting the scheduler changes nothing about node count versus replicas.
Incorrect — A missing image shows ImagePullBackOff, not Pending, and the pod isn't bound to any node yet. The event blames anti-affinity, not the image.
Incorrect — kubernetes.io/hostname is a standard, valid topologyKey. The real constraint is replicas exceeding the number of hostname buckets under a hard rule.
02The rule is written requiredDuringSchedulingIgnoredDuringExecution. In practice, what does the 'IgnoredDuringExecution' half mean?
Incorrect — and it's the opposite: the rule is never re-enforced after placement, so a running pod is never evicted to fix a violation.
Incorrect — That describes the preferred form; the required form is a hard filter at scheduling time, not a preference.
Correct — it is a scheduling-time promise only, so relabeling or a new matching pod never evicts a running pod.
Incorrect — Backwards: the rule binds at scheduling and is dropped during execution, not the reverse.
03You apply the web-cache Deployment (required podAffinity selecting app: web over kubernetes.io/hostname) to a fresh cluster where no app: web pod exists yet. What happens to the cache pods?
Incorrect — No: an unmatched required podAffinity term is not a no-op; it has no pod to attract toward, so it cannot be satisfied.
Correct — podAffinity attracts toward pods that already run, so with zero matches no node satisfies the rule and the pods wait.
Incorrect — No: the scheduler never creates other workloads; affinity places only the pod you submitted, it does not conjure targets.
Incorrect — No: nothing packs them together; without a matching target the required term simply cannot be met and they stay Pending.
Node affinity & selectorsTopology spread & pod priority

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep