Topology spread & pod priority

Even spreading, and who wins under pressure.

Three replicas of your web app, all running, all healthy. Then the cloud provider takes one availability zone offline for maintenance, and all three go dark at the same instant, because the scheduler happened to pack them onto nodes in that one zone. Kubernetes spreads pods around a little on its own, but "a little" is not a promise. When you actually need copies fanned out across separate failure domains, you have to say so out loud. The tool for saying so is topology spread constraints.

Think of a caterer setting chairs for a wedding. You don't want forty chairs jammed at one table and four at the next. You want them even, so if a waiter drops a tray at one table the whole reception isn't ruined. A topology spread constraint is you telling the scheduler to keep roughly the same number of pods in every failure domain. The scheduler is the control-plane component that decides which node each Pod runs on. A Pod is the smallest thing Kubernetes runs: one or more containers that share an address and live and die together. A failure domain is just a set of nodes that can go down together, like every node in one cloud availability zone, or a single physical machine.

Three knobs: maxSkew, topologyKey, whenUnsatisfiable

You describe the spread with three fields. topologyKey names the kind of domain to spread over, and it's matched against a label on each node. Use topology.kubernetes.io/zone to spread across zones, or kubernetes.io/hostname to spread across individual machines. maxSkew is how lopsided you're willing to let things get. The scheduler counts the matching pods in each domain, and the skew is the gap between the fullest domain and the emptiest. Set maxSkew to 1 across three zones and the counts stay within one of each other. whenUnsatisfiable is the field that bites people. DoNotSchedule is a hard rule: if placing a pod would push the skew past maxSkew, no node passes the filter and the pod just waits. ScheduleAnyway is soft: the scheduler leans toward the emptiest domain but will place the pod somewhere rather than leave it stuck.

apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
  replicas: 6
  selector: { matchLabels: { app: web } }
  template:
    metadata: { labels: { app: web } }
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: DoNotSchedule
          labelSelector: { matchLabels: { app: web } }
      containers:
        - { name: web, image: nginx:1.27 }

$ kubectl apply -f deployment.yaml
deployment.apps/web created
$ kubectl get pods -l app=web -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName
POD                   NODE
web-6bf4c9d8f-2xq4h   node-a1
web-6bf4c9d8f-8fp2k   node-a2
web-6bf4c9d8f-lm7dr   node-b1
web-6bf4c9d8f-p4ktz   node-b2
web-6bf4c9d8f-r9wzx   node-c1
web-6bf4c9d8f-tv6hq   node-c2

Read that back. node-a1 and node-a2 sit in zone a, the two b nodes in zone b, the two c nodes in zone c. So the six pods landed 2/2/2, a skew of zero. That's the win. If you ever want to check the skew by hand, count the pods in each domain and subtract the smallest count from the largest. The scheduler runs that same little bit of arithmetic on every placement decision it makes.

Under the hood this is the PodTopologySpread plugin inside the scheduler, and it works in one of two phases depending on your policy. With DoNotSchedule it runs during filtering, throwing out any node whose domain is already too full to accept another pod. With ScheduleAnyway it runs during scoring, handing higher scores to nodes in emptier domains so they win close calls. Two newer fields pull their weight in production. minDomains sets a floor on how many domains must actually hold pods, so the scheduler can't call things "even" when everything piled into the one zone that had room; it only applies with DoNotSchedule. And matchLabelKeys: [pod-template-hash] tells the constraint to count only the pods from the current rollout. A Deployment manages its pods through a ReplicaSet, a controller that keeps a fixed number of identical pods alive, and every rollout spins up a new one. Without matchLabelKeys, the old ReplicaSet's pods still count against the new ones, and a rolling update can wedge itself half-finished.

Who wins when the cluster is full

Sooner or later the cluster fills up and something that really matters can't get a slot. A busy restaurant handles this every Friday night. Walk-ins take whatever table opens, but a guest with a standing reservation can bump them. Pod priority is that reservation. You create a PriorityClass, give it a number, and any pod that names it carries that number. When a high-priority pod can't schedule because every node is full, the scheduler can preempt: evict one or more lower-priority pods to free up room, then place the important one. Preempt is just a polite word for the scheduler deleting a running pod to make space.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata: { name: high-priority }
value: 1000000
globalDefault: false
description: "Payment-path services"
---
apiVersion: v1
kind: Pod
metadata: { name: payments-api, namespace: payments }
spec:
  priorityClassName: high-priority
  containers:
    - { name: api, image: payments:2.3 }

$ kubectl apply -f priorityclass.yaml
priorityclass.scheduling.k8s.io/high-priority created
pod/payments-api created
$ kubectl get priorityclass
NAME                      VALUE        GLOBAL-DEFAULT   AGE
high-priority             1000000      false            6s
system-cluster-critical   2000000000   false            51d
system-node-critical      2000001000   false            51d

The ceiling for a class you create yourself is one billion. The two built-in classes, system-cluster-critical and system-node-critical, sit above that on purpose, so control-plane pods always outrank anything you write. When preemption fires, the scheduler picks a node, then chooses the lowest-priority victims that free up just enough room. It respects PodDisruptionBudgets where it can, though it will override one as a last resort if there's no other way. It sets nominatedNodeName on the waiting pod so you can see where it's headed, and the victims get their normal graceful shutdown instead of a hard kill. Two more things worth filing away. Set preemptionPolicy: Never and the pod jumps the scheduling queue by priority but never evicts anyone, which is right for work that's important yet not worth killing other pods over. And priority is not the same machinery as node-pressure eviction. When a node itself runs low on memory, the kubelet (the agent that runs on every node) starts evicting pods, weighing each pod's Quality of Service class, which is the next lesson. The scheduler preempts by priority; the kubelet evicts under node pressure. Two components, two rulebooks, both deciding who lives when things get tight.

$ kubectl get events -n payments --sort-by=.lastTimestamp | grep -i preempt
90s   Normal   Preempted   pod/batch-report-77x   Preempted by payments/payments-api on node node-b2
$ kubectl get pod payments-api -n payments -o jsonpath='{.status.nominatedNodeName}'
node-b2

whenUnsatisfiable: DoNotSchedule is strict. ScheduleAnyway is guidance. Know which you shipped.

PriorityClasses without disruption budgets can surprise you during drains. Pair them.

Zone labels must exist for zone spreading to work. Empty topology keys fail closed into Pending. whenUnsatisfiable: DoNotSchedule is strict. ScheduleAnyway is guidance. Know which you shipped.

Try this

Apply a Deployment with topology spread constraints across zones or hosts, scale it, and inspect pod distribution. Then give one pod a high priority class in a crowded namespace if you can.

$ kubectl apply -f deployment.yaml
deployment.apps/web created
$ kubectl get pods -l app=web -o custom-columns=POD:.metadata.name,NODE:.spec.nodeName
POD                   NODE
web-6bf4c9d8f-2xq4h   node-a1
web-6bf4c9d8f-8fp2k   node-a2
web-6bf4c9d8f-lm7dr   node-b1
web-6bf4c9d8f-p4ktz   node-b2
web-6bf4c9d8f-r9wzx   node-c1
web-6bf4c9d8f-tv6hq   node-c2
 
$ kubectl apply -f priorityclass.yaml
priorityclass.scheduling.k8s.io/high-priority created
pod/payments-api created
$ kubectl get priorityclass
NAME                      VALUE        GLOBAL-DEFAULT   AGE
high-priority             1000000      false            6s
system-cluster-critical   2000000000   false            51d
system-node-critical      2000001000   false            51d
 
$ kubectl get events -n payments --sort-by=.lastTimestamp | grep -i preempt
90s   Normal   Preempted   pod/batch-report-77x   Preempted by payments/payments-api on node node-b2
$ kubectl get pod payments-api -n payments -o jsonpath='{.status.nominatedNodeName}'
node-b2

Takeaway

Topology spread evens pods across domains. Priority and preemption decide who wins when the cluster is full.

Related