How the scheduler places pods

Filter then score: how a pod is assigned a node.

A Pod you just created has an empty box on its record where the node name should go. Nothing is running it yet. It's just an entry in etcd, the cluster's key-value database, and nothing more. The scheduler exists to fill in that one blank. It reads the Pod, picks a machine, and writes the machine's name back through the API server (the cluster's front desk for every read and write). The kubelet, the agent running on that machine, only notices the Pod once the name is written. So the whole story of placement is one field getting filled. Learn how the scheduler chooses and you'll know why a Pod landed where it did, or why it's stuck at the door.

Think of a gate agent finding you a seat on a full flight. First they cross off every seat that won't work: no exit row for the passenger with a lap infant, no window if you asked for aisle, nothing already taken. Then, from whatever's left, they pick the best one against a few preferences. The scheduler runs those exact two phases, in that order, and it has names for them. Filter throws out every node that can't run the Pod. Score ranks the survivors. The highest-scoring node wins.

Filter, then score

The scheduler watches the API server for Pods with no nodeName, and works through them one at a time. The filter phase runs the Pod against every node and asks a yes/no question at each one. Does this node have enough allocatable CPU and memory for the Pod's requests? Does the Pod tolerate whatever taints the node carries (a taint is a 'keep off unless invited' mark on a node)? Do the Pod's nodeSelector and node affinity match the node's labels? Are the ports it wants free? Can the node attach its volumes? Each check is a separate plugin, and a node has to pass all of them to survive. What's left is the set of feasible nodes.

If exactly one node survives, it's chosen. If several do, the score phase gives each a number from 0 to 100 by running another set of plugins, adds them up with weights, and takes the top. By default the scheduler prefers the emptier node, so Pods spread out instead of piling onto one machine. It also leans toward nodes that already have the container image cached, and honors any 'preferred' affinity you asked for. Once a winner is picked the scheduler binds the Pod: it writes the node name back (technically a small Binding request to the API server), and its job is done. Now the kubelet on that node sees a Pod addressed to it and starts pulling the image.

kubectl run web --image=nginx:1.27
kubectl get pod web -o wide

pod/web created
 
NAME   READY   STATUS    RESTARTS   AGE   IP           NODE     NOMINATED NODE   READINESS GATES
web    1/1     Running   0          9s    10.244.2.7   node-2   <none>           <none>

kubectl get events --field-selector involvedObject.name=web,reason=Scheduled

LAST SEEN   TYPE     REASON      OBJECT      MESSAGE
9s          Normal   Scheduled   pod/web     Successfully assigned default/web to node-2

That single Scheduled event is the whole filter-and-score run, compressed to one line. The scheduler picked node-2 and bound the Pod there. Everything after it in the Pod's event list (Pulling, Started) belongs to the kubelet, not the scheduler.

The filter measures requests, not what's actually running

Here's the part that trips people up. When the filter checks whether a node has room, it does not look at how busy the machine really is. It looks at the sum of the CPU and memory requests of the Pods already placed there, and compares that to the node's allocatable amount. Requests are the reservation you wrote in the Pod spec. Allocatable is the node's total capacity minus what's held back for the kubelet and the operating system. So a node can sit nearly idle and still refuse a Pod, because its requests are already reserved to the ceiling. The reverse bites harder: Pods with no requests set look free to the scheduler, so it happily stacks a dozen of them on one node, which then falls over under real load.

kubectl describe node node-2 | grep -A6 "Allocated resources"

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests      Limits
  --------           --------      ------
  cpu                3800m (95%)   6 (150%)
  memory             6144Mi (37%)  8Gi (50%)

kubectl top node node-2

NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-2   840m         21%    5210Mi          32%

Read the two together. The scheduler sees CPU at 95% reserved and treats node-2 as almost full. The real box is running at 21%. Both numbers are correct. They just measure different things, and the scheduler only cares about the first one. That gap is where most surprising placement decisions come from.

Reading a Pending pod

A Pending Pod almost always means the filter phase rejected every node, and the scheduler writes down exactly why. It fires a FailedScheduling event on the Pod, and the message names the count and the reason. This is the single most useful scheduling skill you can build. Don't guess. Read the event.

apiVersion: v1
kind: Pod
metadata:
  name: big
spec:
  containers:
    - name: app
      image: nginx:1.27
      resources:
        requests:
          cpu: "8"

kubectl apply -f big.yaml
kubectl describe pod big | grep -A5 Events:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  30s   default-scheduler  0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.

Read the message left to right. Zero of three nodes are available, and the reason is Insufficient cpu on all three, because no single node has 8 whole cores free to reserve. The second half, the preemption line, is the scheduler telling you it also tried to make room by evicting lower-priority Pods and found none worth removing. The fix falls straight out of the reason. Lower the request, add a node big enough, or give the Pod a priority that lets it preempt. If the reason had said 'untolerated taint' you'd add a toleration. If it said 'didn't match node selector' you'd fix the label the Pod is asking for. The event points at the exact filter that rejected everything, so you never have to guess your way to the fix.

nodeName short-circuits the scheduler. Use it for emergencies and labs, not as a placement strategy.

Priorities and preemption change who wins under pressure. Understand QoS before you invent priority classes.

Scheduler profiles and plugins matter on advanced clusters. Defaults are enough until you run specialized hardware pools. nodeName short-circuits the scheduler.

Try this

Create a pod, watch it bind, then create one that cannot schedule (huge memory request). Read the Pending events for the filter that rejected every node.

$ kubectl run web --image=nginx:1.27
$ kubectl get pod web -o wide
 
$ kubectl get events --field-selector involvedObject.name=web,reason=Scheduled
 
$ kubectl describe node node-2 | grep -A6 "Allocated resources"
 
$ kubectl top node node-2
 
$ kubectl apply -f big.yaml
$ kubectl describe pod big | grep -A5 Events:

Takeaway

Filter then score. Pending with no node is a scheduling failure — describe the pod and read the predicates.

Related