>_SecOpsLog_
CoursesKubernetes administrationManual scheduling

Manual scheduling

nodeName, and what to do when no scheduler runs.

Intermediate8 min · lesson 22 of 65
In plain terms
Manual scheduling is walking past the host and sitting yourself at a specific table. Handy when the host is out sick — but you’ve skipped every check they would have done for you.

Every Pod (the smallest unit Kubernetes runs, one or more containers that share a single network address) carries a field called spec.nodeName. While that field is empty, the pod has nowhere to live and nothing starts it. The moment something writes a node's name into it, the pod belongs to that exact machine and gets launched there. The scheduler's entire job, all the filtering of unfit nodes and ranking of the rest, comes down to writing one string into that one field. Manual scheduling is you writing it yourself and skipping the scheduler completely.

Think of a wedding with a seating chart. The host reads the guest list, remembers that two cousins can't sit together, checks which tables still have empty chairs, and only then writes your name on a place card. That host is the scheduler. Manual scheduling is walking in and dropping into a chair you labeled with your own name. It's fast and fully under your control. It also means nobody checked whether that table is already full, or whether you were the one guest who was supposed to stay on the far side of the room.

Setting nodeName by hand

Here is the whole trick. You put nodeName straight into the manifest (the YAML text file that describes the pod you want) at creation time. There is no Scheduled event and no default-scheduler line in the logs, because the scheduler never sees the pod. The kubelet (the agent Kubernetes runs on every node) watches the API server (the cluster's front desk, the one component every other part talks through) for pods stamped with its own node name, spots this one, and starts the containers directly.

pinned.yaml
apiVersion: v1
kind: Pod
metadata:
  name: pinned
spec:
  nodeName: worker-2
  containers:
    - name: app
      image: registry.k8s.io/pause:3.10
terminal
$ kubectl apply -f pinned.yaml
pod/pinned created
$ kubectl get pod pinned -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
pinned   1/1     Running   0          6s    10.244.2.9   worker-2   <none>           <none>
$ kubectl describe pod pinned | grep -A6 Events
Events:
  Type    Reason   Age   From      Message
  ----    ------   ---   ----      -------
  Normal  Pulled   6s    kubelet   Container image "registry.k8s.io/pause:3.10" already present on machine
  Normal  Created  6s    kubelet   Created container app
  Normal  Started  6s    kubelet   Started container app

Look at what's missing from those events. A normally scheduled pod's first event reads "Successfully assigned default/pinned to worker-2" and its source is default-scheduler. Here there is no such line, and every event's From column says kubelet. That absence is your proof: this pod was placed by hand, and the scheduler never touched it. It's the fastest way to tell manual placement apart from real scheduling after the fact.

What the kubelet still checks, and what it doesn't

A common belief is that nodeName lets you cram a pod onto a node that's already full. Not quite. When a pod shows up already bound to it, the kubelet runs a short admission check of its own before starting any container, and resource fit is part of that check. Ask for more CPU or memory than the node can actually hand out, and the kubelet refuses. The container never starts. The pod drops into a Failed phase with the reason OutOfcpu or OutOfmemory.

toobig.yaml
apiVersion: v1
kind: Pod
metadata:
  name: toobig
spec:
  nodeName: worker-2
  containers:
    - name: app
      image: registry.k8s.io/pause:3.10
      resources:
        requests:
          cpu: "32"        # worker-2 only has 4 CPUs
terminal
$ kubectl apply -f toobig.yaml
pod/toobig created
$ kubectl get pod toobig
NAME     READY   STATUS     RESTARTS   AGE
toobig   0/1     OutOfcpu   0          4s
$ kubectl get pod toobig -o jsonpath='{.status.reason}{"\n"}{.status.message}{"\n"}'
OutOfcpu
Node didn't have enough resource: cpu, requested: 32000, used: 1200, capacity: 4000

So the kubelet does defend raw capacity. What it does not look at is everything the scheduler weighs on the whole cluster's behalf: taints, node affinity, nodeSelector, pod anti-affinity, topology spread. A node that was deliberately tainted to keep workloads off will run your nodeName pod without complaint, because the toleration check lives in the scheduler and you just walked around it. That's the real hazard of manual scheduling. You can drop a pod exactly where the cluster spent real effort telling it not to go.

Why it matters: static pods and bootstrap

There is a chicken-and-egg problem buried in every self-managed cluster. The control plane runs as pods: the API server, etcd (the key-value database that stores all cluster state), the scheduler, the controller manager. But creating a pod normally requires the API server to already be running and to accept the request. So how does the first API server ever start, when there is no API server yet to accept it?

Static pods are the answer, and they're really just manual scheduling coming through a different door. The kubelet watches a folder on the node's local disk, usually /etc/kubernetes/manifests. Any pod manifest you drop into that folder, the kubelet runs directly, with no scheduler and no API server in the loop. It's the same idea as a nodeName pod, except the instruction arrives as a file on disk instead of an API object. That's exactly how kubeadm brings a cluster to life: it writes the control-plane manifests into that folder, and the kubelet starts them cold.

terminal
$ ls /etc/kubernetes/manifests
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml
$ kubectl -n kube-system get pod kube-apiserver-cp-1 -o wide
NAME                  READY   STATUS    RESTARTS     AGE   IP          NODE   NOMINATED NODE   READINESS GATES
kube-apiserver-cp-1   1/1     Running   2 (5d ago)   20d   10.0.0.10   cp-1   <none>           <none>

Once the API server is up, the kubelet creates a read-only "mirror pod" inside it for each static pod, so they show up in kubectl like anything else. Two tells give them away. The node name is baked right into the pod name (kube-apiserver-cp-1), and you can't delete them with kubectl, the object just comes straight back. The only way to stop a static pod is to move its manifest out of that folder. Worth knowing before you try to kubectl delete a control-plane pod and wonder why it keeps respawning.

Three roads to spec.nodeName
spec.nodeName gets filled in
only then does a pod have a home
normal
the scheduler
filters and scores every node, then POSTs a Binding. Respects taints, affinity, and capacity.
manual
you, in the manifest
nodeName set by hand. Kubelet checks resource fit only, skips taints and affinity.
bootstrap
the kubelet, from disk
static pod in /etc/kubernetes/manifests. No API server needed. How the control plane starts.
Three roads, one field. Whoever writes spec.nodeName, the kubelet on that node is what actually pulls the images and starts the containers.

When the scheduler is down

This is a classic 3am page and a classic exam task. The scheduler crashes, and every new pod you create just sits at Pending, forever, because nothing is left to assign it a node. First you confirm the scheduler really is the problem. Then you get your one critical pod running by hand while you go fix the actual cause.

terminal
$ kubectl -n kube-system get pods -l component=kube-scheduler
NAME                  READY   STATUS             RESTARTS      AGE
kube-scheduler-cp-1   0/1     CrashLoopBackOff   7 (30s ago)   5m
$ kubectl get pod urgent -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
urgent   0/1     Pending   0          2m    <none>   <none>   <none>           <none>

spec.nodeName is immutable once a pod exists, so you can't patch it onto the pending one. The dependable fix is to recreate the pod with nodeName already set in the manifest. Under the hood you're doing the same thing a healthy scheduler does. It doesn't patch the field either. It POSTs a Binding object to the pod's binding subresource, and the API server fills in nodeName from that. Setting the field yourself just cuts out the middleman.

terminal
$ kubectl delete pod urgent
pod "urgent" deleted
$ kubectl apply -f urgent-pinned.yaml     # same pod, now with spec.nodeName: worker-1
pod/urgent created
$ kubectl get pod urgent -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
urgent   1/1     Running   0          9s    10.244.1.5   worker-1   <none>           <none>
A pinned pod has no safety net
A pod placed with nodeName is usually a bare Pod, with no ReplicaSet or Deployment behind it to make copies. If its node reboots or dies, nothing recreates it. It's simply gone. And kubectl drain won't rescue it. Drain refuses to evict a bare pod unless you pass --force, and then it just deletes it and leaves nothing running in its place. Reach for nodeName as a deliberate, short-lived pin or a bootstrap tool, never for something that has to stay up on its own.

If the named node is cordoned or dead, the pod still sits unbound in practice. Manual is not magic.

Static pods also skip the scheduler, but they are kubelet-local. Do not confuse the two mechanisms.

Document every manually pinned workload. The next upgrade drain will otherwise look haunted. Manual is not magic.

Try this

Pin a pod with nodeName to a known worker, then try the same pin to a missing node name and observe the stuck state.

terminal
$ kubectl apply -f pinned.yaml
pod/pinned created
$ kubectl get pod pinned -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
pinned   1/1     Running   0          6s    10.244.2.9   worker-2   <none>           <none>
$ kubectl describe pod pinned | grep -A6 Events
Events:
  Type    Reason   Age   From      Message
  ----    ------   ---   ----      -------
  Normal  Pulled   6s    kubelet   Container image "registry.k8s.io/pause:3.10" already present on machine
  Normal  Created  6s    kubelet   Created container app
  Normal  Started  6s    kubelet   Started container app
 
$ kubectl apply -f toobig.yaml
pod/toobig created
$ kubectl get pod toobig
NAME     READY   STATUS     RESTARTS   AGE
toobig   0/1     OutOfcpu   0          4s
$ kubectl get pod toobig -o jsonpath='{.status.reason}{"\n"}{.status.message}{"\n"}'
OutOfcpu
Node didn't have enough resource: cpu, requested: 32000, used: 1200, capacity: 4000
 
$ ls /etc/kubernetes/manifests
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml
$ kubectl -n kube-system get pod kube-apiserver-cp-1 -o wide
NAME                  READY   STATUS    RESTARTS     AGE   IP          NODE   NOMINATED NODE   READINESS GATES
kube-apiserver-cp-1   1/1     Running   2 (5d ago)   20d   10.0.0.10   cp-1   <none>           <none>
 
$ kubectl -n kube-system get pods -l component=kube-scheduler
NAME                  READY   STATUS             RESTARTS      AGE
kube-scheduler-cp-1   0/1     CrashLoopBackOff   7 (30s ago)   5m
$ kubectl get pod urgent -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP       NODE     NOMINATED NODE   READINESS GATES
urgent   0/1     Pending   0          2m    <none>   <none>   <none>           <none>
 
$ kubectl delete pod urgent
pod "urgent" deleted
$ kubectl apply -f urgent-pinned.yaml     # same pod, now with spec.nodeName: worker-1
pod/urgent created
$ kubectl get pod urgent -o wide
NAME     READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
urgent   1/1     Running   0          9s    10.244.1.5   worker-1   <none>           <none>

Takeaway

nodeName bypasses scheduling. Great for broken-scheduler emergencies; dangerous as everyday placement.

Quick check
01You set spec.nodeName to worker-2 on a pod that requests 32 CPUs, but worker-2 only has 4. What happens?
Incorrect — The scheduler never sees a nodeName pod, so it plays no part here at all.
Correct — The kubelet runs its own resource-fit check on pods bound to it and refuses ones that can't fit.
Incorrect — A frequent myth. nodeName skips taints and affinity, but the kubelet still guards raw capacity.
Incorrect — The API server happily stores it. The rejection happens later, at the kubelet on worker-2.
02A node was tainted node-role.kubernetes.io/control-plane:NoSchedule to keep ordinary workloads off it. You create a Pod with spec.nodeName set to that node and no matching toleration, requesting only modest resources. What happens?
Incorrect — taint enforcement lives in the scheduler, and the kubelet's own admission check never looks at taints.
Incorrect — the API server stores the Pod happily; there's no toleration validation at creation time.
Incorrect — nothing evaluates the taint for a nodeName Pod, so it is never held in Pending over it.
Correct — setting nodeName skips the scheduler, so the taint that was meant to repel workloads is simply never checked.
03On a kubeadm control-plane node you run kubectl delete pod kube-apiserver-cp-1. It vanishes from kubectl for a moment, then reappears with the same name. Why, and how do you actually stop it?
Incorrect — control-plane components run as static pods, not from a Deployment, so there's no ReplicaSet to scale.
Incorrect — static pods never go through the scheduler, so cordoning the node does nothing to them.
Correct — the kubelet runs the pod from an on-disk manifest and only mirrors it into the API server, so the file is the real control.
Incorrect — it reappears because the kubelet recreates the mirror from the on-disk manifest, not from etcd.
Labels, selectors & annotationsTaints & tolerations

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep