Multiple schedulers & profiles

Running and targeting custom scheduling logic.

Every node is up, none is full, and your pod sits in Pending with no events at all. Not FailedScheduling. Nothing. Nine times out of ten the pod asked to be placed by a scheduler that isn't running. That is the door into this topic: a pod does not have to be placed by the built-in scheduler. Think of the scheduler as the host at a busy restaurant, the person who decides which table each party gets. The default host is a solid generalist. But you can hire a second one, or hand the same host a different seating card, and send certain guests to whichever you choose.

Point a pod at a different scheduler

Every pod (the smallest thing Kubernetes runs, normally a single container) carries a spec.schedulerName field. Leave it blank and the API server fills in default-scheduler, the built-in one. A scheduler only ever touches pods that name it, and ignores every other pod completely. That one rule is what lets two schedulers share a cluster without a fight: each picks up only the pods addressed to it. So why run a second scheduler at all? Because some jobs need logic the default one cannot express. The classic case is gang scheduling for batch or HPC (high-performance computing) work: a 500-pod training job that has to start all of its pods at once or none of them, so it doesn't grab half the cluster and then sit deadlocked waiting for the other half. The default scheduler places one pod at a time and has no concept of all-or-nothing. You might wonder why node affinity or taints won't cover this. They steer the default scheduler, but they can't change its one-pod-at-a-time habit, so they still can't promise all-or-nothing. That needs different code, and Volcano is the batch scheduler most teams reach for here.

apiVersion: v1
kind: Pod
metadata:
  name: train-shard-0
  namespace: ml
spec:
  schedulerName: volcano      # place me with Volcano, not the default
  containers:
  - name: trainer
    image: registry.example.com/hpc-train:2.0

$ kubectl -n kube-system get pods -l app=volcano-scheduler
NAME                                READY   STATUS    RESTARTS   AGE
volcano-scheduler-6c9d7f8b5-lm4tq   1/1     Running   0          21m
$ kubectl -n ml describe pod train-shard-0 | grep -A3 Events:
Events:
  Type    Reason     Age   From     Message
  ----    ------     ----  ----     -------
  Normal  Scheduled  4s    volcano  Successfully assigned ml/train-shard-0 to node-7

The From column is your proof of who did the work: it reads volcano, not default-scheduler. That second scheduler is just a Deployment running a scheduler image, so it has no special powers by default. It needs a ServiceAccount (the identity a program logs in as) wired to permission to watch pods and nodes and to write the binding that assigns a pod to a node. Volcano ships that RBAC (Role-Based Access Control, the rules for who may do what) in its install. A hand-rolled scheduler that can't create bindings will sit there watching pods it is never allowed to place. One more detail trips people up. If you turn on leader election (you should, so a standby takes over cleanly), the new scheduler needs its own distinct lease name. Two schedulers sharing one lease name fight over the same lock. Only one wins, and the loser sits idle as a standby and never schedules a thing, so pods addressed to it pile up with no explanation. Give it a unique leaderElection resourceName and confirm it actually holds the lease before you route real work to it.

Profiles: one scheduler, many policies

Running a whole second scheduler is a lot of moving parts for what is often a small change in behavior. Most of the time you don't need custom code. You need the same scheduler to follow a different rule for some of your workloads. That is what profiles are for. Same host, two seating cards. One card fills a section completely before opening the next (packs guests in tight, cheaper to staff). The other spreads guests across the room (quieter, and one bad section takes down less). Guests ask for a card by name. In Kubernetes terms, one kube-scheduler binary reads a config file that defines several profiles. Each profile has its own schedulerName and its own plugin settings. The headline example is bin-packing versus spreading. The scoring plugin NodeResourcesFit rates a node by how full it is. Its default strategy, LeastAllocated, prefers emptier nodes, so pods spread out. Flip that profile to MostAllocated and it prefers the fullest node that still fits, so pods pack tight and you can scale empty nodes away to cut cost.

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: true
profiles:
- schedulerName: default-scheduler      # untouched: spreads pods (LeastAllocated)
- schedulerName: bin-packing            # new profile in the SAME binary
  pluginConfig:
  - name: NodeResourcesFit
    args:
      scoringStrategy:
        type: MostAllocated             # pack onto the fullest node that still fits

Profiles are cheaper than a second scheduler, but they aren't free of rules. Every profile lives in the same process and shares one waiting line for pods, so they must all agree on how that line is ordered. You can vary how nodes get scored per profile, packing on one and spreading on another. You cannot have one profile sort its waiting pods one way while another sorts them a different way, because there is only the single queue. You wire this in by pointing the scheduler at the file with --config=/etc/kubernetes/scheduler-config.yaml in its static pod manifest (the file on the control-plane node that defines the scheduler itself). The scheduler reads that file only at startup, so a change means restarting the scheduler pod, and a malformed config keeps it from coming back up at all. Validate the YAML and watch the scheduler roll before you assume the new profile exists. After that, a workload opts into a profile with the exact same schedulerName field you already used for a separate scheduler. Nothing new to learn on the pod side. Send your cost-sensitive batch pods to bin-packing, leave your latency-sensitive services on default-scheduler, and one scheduler process serves both. A profile's schedulerName must be unique in the config, and default-scheduler is the reserved name for the built-in behavior.

$ kubectl -n team-a get pod cost-web-7bd -o jsonpath='{.spec.schedulerName}'
bin-packing
$ kubectl -n team-a describe pod cost-web-7bd | grep -A3 Events:
Events:
  Type    Reason     Age   From         Message
  ----    ------     ----  ----         -------
  Normal  Scheduled  6s    bin-packing  Successfully assigned team-a/cost-web-7bd to node-2

Same tell as before: From reads bin-packing, so you know the packing profile placed this pod and not the default one. That From value is the fastest way to confirm a profile is live and actually being used, without reading a single line of scheduler logs.

$ kubectl -n ml get pod train-shard-9
NAME            READY   STATUS    RESTARTS   AGE
train-shard-9   0/1     Pending   0          11m
$ kubectl -n ml describe pod train-shard-9 | grep -A1 Events:
Events:  <none>

A typo in schedulerName means Pending forever with no default rescue. Validate the name against what actually runs.

Multiple schedulers must not fight over the same pods. Partition by schedulerName cleanly.

Profiles in one binary are often safer than a second process. Prefer the supported extension points your version documents.

Try this

You do not need Volcano installed for this. Point a pod at a scheduler name nobody answers to and watch it sit Pending with an empty event list, then check what is actually running under that name and whether the built-in scheduler was even handed a config file to read profiles from. Delete the pod, recreate it without the override, and read the From column. That is the whole diagnostic, start to finish.

$ kubectl run ghost --image=nginx:1.27 --overrides='{"apiVersion":"v1","spec":{"schedulerName":"ghost-scheduler"}}'
pod/ghost created
 
$ kubectl get pod ghost
NAME    READY   STATUS    RESTARTS   AGE
ghost   0/1     Pending   0          45s
 
$ kubectl get pod ghost -o jsonpath='{.spec.schedulerName}'
ghost-scheduler
 
$ kubectl describe pod ghost | grep -A3 Events:
Events:  <none>
 
$ kubectl -n kube-system get pods -l component=kube-scheduler
NAME                  READY   STATUS    RESTARTS   AGE
kube-scheduler-cp-1   1/1     Running   0          42d
 
$ kubectl -n kube-system get pod kube-scheduler-cp-1 -o jsonpath='{.spec.containers[0].command}'
["kube-scheduler","--authorization-kubeconfig=/etc/kubernetes/scheduler.conf","--bind-address=127.0.0.1","--kubeconfig=/etc/kubernetes/scheduler.conf","--leader-elect=true"]
 
$ kubectl delete pod ghost
pod "ghost" deleted
 
$ kubectl run ghost --image=nginx:1.27
pod/ghost created
 
$ kubectl describe pod ghost | grep -A3 Events:
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  2s    default-scheduler  Successfully assigned default/ghost to node-3
 
$ kubectl delete pod ghost
pod "ghost" deleted

Takeaway

Most clusters need one scheduler. Custom schedulers and profiles exist for specialized placement — target them explicitly per pod.

Related