A Job sets restartPolicy: OnFailure and backoffLimit: 6. Its container exits non-zero every single run. What actually happens?

The kubelet restarts the container inside the same Pod, and once the backoff limit is hit the Job is marked Failed. OnFailure restarts in place, so you watch one Pod restart rather than many Pods appear. backoffLimit still caps the attempts, then the Job's condition flips to Failed with a BackoffLimitExceeded event.

By default a Job is non-indexed and its Pods are interchangeable. On a Job with completions: 5, what does setting completionMode: Indexed change?

Each Pod is assigned a fixed number from 0 to 4, exposed in the JOB_COMPLETION_INDEX environment variable, so each owns one specific slice of the work. Indexed hands every Pod a stable index via JOB_COMPLETION_INDEX, like dealing a deck into numbered piles, so each Pod processes its own shard.

kubectl describe cronjob nightly-backup shows a Warning: FailedNeedsStart 'too many missed start times (> 100)'. ACTIVE is 0 and LAST SCHEDULE is stale. What actually gets it scheduling again?

Set startingDeadlineSeconds on the CronJob to bound how far back it counts. The look-back can never begin earlier than now minus startingDeadlineSeconds, so setting it on the stuck object drops the tally below 100 and the next pass schedules again. Deleting and re-creating the CronJob clears it too, but you don't have to.

Courses Kubernetes administrationJobs & CronJobs

Jobs & CronJobs

Run-to-completion and scheduled batch work.

Intermediate10 min · lesson 13 of 65

In plain terms

A Job is a task that finishes — wash the car — not a service that runs forever, like keeping the lights on. A CronJob is simply that same task on a repeating alarm clock.

A Deployment never finishes, and that's the whole point of one. So running a database migration as a Deployment is a quiet disaster. The migration does its work, the Pod exits cleanly with a success code, and Kubernetes cheerfully starts it right back up, then again, forever. A Pod, by the way, is the smallest thing Kubernetes runs: one or more containers (packaged, running programs) sharing a single network address. A Deployment keeps a set of those Pods alive around the clock and restarts any that die. Batch work needs the opposite promise. It needs something that knows when to stop.

Two objects make that promise. A Job runs a task until it succeeds, then stands down. A CronJob is the same idea wired to an alarm clock: a task that fires on a schedule. Both live in the batch/v1 API group, which is just the family of Kubernetes objects meant for batch work. And both are driven by controllers baked into the kube-controller-manager. A controller works like a thermostat. It holds a target, watches the current reading, and keeps nudging reality back toward what you asked for.

A Job runs until the work is done

Think of a Job as a work order with a checkbox: do this thing, and don't mark it done until the box is ticked. Three fields shape how it behaves. completions is how many successful runs you need. parallelism is how many Pods may run at once. backoffLimit is how many times a failing Pod may be retried before the Job gives up on the whole thing. The Job controller creates the Pods, watches them, and keeps score. Once enough of them reach Completed, the Job is finished, and it deliberately leaves those completed Pods in place so you can still pull their logs.

data-import.yaml

apiVersion: batch/v1
kind: Job
metadata:
  name: data-import
spec:
  completions: 5
  parallelism: 2
  backoffLimit: 4
  ttlSecondsAfterFinished: 3600
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: import
          image: registry.example.com/importer:1.2
          command: ["/bin/import", "--batch"]

apply-and-watch.sh

kubectl apply -f data-import.yaml
kubectl get job data-import -w

output.txt

job.batch/data-import created
NAME          STATUS     COMPLETIONS   DURATION   AGE
data-import   Running    0/5           2s         2s
data-import   Running    2/5           14s        14s
data-import   Running    4/5           27s        27s
data-import   Complete   5/5           34s        34s

The COMPLETIONS column ticks up as Pods succeed. That ttlSecondsAfterFinished line matters more than it looks. Leave it out and finished Jobs, along with every Pod they created, hang around forever and slowly clog etcd, the key-value database that holds the whole cluster's state. Put it in, and the controller garbage-collects the lot an hour after the Job finishes, the way you'd clear resolved tickets off a help desk board. That's how a busy cluster keeps from drowning in the wreckage of yesterday's Jobs.

By default a Job is non-indexed: any Pod that succeeds counts toward completions, and the five Pods here are interchangeable. Sometimes you want the opposite, where each Pod owns one specific slice of the work. It's like dealing a deck into numbered piles, one per worker, so nobody touches anyone else's cards. Set completionMode: Indexed and each Pod gets a fixed number from 0 to completions minus 1, handed to it in the JOB_COMPLETION_INDEX environment variable. Pod 0 takes the first shard, Pod 1 the next, and on down the line.

There's also podFailurePolicy, on by default since v1.26 and stable since v1.31, so most clusters you'll meet already have it. It looks at how a Pod failed: the container's exit code, or the conditions Kubernetes attached to the Pod, such as DisruptionTarget on a Pod the cluster itself evicted. You can declare that one specific exit code means 'give up on the entire Job right now' (the FailJob action), or that a known-flaky failure shouldn't count against the retry budget at all (the Ignore action). There is one string attached: the API server accepts podFailurePolicy only when the Pod template sets restartPolicy: Never. Without it, every failure looks the same and you just burn through the backoff limit one wasted attempt at a time.

restartPolicy changes what 'retry' even means

A Job template can't use restartPolicy: Always. The API server, the component that validates every change you send the cluster, flatly rejects it, because a task that always restarts can never be finished. That leaves two real choices, and they don't behave the same. With restartPolicy: Never, a failed Pod is left where it died and the controller spins up a brand-new Pod for the next attempt, so you collect one Pod per failure and a clear paper trail. With restartPolicy: OnFailure, the kubelet (the small agent on each node that actually starts and stops containers) restarts the container inside the same Pod, so there are fewer objects to sift through, but no per-attempt history and no podFailurePolicy, which pairs only with Never. Either way, backoffLimit caps the total attempts, and the waits between them grow exponentially, like hitting snooze with a longer gap each time: 10 seconds, then 20, then 40, up to a ceiling of six minutes.

inspect-job.sh

kubectl describe job data-import

events.txt

Events:
  Type    Reason            Age   From            Message
  ----    ------            ----  ----            -------
  Normal  SuccessfulCreate  34s   job-controller  Created pod: data-import-7bxk9
  Normal  SuccessfulCreate  34s   job-controller  Created pod: data-import-q4m2l
  Normal  Completed         0s    job-controller  Job completed

CronJobs: Jobs on a schedule

A CronJob doesn't run your container itself. It's more like a factory that stamps out a fresh Job object every time the clock matches its schedule. The schedule uses classic cron syntax: five fields for minute, hour, day of month, month, and day of week. Since Kubernetes 1.27 you can also set spec.timeZone, so "0 2 * * *" paired with Europe/London means 2am in London, not 2am wherever your control plane happens to be sitting. Get that wrong and your nightly backup slides by an hour twice a year, every time the clocks change.

nightly-backup.yaml

apiVersion: batch/v1
kind: CronJob
metadata:
  name: nightly-backup
spec:
  schedule: "0 2 * * *"
  timeZone: "Europe/London"
  concurrencyPolicy: Forbid
  startingDeadlineSeconds: 300
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      backoffLimit: 2
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: backup
              image: registry.example.com/backup:1.0

get-cronjob.sh

kubectl get cronjob nightly-backup

output.txt

NAME             SCHEDULE    TIMEZONE        SUSPEND   ACTIVE   LAST SCHEDULE   AGE
nightly-backup   0 2 * * *   Europe/London   False     0        8h              3d

concurrencyPolicy decides what happens when one run is still going and the next is already due. Allow lets them overlap. Forbid skips the newcomer and waits for the following slot. Replace kills the run in progress and starts fresh. Choose it on purpose, because two backup Jobs fighting over the same volume can corrupt each other. The two history limits stop finished Job objects from piling up: three successes and one failure are kept here, and older ones get swept away. To pull a schedule offline for maintenance without deleting anything, patch spec.suspend to true, and the ACTIVE and LAST SCHEDULE columns freeze until you flip it back.

A CronJob can quietly stop scheduling itself

Here's a failure mode that hides in plain sight. On every pass, the CronJob controller looks back from the last time it fired a Job to right now, and counts the start times it should have created but didn't. If that count ever climbs past 100, it stops trying. It logs a FailedNeedsStart warning, refuses to create the Job, and hits the same wall on every pass after that. Meanwhile ACTIVE sits at 0 and LAST SCHEDULE goes stale while everyone assumes the backups are still running. What stacks up 100 misses? Almost always a long stretch where the controller couldn't act: the control plane was down, the nodes were starved for resources, or the CronJob sat suspended over a long weekend while a once-a-minute schedule quietly piled up thousands of missed slots. Poking the object at random won't help, because only two things move that window. One is the bookkeeping the count starts from, status.lastScheduleTime, or the object's creation time if it has never fired. The other is startingDeadlineSeconds, which stops the look-back from ever beginning earlier than that many seconds ago. So the repair is the one the warning itself names: set or lower startingDeadlineSeconds on the object that is already stuck, and on the next pass the tally drops to a handful of slots and scheduling resumes. Deleting and re-applying the CronJob clears it too, since a fresh object has no last-scheduled time, but you don't have to go that far. Size the deadline to your schedule: long enough to still catch a run that's merely late, short enough that a frequent schedule can't fit more than 100 slots inside the window.

How a scheduled run becomes finished work

When a scheduled Job doesn't run, don't guess. Read the events the controllers actually wrote down, because they spell out exactly what went wrong.

troubleshoot.sh

kubectl describe cronjob nightly-backup

events.txt

Events:
  Type     Reason            Age   From                 Message
  ----     ------            ----  ----                 -------
  Warning  FailedNeedsStart  2m    cronjob-controller   Cannot determine if job needs to be started: too many missed start times (> 100). Set or decrease .spec.startingDeadlineSeconds or check clock skew.

Three details that bite later

First, parallelism and completions interact rather than stack. parallelism caps how many Pods run at once, completions decides how many successes end the Job, and the controller never starts more Pods than there are successes still owed, so setting parallelism above completions buys you nothing. Second, concurrencyPolicy is the only guardrail against a slow run colliding with the next scheduled one, and it defaults to Allow, which means overlap unless you say otherwise. Third, a failed Job leaves its Pods behind on purpose so you can still read the logs, which makes clearing them your problem: delete them deliberately with kubectl delete job, or let ttlSecondsAfterFinished do it on a timer, rather than losing the evidence the next time someone wipes the namespace.

Try this

Run this end to end on a scratch cluster. Both manifests above point at registry.example.com, which is not a real registry, so first change each image to busybox:1.36 and give each container command: ["sh", "-c", "sleep 5"]. Then watch the Job climb to 5 of 5, apply the CronJob and retime it to fire every minute so you are not waiting until 2am, and check that successfulJobsHistoryLimit really does cap how many finished Jobs stay behind.

terminal

$ kubectl apply -f data-import.yaml
$ kubectl get job data-import -w
$ kubectl describe job data-import
 
$ kubectl apply -f nightly-backup.yaml
$ kubectl patch cronjob nightly-backup -p '{"spec":{"schedule":"*/1 * * * *"}}'
$ kubectl get cronjob nightly-backup
 
# five minutes on: successfulJobsHistoryLimit: 3 should cap this at three backup Jobs
$ kubectl get jobs
$ kubectl describe cronjob nightly-backup
 
$ kubectl delete cronjob nightly-backup
$ kubectl delete job data-import

Takeaway

Jobs run to completion; CronJobs schedule Jobs. backoffLimit and restartPolicy decide whether failure retries or just fails loudly.

Deployment strategiesCommands & arguments

What Kubernetes isKubernetes fundamentals Security domains & threat modelKubernetes security & hardening The cluster attack surfaceKubernetes attack & defense Kubernetes interview questionsInterview prep