Backup, recovery & agents at scale

JENKINS_HOME, restore drills, ephemeral agents.

A hardware shop keeps one safe behind the counter: the cash, the spare keys, the ledger, every customer record. Convenient, right up until the day the safe walks out the door. Jenkins is built the same way. Everything it knows lives in a single directory on the controller (the server that schedules builds and hands work out), called JENKINS_HOME, sitting at /var/jenkins_home by default. Inside it are your job definitions, every build's history, the encrypted credentials store, plugin state, and fingerprints (Jenkins's record of which build produced which file). Backup is easy to reason about, because there is only one directory to protect. Losing that directory means losing every pipeline and every secret in one go. For anything a team depends on, a real backup plus a restore you have actually rehearsed is the difference between a bad afternoon and a week spent rebuilding from memory.

What to keep in $JENKINS_HOME, and what to skip

Not every part of JENKINS_HOME is worth copying, and telling the two apart is what keeps a backup small and a restore fast. Keep the things you cannot re-create: jobs/ (each job's config.xml plus its build history), the top-level config.xml (global Jenkins settings), credentials.xml (the encrypted credential entries), and secrets/, which holds the master keys Jenkins uses to encrypt and decrypt everything inside credentials.xml. Skip the bulk that rebuilds itself: workspace/ (checked-out source and build leftovers, regenerated on the next run) and caches/ (downloaded tools and plugin caches). Dropping those two often takes a backup from tens of gigabytes down to a few hundred megabytes. That size difference is what makes the restore quick on the day you badly need it to be quick.

#!/usr/bin/env bash
set -euo pipefail
JHOME=/var/jenkins_home
STAMP=$(date +%F)
CLI=(java -jar jenkins-cli.jar -s http://localhost:8080/ -auth @/root/.jenkins-auth)
 
# 1. quiesce: -block waits until in-flight builds finish (plain quiet-down does NOT wait),
#    -timeout caps the wait at 300000 ms so the backup can't hang forever
"${CLI[@]}" quiet-down -block -timeout 300000
 
# 2. archive ONLY the state that matters; workspace/ and caches/ are not listed -> skipped
tar czvf "jenkins-backup-${STAMP}.tgz" -C "$JHOME" \
    jobs config.xml credentials.xml secrets
 
# 3. resume normal scheduling
"${CLI[@]}" cancel-quiet-down

jobs/
jobs/deploy-web/config.xml
jobs/deploy-web/builds/
jobs/release-api/config.xml
config.xml
credentials.xml
secrets/master.key
secrets/hudson.util.Secret
$ ls -lh jenkins-backup-2026-07-14.tgz
-rw-r--r-- 1 jenkins jenkins 214M Jul 14 09:02 jenkins-backup-2026-07-14.tgz
$ echo $?
0

Two details make that tar command worth trusting. The first is quiet-down -block, issued through the Jenkins CLI (command line interface, the small tool that talks to Jenkins from a terminal). Plain quiet-down stops new builds from being scheduled and returns straight away, which leaves any build already running free to keep writing files while you archive them. Adding -block makes the command wait until those in-flight builds have finished. Nothing is writing into JENKINS_HOME while tar reads it, so you get a clean snapshot instead of a smear of half-written files. The -timeout guard exists so a stuck build cannot stretch that wait out forever. The second detail is -C plus an explicit list of paths: you archive jobs/, config.xml, credentials.xml and secrets/, and nothing else, so the multi-gigabyte workspace and cache trees never land in the file. Copy the finished .tgz (a tarball, meaning one compressed archive file) off the Jenkins host the moment it exists. A backup sitting on the same disk it protects dies with that disk.

A backup you have never restored is only a hope

The only backup that counts is one you have already brought back to life. Every so often, stand up a throwaway Jenkins from the tarball and check three things: the jobs are there, the build history is there, and the credentials actually work. That last one is the real test, and you prove it with a canary pipeline (a tiny job that exists for no reason except to use a credential and report whether it worked). This is a restore drill, and it catches the failures that quietly ruin real backups: an archive that got truncated, a missing secrets/ directory, a plugin mismatch that refuses to load your job configs. Two pieces make recovery fast when you pair them. Configuration as Code (JCasC, a plugin that reads a jenkins.yaml file describing how the instance should be set up, kept in Git) gives you the shape of Jenkins. The JENKINS_HOME backup gives you the state and the secrets that JCasC deliberately does not hold.

#!/usr/bin/env bash
set -euo pipefail
 
# 1. unpack the backup into a fresh, throwaway home
mkdir -p /tmp/restore-home
tar xzf jenkins-backup-2026-07-14.tgz -C /tmp/restore-home
 
# 2. boot a disposable controller against that home (different port)
docker run -d --name jenkins-restore -p 8081:8080 \
    -v /tmp/restore-home:/var/jenkins_home \
    jenkins/jenkins:lts-jdk17
 
# 3. verify: jobs came back AND a restored credential actually decrypts
CLI=(java -jar jenkins-cli.jar -s http://localhost:8081/ -auth @/root/.jenkins-auth)
"${CLI[@]}" list-jobs
"${CLI[@]}" build cred-smoke-test -s -v   # canary pipeline that uses withCredentials

deploy-web
release-api
cred-smoke-test
Started cred-smoke-test #7
[Pipeline] withCredentials
Masking supported pattern matches of $REG_PASS
[Pipeline] sh
+ docker login -u ci-bot --password-stdin registry.acme.internal
Login Succeeded
[Pipeline] End of Pipeline
Finished: SUCCESS

When a restore goes sideways, two causes account for nearly all of it. Plugin drift comes first. Job config.xml files name plugins and their versions, so restoring them onto a controller with older or missing plugins throws errors like 'no such DSL method' and leaves the jobs unconfigurable. Pin your plugin set (a plugins.txt file, or JCasC) and restore it alongside the home directory. The second cause is an inconsistent snapshot. Tarring a live JENKINS_HOME while builds write into it can catch a config file halfway through a save, which is exactly why you run quiet-down -block first. Better still, snapshot the underlying storage volume and back up the snapshot, so the running instance is never touched at all.

If you would rather not babysit a script, the ThinBackup plugin schedules JENKINS_HOME backups from the Jenkins web interface, both full ones and differential ones (only what changed since the last full backup), and lets you exclude workspace and build data with a checkbox. Convenient, yes, but by default it writes to the controller's own disk, so shipping those archives off-host is still your job. Whichever way you take the backup, two rules hold. Put a copy somewhere the Jenkins host cannot reach. And treat that file as being exactly as sensitive as the secrets sitting inside it.

Agents at scale: static or ephemeral

An agent is the machine that actually runs your build steps. The controller only schedules the work and keeps score. A static agent is a long-lived virtual machine or a physical box that stays connected to the controller and runs build after build on the same disk. That is easy to set up, and it rots. State piles up between builds: leftover files, cached tokens, a poisoned dependency that one job pulled down last Tuesday. Builds start interfering with each other, and anyone who gets a foothold on that host keeps it. The box also bills you for sitting idle between busy periods.

Rental cars work differently from a company car. You take a clean one, use it, hand it back, and the next trip starts from a fresh vehicle. Ephemeral agents follow that pattern. Instead of keeping agents around, Jenkins creates a new one for each build and destroys it afterwards. With the Kubernetes plugin, every build runs in a brand new pod (the smallest thing Kubernetes can deploy, one or more containers sharing a network address), described by a pod template, which is the spec listing the container images that build pod is made of. The pod appears when the build starts, the build runs inside it, and the pod is deleted when the build ends. Every build gets a workspace nobody has touched, there is no long-lived host for an attacker to camp on, and the fleet shrinks to zero when the queue is empty.

pipeline {
  agent {
    kubernetes {
      // one throwaway pod per build; the plugin deletes it when the build ends
      yaml '''
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: node
    image: node:20-alpine
    command:
    - sleep
    args:
    - infinity
'''
    }
  }
  stages {
    stage('Build') {
      steps {
        container('node') {          // run these steps inside the node container
          sh 'npm ci && npm run build'
        }
      }
    }
  }
}

[Pipeline] podTemplate
[Pipeline] node
Agent build-web-42-p9k2f is provisioned from template build-web
Created Pod: kubernetes jenkins/build-web-42-p9k2f
Running on build-web-42-p9k2f in /home/jenkins/agent/workspace/build-web
[Pipeline] container
[Pipeline] sh
+ npm ci && npm run build
added 214 packages in 6s
[Pipeline] // container
[Pipeline] // node
Terminated Kubernetes instance for agent build-web-42-p9k2f
Disconnected computer build-web-42-p9k2f
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

The price you pay is cold start. A fresh pod has to pull its image and connect before the first build step runs, adding anywhere from a few seconds to a minute per build. Pre-pulled images and a small pool of warm agents for your busiest queue take most of that sting away. Security is where ephemeral agents win outright. They enforce the trust boundary this whole course keeps returning to: the controller holds the secrets, and the agent is disposable and treated as untrusted. A build cannot leave anything behind for the next one, and a compromised build dies with its pod.

Running Jenkins so a lost controller stays boring

Pick a date this week and run the drill end to end: tar the home, boot a throwaway controller on port 8081, run the canary pipeline, and watch a credential decrypt. That one hour tells you more than any checklist. The rest of the operating stance follows from it. Keep the instance's shape in Git with JCasC so every change gets reviewed. Back up JENKINS_HOME for the state and secrets JCasC deliberately leaves out. Run builds on ephemeral agents so nothing leaks from one build into the next and the fleet costs nothing while idle. Scope credentials tightly, keep the plugin list short and patched, and hold the controller/agent line: the controller is the thing you guard, and every build, being untrusted code, runs on a machine you throw away.

Try this

Work through “Running Jenkins so a lost controller stays boring” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: credentials.xml is dead weight without secrets/. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related