Traps and cleanup

EXIT traps that always run.

A good line cook keeps one habit no matter how slammed the kitchen gets: you never walk away from a lit burner. Even when the fire alarm goes off in the middle of an order, you kill the flame on your way out the door. Shell scripts borrow things from the system the same way a cook borrows the stove. They carve out temporary folders, grab a lock file so two copies never run at once, start background helpers, flip a firewall rule. When the script stops early, and scripts stop early all the time, none of that gets put back on its own.

Someone presses Ctrl-C. A CI runner (continuous integration, the automated system that builds and tests your code) cancels the job. A server drains before a deploy and shoves your script aside. In every one of those cases the tidy 'undo' code you wrote at the bottom of the file never runs, because control never reached the bottom. A trap is how you fix that. It is a fire-drill procedure you register up front, and the shell runs it on the way out no matter which exit the script takes.

Two words first. A signal is a tiny message the kernel (the core part of the operating system that talks to the hardware) hand-delivers to a running program, the way a dispatcher radios a driver mid-route. Pressing Ctrl-C sends SIGINT (signal interrupt). Running kill <pid> sends SIGTERM (signal terminate, the polite 'please wrap up'); the pid is the process ID, the number the system uses to name one running program. Running kill -9 sends SIGKILL (die now, no discussion). The trap builtin is you telling bash: when signal X shows up, run this code instead of dropping dead. Bash adds two fake signals the kernel never actually sends. EXIT fires whenever the shell stops for any reason. ERR fires whenever a command fails. EXIT is the one you will reach for ninety percent of the time.

What leaks when a script dies

Skip the trap and every abnormal exit leaves a mess. A script that dies right after mktemp -d (make temporary directory) leaves an empty folder in /tmp, and another, and another, until the disk fills. Slow on your laptop. Fast on a busy CI runner that fires the job a thousand times a day. A cron job (the scheduler that runs commands at set times) that dies while holding a lock file blocks every future run silently, and you find out days later when nobody rotated the logs.

The nastiest cases come in pairs. You open a firewall hole with iptables -I (iptables is the long-standing Linux firewall command; -I inserts a rule) at the top and close it with iptables -D (delete that rule) at the bottom. You cordon a Kubernetes node (mark it so no new work lands there), do maintenance, then uncordon it. Kill the script between the two halves and the second half never happens. The firewall stays open. The node stays cordoned and quietly starves. For a defender this is the whole payoff: a trap turns 'the undo lives at the bottom of the script' into 'the undo runs on any exit,' so a cancelled job cannot leave the system more exposed than it found it.

#!/usr/bin/env bash
set -euo pipefail
 
workdir=$(mktemp -d)
cleanup() { rm -rf -- "$workdir"; }
trap cleanup EXIT
 
echo "working in $workdir"
sleep 30            # press Ctrl-C while this waits

$ ./prep.sh
working in /tmp/tmp.Xq3kZ9vLbM
^C
$ ls -d /tmp/tmp.Xq3kZ9vLbM
ls: cannot access '/tmp/tmp.Xq3kZ9vLbM': No such file or directory

Three things make this hold together. cleanup is a function, so the teardown lives in one readable place and you register it once. The line trap cleanup EXIT pins it to the EXIT pseudo-signal, so it fires on a clean finish, on exit 1, and on a failure under set -e (the setting from the earlier lessons that aborts the script the moment any command fails). It also fires when an untrapped SIGINT or SIGTERM cuts the script down, though that last part is bash being generous; plain POSIX sh (the portable shell standard that smaller shells like dash follow) makes no such promise. And you register the trap on the very next line after creating the thing it cleans up. Register it before mktemp -d and there is nothing to delete yet. Register it three commands later and a failure in that gap still leaks.

The order inside bash is worth knowing. When a real signal lands, bash lets the command that is currently running finish, then runs your signal trap, and runs the EXIT trap dead last, in the same shell process, with all your variables still set. Inside the EXIT trap the special variable $? holds the exit status the script was dying with. Capture it on the first line (status=$?) if your cleanup runs any commands of its own, otherwise a stray failure in teardown overwrites the real cause of death before anyone reads it.

Die honestly so your supervisor believes you

A program that is killed by signal number N exits with status 128 plus N. That is a Unix convention every supervisor trusts. SIGINT is signal 2, so Ctrl-C gives you 130. SIGTERM is signal 15, so a polite kill gives you 143. systemd (the program that starts and supervises services on modern Linux), Kubernetes, and CI runners all read that number to tell 'the user cancelled me' apart from 'I crashed' apart from 'I finished fine.'

Here is the trap that lies. You catch SIGTERM, clean up, and end the handler with exit 0. You have just told your supervisor the job succeeded, when the truth is it was cancelled mid-run. The honest version cleans up, resets the signal trap back to bash's default, and re-sends the same signal to itself, so the process dies exactly the way it was asked to.

#!/usr/bin/env bash
set -euo pipefail
 
workdir=$(mktemp -d)
cleanup() { rm -rf -- "$workdir"; }
trap cleanup EXIT
trap 'trap - TERM; kill -s TERM "$$"' TERM
trap 'trap - INT;  kill -s INT  "$$"' INT
 
sleep 30            # stand-in for the real work

$ ./job.sh &
[1] 48213
$ kill -TERM %1; wait %1; echo "status: $?"
[1]+  Terminated              ./job.sh
status: 143

The $$ holds the script's own process ID, the number from a moment ago. The trap - TERM part puts SIGTERM back to its default behavior before the re-send, so the handler does not catch its own signal and loop forever. The result, status 143, is the honest answer. Your EXIT trap still deleted the temp folder, and the supervisor still sees a genuine SIGTERM death, so it records the job as cancelled instead of passed.

The ERR trap is a flight recorder

When a script dies three functions deep, bash's default death is silent about where it happened. The ERR pseudo-signal fixes that. It fires the instant a command fails, and you can have it print one line naming what died, on which line, with what status.

#!/usr/bin/env bash
set -Eeuo pipefail
trap 'echo "ERR line $LINENO: $BASH_COMMAND (exit $?)" >&2' ERR
cp /etc/missing.conf /tmp/

$ ./debug.sh
cp: cannot stat '/etc/missing.conf': No such file or directory
ERR line 4: cp /etc/missing.conf /tmp/ (exit 1)

One flag decides whether this trap can be trusted. The -E in set -Eeuo pipefail is errtrace, and it is what makes the ERR trap fire inside functions and inside command substitutions like $( ... ). Leave -E off and the trap silently covers only top-level commands, so a failure inside a helper function slips past your recorder without a word. $LINENO is the current line number, and $BASH_COMMAND is the command bash is running or about to run.

Where traps stop working

Traps have edges, and you need to know them before you trust the safety of a system to one. Except for that ERR inheritance, traps do not cross into subshells (a subshell is a child copy of the shell that bash spawns for anything wrapped in parentheses). The EXIT and signal traps you set are wiped back to default inside ( ... ) or $( ... ), so a subshell that creates its own temp file needs its own cleanup trap. SIGKILL and a pulled power cord are untrappable by design. When the kernel runs kill -9 on a process, zero cleanup happens, which is exactly why correctness can never depend on a trap. The stale-lock detection in the next lesson exists for precisely the runs where cleanup never fired.

And your cleanup runs under the same set -e as the rest of the script. If its first command fails, every teardown step after it is skipped. Write teardown to survive itself: rm -f instead of rm, a trailing || true on best-effort steps, and the actions that hurt most to skip placed first.

Hardening for production

At scale, traps run against a stopwatch. Kubernetes sends SIGTERM, waits terminationGracePeriodSeconds (30 by default), then sends SIGKILL and stops caring. GitLab and GitHub CI runners do much the same when a job is cancelled. Cleanup that uploads logs or drains connections has to finish inside that window, so order your teardown by what hurts most to lose.

spec:
  terminationGracePeriodSeconds: 30   # SIGTERM now, SIGKILL after 30s
  containers:
    - name: batch
      command: ["/bin/bash", "/app/job.sh"]

On the security side, the trap is where secrets go to die. If your script writes a token to a temp file (a common move so the secret stays out of ps output; ps prints every running process with its full command line, which any other user on the box can read), the EXIT trap is the one guarantee that token does not outlive the job on a shared runner. And always write the deletion as rm -rf -- "$workdir". The double quotes stop word-splitting, and the -- tells rm that whatever follows is a path, not a flag. An unquoted rm -rf $workdir/ with workdir somehow empty expands to rm -rf /, the classic self-inflicted disaster. Modern GNU rm refuses that exact target (it treats / as protected), but not the countless sub-paths a half-set variable can still point at, so do not lean on that safety net. The set -u and quoting habits from the last two lessons shut the door properly.

Traps decide when cleanup runs. They say nothing about whether the resource was safe to create in the first place. mktemp -d showed up here on trust. A predictable temp file name is an attack vector, /tmp is a shared room where other users are watching, and a lock file is how you catch the crashed run that no trap could clean up. That is the next lesson: safe temp files and locks.

Related