Traps, signals and cleanup

EXIT traps, INT and TERM handlers that exit honestly, and what no trap can catch.

Scripts borrow things while they run: a scratch directory, a lock, a maintenance flag that tells a load balancer to stop sending traffic, a firewall rule opened for the duration of a job. When a script stops early, the undo code at the bottom of the file never runs, and the thing it borrowed stays behind. In this lesson you will use an EXIT trap to clean up on every way a script can end, see what happens when a signal handler forgets to stop the script, write INT and TERM handlers that stop it with the honest statuses 130 and 143, learn when Bash runs a trap while a command is still running, and see what kill -9 leaves behind. The examples run in ~/bash-ops/traps.

Clean up on every exit: the EXIT trap

A signal is a short notice the kernel delivers to a process. Three matter here. SIGTERM (number 15) is the polite "please stop" that kill, systemd and timeout send by default. SIGINT (2) is what Ctrl-C sends. SIGKILL (9, kill -9) ends a process at once and cannot be caught. As "Exit status and tests" showed, a process killed by signal N reports status 128 + N: 143, 130 and 137.

trap 'commands' EXIT tells Bash to run the commands when the script exits, whatever the reason. EXIT is not a real signal but a name Bash uses for "the shell is exiting". Here the commands are a function:

#!/usr/bin/env bash
# prep.sh FILE: copy FILE into a private work directory and process it in three steps.
set -Eeuo pipefail
src=${1:?usage: prep.sh FILE}
workdir=$(mktemp -d)
cleanup() { rm -rf -- "$workdir"; }
trap cleanup EXIT
echo "work directory: $workdir"
cp -- "$src" "$workdir/"
for step in 1 2 3; do
  sleep 1
  echo "step $step of 3 done"
done

mktemp -d creates a new directory with a random name that only you can use, under the directory named by $TMPDIR (or /tmp if it is unset); the next lesson covers it. The trap is set on the very next line: set it earlier and there is nothing to delete yet, set it later and a failure in between leaves the directory behind. The runs below set TMPDIR=$PWD/scratch in front of the command, the one-command form from "Variables, quotes and command output", so ls -A scratch shows whether anything was left and your shell keeps its own TMPDIR. The first step also creates the app.conf that prep.sh copies.

The normal run ended with 0 and the failed cp with 1 (set -e stopped the script), and scratch/ was empty both times. To test a signal, the step starts the script in the background with &, which stores its process ID in $!, sends SIGTERM with kill, and reads the status with wait. At your prompt Bash also prints a job number when the job starts ([1] 12345) and a line such as [1]+ Terminated when it ends; the recordings come from a shell without job control, which prints neither.

The script had no handler for SIGTERM, so it died of it (143), and Bash still ran the EXIT trap on the way out: scratch/ is empty. Look at the last line, though. pgrep -a found a sleep 1 still running: Bash exited at once, and the command it was running in the foreground carried on without it. In a real script that child could still be writing into the directory the trap just removed.

Running the EXIT trap after a signal is a Bash feature. POSIX does not require it, and Dash, which is /bin/sh on Ubuntu and what sh script.sh runs, does not do it:

Both died with 143, but only Bash printed "EXIT trap ran". A script that depends on its EXIT trap must run under Bash, which is one more reason for the #!/usr/bin/env bash line from "Your first script".

A handler that does not exit

trap 'commands' TERM replaces what SIGTERM normally does (end the process) with your commands. When they finish, the script carries on from where it was interrupted. Many scripts register one cleanup function for everything, like this:

#!/usr/bin/env bash
# noexit.sh: the same job with one handler for everything (a bug).
set -Eeuo pipefail
workdir=$(mktemp -d)
cleanup() { echo "cleanup: removing $workdir"; rm -rf -- "$workdir"; }
trap cleanup INT TERM EXIT
present() { if [[ -d $1 ]]; then echo yes; else echo no; fi; }
for step in 1 2 3; do
  sleep 1
  echo "step $step of 3 done, work directory present: $(present "$workdir")"
done
echo "report complete"

SIGTERM arrived during step 2. The handler removed the work directory, and then the script simply continued: steps 2 and 3 ran without their directory, "report complete" was printed, the EXIT trap ran cleanup a second time, and the status was 0. Strict mode was on and did not help, because nothing failed. The service manager asked the job to stop; it deleted its own data, kept going and reported success. A cleanup that runs twice can also do damage of its own, for example killing a process ID that the system has since given to another process.

The rule: a handler for INT or TERM must end the script. Keep cleanup in the EXIT trap only, and make the signal handlers exit.

Exit honestly: re-raise the signal

The honest way to end is to die of the same signal. The handler resets the signal to its default action with trap - TERM, then sends it to the script itself with kill -s TERM "$$" ($$ is the script's own process ID). Because the default action is now "terminate", the script dies of SIGTERM, and Bash runs the EXIT trap on the way out.

#!/usr/bin/env bash
# job.sh BATCH: process one batch; on INT or TERM, clean up and die of that signal.
set -Eeuo pipefail
batch=${1:?usage: job.sh BATCH}
workdir=$(mktemp -d)
cleanup() { rm -rf -- "$workdir"; echo "job $batch: cleaned up"; }
trap cleanup EXIT
trap 'trap - INT; kill -s INT "$$"' INT
trap 'trap - TERM; kill -s TERM "$$"' TERM
for step in 1 2 3; do
  sleep 1
  echo "job $batch: step $step of 3 done"
done

Status 143, cleanup done, nothing left in scratch/, and no step ran after the signal. Ctrl-C is harder to reproduce from a lab, because the terminal sends SIGINT to every process of the job in the foreground (the script and the sleep it is running), not only to the script. The step below does the same thing: set -m turns on job control, which your interactive shell already has, so the background job gets its own process group, and kill -INT -- "-$pid" (a negative number means "the whole group") sends SIGINT to all of it. Without set -m, Bash starts background commands with SIGINT ignored, and a script cannot trap a signal that was ignored when it started.

Status 130, and the cleanup ran. Why not simply write trap 'exit 130' INT? A caller can tell the difference, and it matters. run-all.sh processes three batches and keeps going when one fails, a common shape for batch jobs. It will run either job.sh or this variant, whose handlers exit with the same numbers:

#!/usr/bin/env bash
# run-all.sh JOB: run ./JOB for batches 1 to 3; a failed batch does not stop the others.
set -Eeuo pipefail
job=${1:?usage: run-all.sh JOB}
for batch in 1 2 3; do
  if "./$job" "$batch"; then
    echo "run-all: batch $batch done"
  else
    echo "run-all: batch $batch failed with status $?, going on"
  fi
done
echo "run-all: finished"

#!/usr/bin/env bash
# job-exit.sh BATCH: job.sh with handlers that exit with 130 and 143 instead.
set -Eeuo pipefail
batch=${1:?usage: job-exit.sh BATCH}
workdir=$(mktemp -d)
cleanup() { rm -rf -- "$workdir"; echo "job $batch: cleaned up"; }
trap cleanup EXIT
trap 'exit 130' INT
trap 'exit 143' TERM
for step in 1 2 3; do
  sleep 1
  echo "job $batch: step $step of 3 done"
done

Now "press Ctrl-C" during batch 1, once with each job script:

With exit 130 the interrupt stopped batch 1 only. run-all.sh logged a failure and ran batches 2 and 3 to the end, then reported 0: Ctrl-C was, in effect, ignored. With the re-raising job.sh, run-all.sh stopped at once with 130.

The Bash manual describes why. When Bash receives SIGINT while waiting for a command, it waits for the command to end and then looks at how it ended. If the command died of SIGINT, Bash concludes the interrupt was meant for it too and acts on it. If the command exited normally, even with status 130, Bash assumes the command handled Ctrl-C and carries on.

When the trap runs, and what no trap catches

A handler does not interrupt the command that is running. The Bash manual: if Bash is waiting for a command to complete and receives a signal for which a trap has been set, it will not run the trap until the command completes. This script's long command is a stand-in for something like a large copy:

#!/usr/bin/env bash
# long-step.sh: a TERM handler and one long foreground command.
trap 'echo "TERM handler runs"; trap - TERM; kill -s TERM "$$"' TERM
echo "long command starts"
bash -c 'sleep 3; echo "long command finished"'
echo "not printed"

SIGTERM arrived half a second in, but the handler ran only after "long command finished", three seconds later. That is why job.sh works in one-second steps. A script that must react at once runs the long command in the background and waits for it with wait, which a trapped signal does interrupt; the advanced lesson on signals covers that pattern. The same timing explains noexit.sh: its handler ran when the sleep 1 of step 2 ended.

SIGKILL and SIGSTOP (which pauses a process) cannot be caught, blocked or ignored, so no trap runs for them:

Status 137, and the work directory is still in scratch/. The Killed line comes from the shell, which reports a job that died of SIGKILL; at your prompt it reads [1]+ Killed instead of starting with -bash: line 4:. Remove the leftover by hand:

Bash even accepts a trap on KILL and lists it, and it still never runs. ShellCheck reports that line as SC2173:

This is why tools that stop processes send SIGTERM first and SIGKILL only after a grace period (timeout -k in the next lesson, systemd and Kubernetes too), and why a job must cope with leftovers from a killed run, for example by using a fresh mktemp directory each time. One more caution: cleanup runs under the script's own set -e, so a failing command in the EXIT trap ends the trap early and changes the exit status:

The script asked for exit 0. rm failed on a file that was not there, "more cleanup" never ran, and the status became 1. Use cleanup commands that cannot fail, such as rm -rf -- "$dir" or rm -f -- "$file".

Try this

Write maint.sh in a new directory. It starts with set -Eeuo pipefail, sets flag=maintenance.on, removes the flag in an EXIT trap with rm -f -- "$flag", re-raises INT and TERM as job.sh does, then creates the flag with touch -- "$flag" and runs three steps of sleep 1 that each print maintenance step N of 3 done. Check it:

1. ./maint.sh; echo "status: $?"; ls: three steps, status 0, no maintenance.on.

2. ./maint.sh & pid=$!; sleep 1.5; kill -TERM "$pid"; wait "$pid"; echo "status: $?"; ls: one step, status 143, no flag.

3. The same with kill -KILL: one step, a line reporting Killed, status 137, and maintenance.on still there. Remove it by hand.

4. Break it: replace the TERM handler with trap cleanup TERM, send SIGTERM after 1.5 seconds as in check 2, and run ls one second later, before wait. Expect steps 2 and 3 to run while ls shows no flag, and status 0: the load balancer would send traffic to a machine that is still in maintenance.

Takeaway

Put cleanup in one EXIT trap set right after you create the resource, and make every INT and TERM handler end the script by re-raising the signal (trap - TERM; kill -s TERM "$$"). Nothing runs after kill -9, so design the job to tolerate what a killed run leaves behind.

Related