Signals, traps, timeouts & graceful shutdown

SIGTERM/SIGINT handling, cleanup, timeouts and job control.

A shell script that opens a temp file full of tokens, starts a background job, or grabs a lock has quietly made promises to the rest of the machine. The interesting question is what happens when someone tells it to stop before it is finished. A line cook mid-service doesn't get to ignore "kitchen's closing." They finish the plate in their hands, turn off the burner, put the knife away. Your automation needs the same reflex, and signals are how that message arrives.

Get this wrong and you leave a trail: half-written files, orphaned child processes still burning CPU, a lock nobody releases, or a secret sitting on disk that was supposed to be scratch. Get it right and your script cleans up and exits promptly whether it finished normally, hit an error, or was told to die by systemd (the init system and service manager on most modern Linux), Kubernetes (the container orchestrator), or a CI (continuous integration) runner. This lesson is about building that reflex the reliable way.

The taps on the shoulder: which signals matter

A signal is a short, fixed message the kernel (the core of the operating system that talks to the hardware) delivers to a process, like a knock on a door. It carries no words, only a number, and the process decides how to answer. You can list them all with one command.

Four of these do most of the work. SIGTERM (signal to terminate, number 15) is the polite "please wrap up" that orchestrators send first. SIGINT (signal to interrupt, number 2) is what Ctrl-C sends from your keyboard. SIGHUP (signal for hang-up, number 1) originally meant the terminal line dropped, and many daemons now read it as "reload your config." SIGKILL (number 9) and SIGSTOP (number 19) are the two you cannot catch, block, or ignore; the kernel handles them for you. That is the point of them, and it is also a defender's blind spot. A local attacker running as your UID (user identifier) can freeze your monitoring script with a single SIGSTOP. The process does not die. It sits in state T (stopped), emitting nothing, until something resumes it. Watch for stopped processes that should be running.

When a signal ends a process, the exit code is 128 plus the signal number. So SIGTERM leaves 143, SIGINT leaves 130, and SIGKILL leaves 137. Memorize those three. In a log or a kubectl status, 143 tells a responder the process cooperated and shut down on request, while 137 tells them it was force-killed after it refused to stop (or the out-of-memory killer got it). That single number is often the first clue in an incident.

One cleanup path, every exit

Here is the trick that keeps this sane. Instead of writing cleanup code at every place the script might leave, you write it once, in a single function, and attach it to the EXIT trap. Think of it as the checklist taped by the kitchen's back door: whoever leaves, by whatever route, runs the list. A trap is Bash's word for "when this signal or event happens, run this code instead of the default." The EXIT trap runs on a normal finish, on an error, and after a caught signal. Your signal traps then stay tiny: they record what happened and exit, which triggers the one real teardown.

#!/usr/bin/env bash
set -Eeuo pipefail   # -E: ERR trap reaches functions; -e: stop on error;
                     # -u: error on unset var; pipefail: a pipe fails if any part does
 
work="$(mktemp -d)"   # scratch dir this script owns; may hold a token
child=""
 
cleanup() {
  local rc=$?                                   # code of whatever ended the script
  [[ -n "$child" ]] && kill "$child" 2>/dev/null || true
  rm -rf "$work"
  echo "cleaned up ${work}, exit ${rc}" >&2
}
trap cleanup EXIT                               # one teardown, every exit path
trap 'exit 143' TERM                            # SIGTERM -> exit 128+15, fires EXIT
trap 'exit 130' INT                             # Ctrl-C  -> exit 128+2,  fires EXIT
 
echo "work dir: ${work}" >&2
sleep 30 & child=$!                              # stand-in for the real background job
wait "$child"                                   # signals can land here now (see below)

Two details carry real weight. First, local rc=$? must be one line. The $? captures the exit status of whatever ran last before cleanup started, and if you split it across two lines the local succeeds first and resets $? to 0, so you would report success on a failure. Second, child is set to an empty string up front. Under set -u an unset variable is an error, and if the signal arrives before the background job launches, cleanup still needs a defined value to test.

Now send it the exact signal an orchestrator sends, and watch the single cleanup path run.

Catch the signal while you wait

There is a reason the long job runs with & in the background and the script sits in wait, and it is the sharpest edge in this whole topic. Bash does not interrupt a running foreground command to run a trap. If your script is sitting on a plain sleep 30 in the foreground and SIGTERM arrives for the shell, Bash writes the request down and runs your handler only after sleep returns, up to 30 seconds later. The wait builtin is the exception: when a signal you have trapped arrives while the shell is blocked in wait, wait returns right away with a status above 128, and your trap runs immediately. So backgrounding the real work and waiting on it is what makes your script actually responsive to "stop now."

If dropping in-flight work is unacceptable (a half-processed batch, a partial upload), your TERM handler should forward the signal to the child and wait for it to finish, rather than exiting straight away. For idempotent or retryable work, exiting fast and letting the next run pick it up is the simpler, safer choice. Decide which one your job is before you write the handler.

Timeouts so nothing hangs forever

Any command that touches the network can hang, and a hung automation job is worse than a failed one because a failure alerts and a hang just sits there silent. A stuck vulnerability scan produces no findings and no error, so nobody looks. Wrap external calls in timeout, which is a fence with a clock: it sends SIGTERM at the deadline, and with -k it follows up with SIGKILL a few seconds later in case the command ignored the polite request.

Exit 124 is timeout's way of saying "the deadline passed and I stopped it." If the follow-up SIGKILL was needed, you get 137 instead, the same force-kill signature you would read anywhere else. For a ceiling on the whole script rather than one command, set an alarm with a background subshell that signals the main shell.

# hard ceiling for the whole run: self-destruct after 10 minutes
( sleep 600 && kill -TERM "$$" ) &     # $$ is THIS script's PID, even inside the subshell
watchdog=$!
trap 'kill "$watchdog" 2>/dev/null || true' EXIT   # cancel the timer if we finish early

The $$ there is worth a pause. Inside a ( ... ) subshell, $$ still expands to the PID of the original script, not the subshell, so kill -TERM "$$" reaches your main process. (If you actually wanted the subshell's own PID, that is $BASHPID.) The EXIT trap cancels the watchdog when the real work finishes on time, so the alarm never fires on a healthy run.

Graceful shutdown under systemd and Kubernetes

Under an orchestrator your script is often PID 1, or close to it, and the platform runs a fixed routine when it wants the service gone: send SIGTERM, wait a grace period, then send SIGKILL to anything still breathing. systemd does this across the whole cgroup (control group, the kernel feature that groups a service's processes together) by default, so every child gets the signal, not only the top process. You tell systemd how long to wait in the unit file.

[Unit]
Description=Nightly vulnerability scan worker
After=network-online.target
 
[Service]
Type=exec
ExecStart=/usr/local/bin/scan-worker.sh
KillSignal=SIGTERM        # signal sent on stop (this is the default, shown for clarity)
KillMode=control-group    # signal the whole cgroup, not just the main PID (default)
TimeoutStopSec=45s        # how long to wait after SIGTERM before sending SIGKILL
Restart=on-failure
 
[Install]
WantedBy=multi-user.target

When the script honors SIGTERM, systemctl stop is clean and the journal shows a tidy deactivation. When the script ignores it, the same journal hands you the exact forensic signature of a script that never installed a trap.

That status=9/KILL after stop-sigterm timed out means the process sat through the whole 45 seconds and had to be shot. Kubernetes shows you the same story through the exit code. It sends SIGTERM to PID 1, waits terminationGracePeriodSeconds (30 by default), then force-kills, and you read the result off the pod.

Exit 137 is 128 plus 9, so this container was SIGKILLed after ignoring SIGTERM for the full grace period (if Reason said OOMKilled, the same 137 would mean the memory limit did it instead). A cooperative shutdown would read 143. The usual culprit behind a pod that takes the entire grace period to die is a wrapper shell that never forwarded the signal. When your script is only a launcher, exec the final command so the workload replaces the shell and becomes PID 1 itself, receiving SIGTERM directly with no middleman to drop it. A plain shell as PID 1 also fails to reap zombie children, which is a second reason to exec the real process or run a tiny init like tini.

To confirm your handling actually works, do not trust the code review, drive it: start the service, run systemctl stop (or send kill -TERM to the PID), and read the exit code and the journal. A 143 and a clean deactivation line mean the traps fired and cleanup ran. A 137 with stop-sigterm timed out means something is still swallowing the signal, and now you know exactly where to look.

Try this

Work through “Graceful shutdown under systemd and Kubernetes” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a foreground command can swallow your handler. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related