Industrial-grade error handling in Bash

set -Eeuo pipefail honestly, ERR traps, error propagation and stack traces.

A factory line has a pull-cord. When a station hits a bad part, someone yanks the cord, the whole belt stops, a light flashes over the station that failed, and a cleanup crew sweeps up before anything moves again. A shell script with no error handling is that same line with the cord cut. One command jams, the belt keeps rolling, and at the end you ship a crate that looks sealed and correct but is missing half its parts.

For security work, that crate is dangerous. Take a script that pulls a fresh blocklist (a list of bad network addresses to reject) and loads it into the firewall. If the download quietly fails and the script rolls on, you flush yesterday's rules and load nothing in their place. The machine reports success and now blocks no one. That is failing open, and it is exactly the outcome an attacker hopes for. The point of heavy-duty error handling in Bash (Bourne Again Shell, the shell that runs most Linux server scripts) is to make a script stop at the first real problem, say where it stopped, tidy up after itself, and fail closed.

The Preamble That Stops the Line

Four settings turn an ordinary script into one with a working cord, and you set them together on the first real line: set -Eeuo pipefail. Here is what each letter buys you. The -e flag, whose long name is errexit (short for 'exit on error'), stops the script the moment a command fails and nobody is checking that command's result. The -u flag (nounset) treats reading a variable you never set as an error instead of quietly handing back an empty string. That catches a typo like $CUSTMER_ID before it expands to nothing and a line like rm -rf "$DIR/data" starts deleting from the wrong place. The -o pipefail option deals with pipelines (two or more commands joined by |, the bar that feeds one command's output straight into the next). By default a pipeline reports only its last stage's exit status. Turn on pipefail and the whole pipeline fails when any stage fails. Without it, curl (the command-line tool that downloads files over the network) piped into jq (a small program for pulling values out of JSON, the text format most web APIs return) reports success whenever jq is happy, even when curl fetched nothing. And -E (errtrace) is the flag people forget. It carries your ERR trap down into functions, subshells (separate copies of the shell), and command substitutions (the $(...) form that runs a command and pastes its output into the line), so the handler you install fires where the real work happens instead of getting dropped at the first function boundary.

Two more lines finish the header. Think of a form-scanner that reads a line of text and starts a new field at every blank space. Bash does the same thing when it splits unquoted text into words, and IFS (Internal Field Separator, the characters Bash treats as the gaps between words) is the list it cuts on. It defaults to space, tab, and newline, so a filename like report Q3.pdf quietly splits into two arguments, report and Q3.pdf. Set IFS to only newline and tab and that whole class of surprise goes away. Then an EXIT trap gives you a cleanup crew that runs however the script ends: on success, on failure, or when someone hits Ctrl-C to kill it. For security work that crew is not optional. Your script probably wrote the downloaded blocklist, an access token, or a private key into a scratch folder, and you do not want that sitting in /tmp (the shared temporary directory every user on the box can list) after a crash. mktemp -d creates a fresh directory only you can read, and the trap wipes it on the way out, every time.

#!/usr/bin/env bash
set -Eeuo pipefail
IFS=$'\n\t'
 
# a private scratch dir, wiped on ANY exit (success, error, or Ctrl-C)
tmp="$(mktemp -d)"
cleanup() { rm -rf "$tmp"; }
trap cleanup EXIT
 
# on the first uncaught failure, say what broke and where, then let -e stop us
trap 'rc=$?; printf "ERR rc=%d at %s:%d: %s\n" \
  "$rc" "${BASH_SOURCE[0]}" "$LINENO" "$BASH_COMMAND" >&2' ERR

That is the whole safety rig. Two flags to stop and report, one to make pipes honest, one to make the traps reach into your functions, and two traps for location and cleanup. Before you trust it, watch two of the flags actually bite.

Where set -e Looks the Other Way

Here is the part that trips people, and the part an attacker's luck rides on. A smoke alarm has a mute button you press on purpose while you sear a steak, because right then the smoke is expected, not an emergency. The errexit flag works the same way. It stays quiet whenever the shell is already inspecting a command's exit status, because in those spots a failure is the answer you asked for, not a fault. Write if grep -q root /etc/passwd and the shell has to see whether grep found the line before it can pick a branch, so bailing out there would make no sense. The same logic silences errexit in four places: anything to the left of && or ||, the test of an if, while, or until, any command after a ! (the 'not' that flips success and failure), and the quiet killer, a command substitution that shares its line with a declaration such as local, declare, export, or readonly.

That last one is a genuine foot-gun. Writing local body=$(curl -fsS https://ti.internal/blocklist.txt) looks like it runs curl and stops if the fetch fails. It does not. Bash reads that line as the local builtin being handed one already-finished argument, and local almost always succeeds, so its exit status of zero is what errexit checks. So curl's failure gets thrown on the floor. Point it at a feed that returns 404 (a web server's way of saying 'not found') and watch the script stroll right past the problem.

#!/usr/bin/env bash
set -Eeuo pipefail
 
load() {
  local body=$(curl -fsS https://ti.internal/blocklist.txt)
  echo "loaded ${#body} bytes"
}
 
load
echo "applied blocklist"

Read that output the way an operator would. The curl call plainly failed. The script still printed loaded 0 bytes, still printed applied blocklist, and still exited zero, which any job that called it will read as success. You have deployed an empty blocklist and reported a clean run. The fix is to split the declaration from the assignment. On its own line, body=$(...) is a plain assignment whose exit status is curl's, so errexit finally sees the truth. Add an explicit handler so the failure leaves a readable note and returns non-zero.

#!/usr/bin/env bash
set -Eeuo pipefail
 
load() {
  local body
  body=$(curl -fsS https://ti.internal/blocklist.txt) \
    || { echo "blocklist fetch failed, keeping current rules" >&2; return 1; }
  echo "loaded ${#body} bytes"
}
 
load
echo "applied blocklist"

Now it fails closed. The applied blocklist line never runs, the return 1 travels up out of load, and errexit stops the script cold. When you do want to test a command yourself and act on the result, write that intent out in the open. Process substitution makes it obvious, for example if ! grep -q ready <(status_cmd); then echo 'not ready' >&2; exit 1; fi, where <(status_cmd) hands the output of status_cmd to grep as if it were a file. Anyone reading it can see which failure you are handling and which you are letting through.

A Trap That Says Where It Died

Stopping is half the job. The other half is telling whoever runs the script exactly where it broke, because that person is often you at 3am during an incident. An ERR trap is like a dashcam that keeps rolling until the crash: it is a handler Bash runs the moment a command trips errexit. Two built-in variables make its report worth reading. $BASH_COMMAND holds the text of the command that just failed, and $LINENO holds the line it sat on. Print those two and a useless 'something broke' turns into 'curl on line 15 returned 22.' The -E flag from the header is what lets this handler follow execution down into your functions. Without it, a failure inside a function never reaches the trap at all.

For a script other people run, go one step further and print the full call chain, the way programs in bigger languages print a stack trace (the who-called-whom list that led to the crash). Bash keeps that chain in three arrays: FUNCNAME for the functions currently running, BASH_SOURCE for their files, and BASH_LINENO for the lines. You can walk them by hand, but the caller builtin does it for you: caller 0 is the closest frame, caller 1 the one that called it, and so on until it runs out and returns non-zero. Loop over it and you get a map of exactly how execution reached the failing line.

#!/usr/bin/env bash
set -Eeuo pipefail
 
trace() {
  local rc=$? i=0
  echo "FATAL rc=$rc: ${BASH_COMMAND}" >&2
  while caller "$i" >/dev/null 2>&1; do   # caller prints: <line> <function> <file>
    echo "  at $(caller "$i")" >&2
    i=$((i+1))
  done
  exit "$rc"
}
trap trace ERR
 
fetch()   { curl -fsS "$1" -o "$2"; }
refresh() { fetch "https://ti.internal/blocklist.txt" "$dst"; }
 
dst=/run/blocklist.txt
refresh

Read that from the bottom up and you have the whole path: line 19 called refresh, line 16 called fetch, and the curl on line 15 is where it died with code 22. On the top-level line the function name shows as main, which is Bash's name for the body of the script itself. That four-line report is the difference between a colleague fixing it on the spot and a colleague opening a ticket.

Prove the rig works before you lean on it. Break a step on purpose, point the fetch at a dead host or a 404, run the script, and check two things: it exits non-zero, and the reported line names the step you broke. If it prints success instead, your check is sitting in one of the blind spots above. You can also let a tool find the worst one for you. ShellCheck (a free static analyzer for shell scripts) flags local x=$(...) as SC2155 with the message 'Declare and assign separately to avoid masking return values,' which is the exact bug from the broken example.

One honest habit keeps all of this trustworthy. When you truly want a command to be allowed to fail, say so in the open with cmd || true, and leave a comment saying why. Every || true is a spot where you chose to keep going through a failure, which is a spot where the script can fail open. Treat each one like a door you left unlocked on purpose: fine if you meant it, worth a label so the next person on call knows you did.

Try this

Work through “A Trap That Says Where It Died” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: set -u bites empty arrays and old bash. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related