Temp files, locks and timeouts

mktemp, write-then-rename, flock, and timeout for commands that hang.

Three problems turn up in almost every ops script that runs unattended: it needs scratch files in a directory other users share, it replaces a file that other programs read, and it must not run twice at once or hang forever. In this lesson you will see what can go wrong with a predictable name in /tmp and what Ubuntu's kernel settings do and do not protect, create temporary files and directories with mktemp, publish a file with write-then-rename so readers never see half of it, allow only one run at a time with flock, and put a time limit on a command with timeout. The examples run in ~/bash-ops/temp. The first section borrows root for a moment to create a second account, and removes it again at the end of the section.

Shared temp directories and predictable names

All three directories are writable by everyone (rwx for others), so any user and any compromised service account can create names in them. The t at the end is the sticky bit: in a sticky directory only a file's owner (or the directory's owner, or root) can delete or rename it. It does nothing to stop someone from creating a name first. On Ubuntu 26.04, /tmp is a tmpfs (it lives in memory and is empty after a reboot), and systemd-tmpfiles deletes entries not used for 10 days in /tmp and 30 days in /var/tmp.

Now imagine a job that always writes /tmp/bo-temp-report.txt. Another user who knows the name can create it first, either as a symbolic link to a file they want the job to overwrite, or as an ordinary file they can read. Two kernel settings defend against exactly this, and you can read them without root:

The kernel documentation defines them. protected_symlinks = 1 applies in sticky, world-writable directories: a symbolic link there is followed only if it belongs to the user following it or to the directory's owner. protected_regular = 2 applies in sticky directories that are world-writable or group-writable: opening an existing file with create, as > does, is refused if the file belongs to someone else (again, unless it belongs to the directory's owner).

To see them work you need a second user. With sudo (as root), create a test account, bo-temp-other, and a shared drop directory, /srv/bo-temp-drop, that is world-writable but not sticky. That directory is deliberately unsafe; the end of this section removes both. Also create the app.conf the traps will aim at:

sudo -u bo-temp-other runs one command as that user. The other user plants a symbolic link and a 0666 file under the names the job will write in /tmp, and a third link in the drop directory:

Both links point at your app.conf. Now the job, running as you, writes its two files in /tmp:

Both writes were refused and app.conf is intact. Note what that means for the job, though: it failed. Whoever planted the names can stop it from ever producing its report. In the directory without the sticky bit, neither setting applies:

> followed the link and replaced your own app.conf with the report. So the settings are a useful safety net for /tmp and /var/tmp on this system, not a fix: they can be switched off, they cover only sticky directories, and a predictable name still lets another user make your job fail. The fix is to never use a name someone else can predict or create first.

Clean up now: remove the planted files (they belong to the other user, so this needs sudo), the drop directory and the account, and restore app.conf. userdel notes that the account had no mail spool, and id confirms the account is gone:

mktemp: a new, private name every time

mktemp creates a file with a random name and prints the name; mktemp -d creates a directory. It creates it in one step that fails if anything already exists at that path, so it never opens a planted file or follows a planted link, and only you can use the result. A umask lists the permission bits that new files and directories do not get: the usual 022 removes write permission for group and others, 077 removes everything for them, and 000 removes nothing. The step below sets umask 000, the most permissive setting, in a subshell ( ) so the setting does not stay in your terminal:

The file got mode 600 and the directory 700 anyway: read and write (and enter, for the directory) for the owner only. A umask can only take permissions away, and mktemp asks for no more than that. strace shows the system call that creates the file:

O_CREAT|O_EXCL means "create, and fail if the name exists", and 0600 is the mode. On a name collision mktemp does not open the existing file. Two more options matter in scripts:

--tmpdir with a template creates the name in $TMPDIR if it is set, otherwise in /tmp, so an operator can move your scratch space without editing the script; the template's Xs become random characters. Too few Xs is an error; use at least six. Pair every mktemp with an EXIT trap on the next line, as "Traps, signals and cleanup" did, so the file or directory goes away on every exit path.

Publish a file in one step: write, then rename

A blocklist, a generated config or a status file is read by other programs while your script updates it. Writing it in place with > empties the file first and then fills it, so a reader can see a partial file, and a failure halfway leaves the partial file behind. This generator stops at the first line that does not look like an IPv4 address (the pattern checks the shape, not that each number is at most 255):

#!/usr/bin/env bash
# make-list.sh SRC: print the IPv4 addresses in SRC, one per line; stop at the first bad line.
set -Eeuo pipefail
src=${1:?usage: make-list.sh SRC}
n=0
while IFS= read -r line; do
  n=$((n + 1))
  if [[ ! $line =~ ^[0-9]{1,3}(\.[0-9]{1,3}){3}$ ]]; then
    echo "make-list.sh: $src line $n does not look like an IPv4 address: $line" >&2
    exit 1
  fi
  printf '%s\n' "$line"
done < "$src"

Create a good source list and one with a hand-edited comment on its second line, then write the list in place:

The generator stopped at the comment with status 1, but the redirection had already emptied blocklist.txt: it now holds one address instead of three, and a firewall reloading it would unblock the other two. The safe pattern writes a new file next to the old one and swaps it in with mv:

#!/usr/bin/env bash
# publish.sh SRC DEST: rebuild DEST from SRC, replacing it in one step or not at all.
set -Eeuo pipefail
src=${1:?usage: publish.sh SRC DEST}
dest=${2:?usage: publish.sh SRC DEST}
# The temporary file sits next to DEST, so mv is a rename within one file system.
tmp=$(mktemp -- "$dest.XXXXXX")
trap 'rm -f -- "$tmp"' EXIT
./make-list.sh "$src" > "$tmp"
# An empty result is valid output, but for a blocklist it would unblock everything: refuse it.
if [[ ! -s $tmp ]]; then
  echo "publish.sh: refusing to publish an empty list from $src" >&2
  exit 1
fi
chmod 0644 -- "$tmp"   # mktemp created it 0600; readers of DEST need 0644
mv -f -- "$tmp" "$dest"
echo "published $dest: $(wc -l < "$dest") entries"

When the source and target are in the same file system, mv is a single rename system call, which replaces the target atomically: a reader opening blocklist.txt gets either the whole old file or the whole new one, never a mix. That is why the temporary file is created next to the target (mktemp -- "$dest.XXXXXX") and not in /tmp: /tmp is a separate tmpfs here, and across file systems mv has to copy and delete, which is not atomic. The chmod matters too: without it the published file would keep mktemp's 0600 and a reader running as another user could not open it.

The failed run left blocklist.txt exactly as it was, and the EXIT trap removed the half-written temporary file: ls finds no blocklist.txt.XXXXXX. An atomic swap guarantees a complete file, not a sensible one: an empty source is not an error for make-list.sh, and an empty blocklist would unblock everything. So publish.sh refuses an empty result:

Status 1, and the three good entries stay. A failure, or a result that cannot be right, leaves the last good version in place.

One run at a time: flock

A job scheduled every 30 minutes will one day take 40, and the scheduler will start the next run on top of it. Checking for a "lock file" by hand (if it exists, exit; otherwise create it) has the same gap between check and create as a predictable temp name, and after a crash the file stays and blocks every later run. flock (util-linux) asks the kernel for a lock on an open file instead. The kernel releases it when the last descriptor for that open file is closed, which happens when the process ends, even after kill -9.

#!/usr/bin/env bash
# nightly.sh: a job that must never run twice at the same time.
# Exit status: 0 done, 1 failed, 3 another run holds the lock.
set -Eeuo pipefail
lock_dir=${LOCK_DIR:-$HOME/.local/state/nightly}
mkdir -p -- "$lock_dir"
exec 9> "$lock_dir/nightly.lock"
if ! flock -n 9; then
  echo "nightly.sh: another run holds $lock_dir/nightly.lock" >&2
  exit 3
fi
echo "nightly.sh: lock taken by PID $$"
sleep "${WORK_SECONDS:-3}"
echo "nightly.sh: PID $$ done"

exec 9> file opens the lock file on file descriptor 9 for the rest of the script (creating it if needed). flock -n 9 asks for an exclusive lock on that descriptor and fails at once, with status 1, if another process holds it; without -n it would wait. The script turns that into status 3, documented at the top, following the exit-status contract from "Command-line options". The lock file lives in a directory only deploy can write. A shared, world-writable directory such as /run/lock would let another user create the file first, or hold a lock on it forever. For a system service, systemd's RuntimeDirectory= gives the service a directory of its own under /run.

The step starts one run in the background, waits until it reports the lock (otherwise the second run could start first), then starts another. The wait loop runs inside timeout 10 bash -c '...', so a run that never reports the lock cannot keep the step waiting forever; the last section explains timeout:

The second run exited 3 without doing anything. lslocks lists the lock: held by the first run's Bash process, whose PID matches the one in first.out, which finished with 0. flock also has a command form, handy in a crontab line, and -w SECONDS waits for the lock up to that long:

flock -w 1 gave up after a second with status 1; -E chooses a different status if you want one. flock -w 10 waited until the first run ended and then ran echo. One rule is easy to break: never delete the lock file, not even in cleanup. The lock belongs to the open file, not to its name:

After rm, the second run created a new file with the same name and locked it, while the first run still held its lock on the old, deleted file. Both said "lock taken" and both ran at the same time. An empty lock file costs nothing, so leave it where it is. A related trap: a command started in the background by the script inherits descriptor 9 and keeps the lock after the script exits; the advanced lesson on processes and file descriptors shows it.

Time limits: timeout

A command that waits on a network or a stuck mount can hang forever, and with it the lock it holds. timeout DURATION command runs the command and sends it SIGTERM when the time is up:

Status 124 means "timed out". A command that finishes in time returns its own status: 0 for sleep 1, 1 for the grep that found nothing. A command can ignore SIGTERM; here trap "" TERM makes Bash and its sleep ignore it, standing in for a stuck program. -k 2 sends SIGKILL two seconds after the SIGTERM:

137 is 128 + 9: the command was killed. Ubuntu 26.04's timeout comes from uutils (the Rust coreutils that Ubuntu now installs by default), and one detail differs from GNU's. GNU coreutils documents that when you choose SIGKILL as the signal with -s KILL, the exit status is 128 + 9 rather than 124. The GNU version is installed as gnutimeout:

uutils returned 124. GNU timeout sends the signal to its whole process group, itself included, so it died of SIGKILL too: the shell printed Killed (the -bash: line 5: prefix comes from the lab's shell) and reported 137. If a script decides "timed out" from the status, use the default SIGTERM with -k for the fallback, and treat both 124 and 137 as a timeout.

Try this

Copy nightly.sh to nightly-wait.sh in a new directory and change two things. Replace flock -n 9 with flock -w "${LOCK_WAIT:-5}" 9 and make the message say lock still held after ${LOCK_WAIT:-5}s. Replace the sleep with timeout "${WORK_LIMIT:-10}" sleep "${WORK_SECONDS:-3}", capture its status with the rc=0; ... || rc=$? idiom from "Strict mode, honestly", and exit 1 with work timed out after ${WORK_LIMIT:-10}s on 124 (and with a message for any other non-zero status). Then check, starting a first run in the background and waiting for its "lock taken" line as the lesson did:

1. A second run with LOCK_WAIT=1: expect lock still held after 1s and status 3.

2. A second run with LOCK_WAIT=10: expect it to wait, print lock taken and done with its own PID, and exit 0.

3. WORK_SECONDS=5 WORK_LIMIT=2 ./nightly-wait.sh on its own: expect lock taken, then work timed out after 2s and status 1.

Takeaway

Never write to a name someone else can predict: create scratch space with mktemp and remove it in an EXIT trap, publish shared files with a temporary file in the same directory and mv, guard single-run jobs with flock on a lock file in a directory you own (and never delete it), and give anything that can hang a timeout.

Related