Safe temp files and locks

mktemp, flock, and race conditions.

Every Linux machine ships with a communal scratch space called `/tmp` (short for temporary). Treat it like the counter in a shared office kitchen. Handy, because anyone can set something down for a minute. Risky, because anyone can also set something down, including the person who wants to poison your lunch. If your script always drops its work at the same labeled spot, another user can swap that spot for a trap before your script reaches back for it. This lesson covers the two failures that live in any shared space: making scratch files nobody can hijack, and making sure only one copy of a job runs at a time.

Why /tmp Is Hostile Territory

`/tmp` is world-writable, meaning any local user (and any service account that gets compromised) can create files inside it. That is by design, and you can read the design straight off the directory.

Two things matter in that permission string. The `w` bits say everyone can write. The trailing `t` is the sticky bit, an extra rule that stops one user from deleting or renaming another user's files in the directory. Notice what the sticky bit does not do: it does nothing to stop you from creating a brand new name first. An attacker cannot delete your file, but nothing stops them from getting to a name before you do.

Here is how that turns into an attack. Suppose a job running as root writes to a predictable path like `/tmp/report.$$`, where `$$` is the shell's process ID number (the identifier the kernel gives each running program). Process IDs are easy to guess and easy to watch, and an attacker can pre-plant links for a whole range of likely values. They point that name at a file they want to clobber, using a symlink (a symbolic link, a path that quietly redirects to another path).

On a modern box that write does not go through, and the reason is worth understanding. The redirect (`>`) opens the path to create and truncate it, and it does nothing to stop itself from following a link, so left alone it would follow the symlink and truncate `/root/.ssh/authorized_keys` with root's authority. The only thing standing in the way is a kernel seatbelt that ships switched on by default.

`sysctl` reads and sets kernel settings while the machine is running. `fs.protected_symlinks = 1` tells the kernel a simple rule: inside a sticky, world-writable directory like `/tmp`, refuse to follow a symlink unless the person following it owns the link, or the directory's owner owns the link. Root following an attacker's link matches neither, so the open fails with EACCES (the permission-denied error). That one knob kills the classic `/tmp` symlink attack outright.

Do not mistake a seatbelt for a reason to drive into walls. The knob can be switched off. It only guards sticky, world-writable directories, so a predictable name in a directory that lacks the sticky bit is exposed all over again. And it does nothing about a different plant: instead of a symlink, the attacker creates an ordinary file at `/tmp/report.4127` and waits. Root's redirect truncates and fills a file the attacker already owns and can read, so a root-only report leaks straight into their hands. No exotic exploit is needed. The only ingredient is a name the attacker can predict, and no sysctl removes that. `mktemp` does.

The instinct is to check first: if the file already exists, bail out. That fails too, and the reason is subtle. Checking the path and then writing to it are two separate steps, and an attacker can slip a file or a link into the gap between them. Security people call this a TOCTOU bug (time of check to time of use), the window between when you look and when you act. The only safe move is an operation that checks and creates in one unbreakable step, so there is no gap to slip into. That is exactly what `mktemp` does.

What mktemp Does Differently

`mktemp` builds a random file name and asks the kernel to create it with two flags bolted together: `O_CREAT` (create this file) and `O_EXCL` (fail if anything already exists at this path, including a symlink). You can watch it happen with `strace`, a tool that prints every system call a program makes.

That single `openat` call is the whole defense. Because one system call cannot be interrupted halfway, there is no moment between the check and the create for an attacker to act. If the name already exists, `mktemp` does not overwrite it and does not follow it; it retries with a fresh random name. The file lands at mode `0600` (readable and writable by its owner, nobody else), and your umask (the setting that decides default permissions for new files) can only tighten that, never loosen it. So even a careless `umask 000` cannot widen a temp file past its owner.

A few habits ride along with that. Use `mktemp -d` when a job needs several scratch files; it makes a private directory at mode `0700` and gives you one thing to clean up. Templates need at least three trailing `X` characters, and you want six or more, because those random characters are the entire defense against a guessed name. Honor the caller's `TMPDIR` environment variable instead of hardcoding `/tmp`, so an operator can redirect your scratch space to a safer disk. And pair every temp resource with a `trap ... EXIT` so it disappears on every exit path (the mechanics of traps were the previous lesson).

One Job at a Time With flock

The second shared-space failure is overlap. A nightly backup that usually finishes in 40 minutes will one day take 70, and cron (the scheduler that runs jobs at set times) will happily launch the next run on top of it. Now two processes append to the same archive, dump the same database, and saturate the same link. You want a rule like a single meeting room with one physical key on a hook by the door: to go in you take the key, if the key is gone the room is busy, and when you leave, however you leave, the key goes back on the hook. `flock` is that key, kept by the kernel.

The folk version of this is a lock file you check by hand: if `/tmp/backup.lock` exists, exit, otherwise create it. That carries the same TOCTOU race as the temp-file check, plus a nastier failure. After a crash or a `kill -9`, the file is still sitting there, and every future run refuses to start until a human deletes it. `flock` avoids both problems by putting the lock inside the kernel. You open a lock file on a file descriptor (the small number the kernel uses to track one open file), then ask for an advisory lock on it. Advisory means the lock only constrains programs that also ask for it, like a key system everyone on your team agrees to use. It will not stop a hostile writer, and for coordinating copies of your own script that is all you need. The kernel drops the lock the instant the process ends, so there is no stale state to clean up.

#!/usr/bin/env bash
set -euo pipefail
 
exec 9>/run/lock/nightly-backup.lock   # open (or create) the lock file on file descriptor 9
if ! flock -n 9; then                  # -n: fail now instead of waiting in line
    echo "another run holds the lock; exiting" >&2
    exit 1
fi
 
dump=$(mktemp /var/tmp/dump.XXXXXX)
pg_dump prod > "$dump"
# ...rest of the job. the kernel drops the lock when this process exits.

Start one run in the background, then try to start a second.

The same guarantee fits on one line straight from a cron file, which is the form you will reach for most often.

# run every 30 minutes; skip the run entirely if the last one is still going
*/30 * * * *  deploy  flock -n /run/lock/sync.lock /usr/local/bin/sync.sh

To confirm a lock is actually held, ask the kernel who owns it with `lslocks`.

The holder shows up as `bash` (the interpreter running your script) at the process ID you backgrounded, the `0B` is the empty lock file's size, and the last column points at the lock file itself. That is your proof the lock took, and a handy thing to reach for when a job mysteriously refuses to start and you want to see which run is still holding the key.

Where This Breaks in Production

`flock` coordinates processes on one machine. If you need two servers to agree that only one of them runs a job, that is a distributed lock, and it belongs to a different class of tool (etcd or Consul, which keep a shared ledger across hosts). Keep lock files on local disks too. Over NFS (Network File System, storage shared across a network), `flock` behavior depends on the protocol version and mount options, and a lock that silently fails to apply is worse than no lock at all, because it lulls you into thinking you are protected.

Placement matters. Root jobs conventionally lock under `/run/lock`, a directory that lives in memory and is wiped clean at every boot, which is exactly what you want from a lock file. Jobs that do not run as root should lock a file in a directory they own, such as the one the system hands each user at `$XDG_RUNTIME_DIR` (a private per-user scratch directory, usually `/run/user/1000`).

Keeping Scratch From Vanishing

Files under `/tmp` are not permanent. A daily timer, `systemd-tmpfiles-clean`, deletes old files based on an age set in `/usr/lib/tmpfiles.d/tmp.conf`. Read yours before you park anything you care about there.

# See tmpfiles.d(5) for details
 
# Clear tmp directories separately, to make them easier to override
q /tmp 1777 root root -
q /var/tmp 1777 root root 30d

The last column on each line is the age. Here `/var/tmp` gets swept of anything untouched for 30 days, while `/tmp` shows `-`, meaning this box applies no time-based sweep to `/tmp` at all. Other distributions set `/tmp` to something like `10d`. The point is to check rather than assume: if a job needs scratch that outlives a reboot or a cleanup sweep, put it in `/var/tmp` or a directory your service owns, not `/tmp`.

For a systemd service (systemd is the program that starts and supervises services on boot), one line closes the entire symlink-attack class. `PrivateTmp=yes` gives the service its own private `/tmp` and `/var/tmp`, invisible to everyone else on the box.

[Unit]
Description=Nightly report generator
 
[Service]
Type=oneshot
ExecStart=/usr/local/bin/report.sh
PrivateTmp=yes

There is a catch worth knowing before it bites you. With a private `/tmp`, two services can no longer meet through a shared `/tmp` path, so a script and its helper that used to pass files that way will suddenly see different, empty directories. Give each one an explicit path they both own instead. And when you are not using a private `/tmp`, remember that the ordinary `/tmp` is world-readable: mode `0600` hides the contents of your file, but anyone can run `ls /tmp` and read the names. Never encode a hostname, username, or ticket number into a temp file name.

Every rule here is a convention, and conventions rot unless a machine enforces them. The good news is that most of these mistakes are mechanical enough for a linter to catch: an unquoted variable, a predictable temp path, a `trap` string that expands at definition time instead of when the signal fires. The next lesson wires ShellCheck into your pipeline so a script that trips one of these checks never reaches a production crontab in the first place.

Related