Bash internals: subshells, FDs & process substitution

Subshells vs the current shell, file descriptors, process substitution and coprocesses.

Most advanced Bash bugs trace back to one question you never thought to ask: did the shell start a brand-new process to run this, or did it run it right here in the shell you are sitting in? Get that wrong and your script quietly loses track of things. A counter that should read 47 reads 0. A secret you fetched comes back empty. A security gate that should fail passes clean.

Here is the mental model. When Bash runs part of your script in a subshell, it is like handing a photocopy of your notebook to an assistant, sending them into a side room, and locking the door. They can scribble all over that copy. When they walk back out, they hand you nothing but a thumbs-up or thumbs-down (the exit status) and whatever they read aloud on the way (their output). Your original notebook is untouched. A subshell (a child process, meaning a separate running copy of the shell) gets its own copy of every variable, its own current directory, its own everything. When it ends, its changes evaporate.

Where Bash Quietly Forks

So the real skill is spotting where subshells appear, because they are not always obvious. Parentheses ( ... ) run their contents in a subshell. A brace group { ...; } runs in the current shell, same process, so its changes stick. Command substitution $(...) runs in a subshell. Backgrounding a job with & runs it in a subshell. And the one that catches everybody: every stage of a pipeline a | b | c runs in its own subshell.

The brace group changed x for real. The parentheses changed a copy that no longer exists. Same-looking syntax, opposite outcome.

The Pipeline That Eats Your Counter

This is where the security payoff starts. Say you are counting something that matters: failed SSH (Secure Shell, the encrypted remote-login protocol) logins, packages with known vulnerabilities, files an attacker touched, and the count decides whether a check passes. You pipe a stream into a while loop. The loop runs in a subshell. It counts perfectly. Then it dies, and takes your count with it.

Three lines went by. The count is 0. The loop really did run and really did reach 3, but that 3 lived in the subshell on the right side of the pipe and died with it. If this loop were counting critical findings to gate a deploy, you would ship with the gate wide open and never see it.

Two clean fixes. The first keeps the loop in your current shell by feeding it through a redirection instead of a pipe, using process substitution (which the next sections explain in full):

The second fix tells Bash to run the last stage of a pipeline in the current shell. That is the lastpipe option, added in Bash 4.2. It only takes effect when job control is off, which is already the default inside scripts; at an interactive prompt you turn job control off with set +m first:

Now the count survives. Pick whichever reads clearer. The redirection form works even on Bash older than 4.2, so it is the portable choice; lastpipe is tidier when you already have a natural pipeline you do not want to rewrite.

File Descriptors: The Wiring Behind Redirection

Redirection looks like special syntax. It is really you rewiring numbered slots. Every process is born with three numbered mail slots, and the kernel (the core of the operating system that talks to the hardware) uses those numbers to know where reading and writing go. These slots are file descriptors (small integers that stand for an open file, pipe, socket, or terminal). Slot 0 is stdin (standard input, where input arrives). Slot 1 is stdout (standard output, where normal results go). Slot 2 is stderr (standard error, where error messages go). You can open more slots yourself: 3, 4, and up.

The notation 2>&1 means point slot 2 at wherever slot 1 currently points. Read that slowly. It copies the destination that slot 1 has at that exact instant. It does not tie the two slots together for the future. That is why the order of your redirections changes the result.

Same two words, swapped order, and the error lands in a different place. This bites in logging. A monitoring agent that ingests your script's stdout expects clean data; the errors belong on stderr where a human or an alert pipeline reads them. Write cmd 2>&1 >log when you meant cmd >log 2>&1 and your errors leak into the data stream or drop on the floor, and your alerting goes silent at the exact moment something is breaking.

Open Your Own Descriptors

You are not stuck with the three slots you were born with. You can open your own, park the original stdout safely to one side, redirect for a while, then put it back exactly as it was. Think of it as propping a specific door open with a numbered wedge so you can find that same door again later.

exec with no command after it changes the shell's own descriptors and keeps them changed. Here is an append-only log sink on descriptor 3, the real stdout stashed in descriptor 4, and a clean restore afterward:

audit: scan started at 2026-07-17T10:22:14+00:00

Only the last line reached your terminal. The audit line went into the log through descriptor 3. Two details carry weight for security work. Using >> (append) instead of > (truncate) means reopening the log never wipes what is already there, which is exactly what you want from an audit trail. And stashing the real stdout in descriptor 4 before you redirect gives you a guaranteed way back; you are not guessing where stdout used to point, you saved it.

Process Substitution: A Command Dressed As A File

Plenty of tools refuse to read from a pipe. They demand a filename you can point at. Process substitution is Bash handing such a tool a fake filename that is secretly a live command. It is like giving someone a P.O. box number instead of a house: mail addressed to the box still reaches you, but there is no house on any street.

Write <(cmd) and Bash runs cmd, wires its output to a pipe, and hands you a path to that pipe, usually /dev/fd/63. The tool opens that path like a file and reads the command's output. Write >(cmd) for the reverse: a path you write into, which feeds the command's input.

That /dev/fd/63 is not a file on disk. The arrow gives away what it really is: a pipe. Nothing touched the filesystem, and that is the whole point for security work.

Temporary files are a classic weak spot. You write sensitive output to /tmp/scan.json, a tool reads it a moment later, and in that gap an attacker on the same box can swap the file for a symlink or read your secrets straight off a world-readable path. That gap has a name: a TOCTOU bug (time-of-check to time-of-use, a race where the file you checked or created is not the file you actually end up using). Process substitution shuts the gap by never creating a file at all. Three patterns come up constantly (kubectl is the Kubernetes command-line tool):

Coprocesses: A Helper You Keep On The Line

Sometimes you do not want a fresh process for every question. You want one helper running and you want to keep talking to it: ask, hear back, ask again, without hanging up and redialing each time. That is a coprocess. Bash starts a background command and wires two pipes to it, one you write into (its input) and one you read from (its output). It even remembers its process id for you.

coproc NAME { command; } launches command in the background. Bash fills an array: ${NAME[1]} is the descriptor you write to, ${NAME[0]} is the descriptor you read from, and NAME_PID holds its process id. Here is a coprocess that upper-cases whatever line you send it:

Notice the fflush() in that awk. It is not decoration. Many programs, when they detect their output is a pipe rather than a terminal, hold output in a buffer instead of sending it line by line. Your read then waits forever for a line that is stuck in the buffer. That is the number one reason coprocesses hang. The fix is to make the helper flush after each line (fflush() in awk, or stdbuf -oL in front of other tools, which forces line buffering). Coprocesses earn their keep when the helper is expensive to start, a policy engine you query per resource, a signing or decryption agent you keep warm, so you pay the startup cost once instead of thousands of times.

The failed fetch left token empty, local swallowed the non-zero status, and set -e never fired. A script written this way will happily authenticate with nothing and keep running. Before you trust any script that gates a deploy or counts security findings, run it once under set -x (which prints each command as Bash executes it) and watch for the parentheses, pipes, and $(...) where your state forks off and dies. That one habit catches most of these bugs before they catch you.

Try this

Work through “Coprocesses: A Helper You Keep On The Line” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: set -e is not the safety net you think. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related