Pipes, redirection and exit status

stdin, stdout, stderr, pipes and exit codes.

Every command reads input and writes output, and the shell decides where both go before the command starts. This lesson shows how to send a command's results into a file, keep its error messages apart from its results, connect commands with pipes, write to a file that only root may change, and let the exit status of one command decide whether the next one runs. Every later lesson, and every script you write, is built from these pieces, and most of their surprises come from the few rules below.

Standard output and standard error

A program starts with three files already open, each with a number called a file descriptor (the handle the kernel hands back for an open file, as the first lesson showed). Descriptor 0 is standard input, or stdin, where the program reads. Descriptor 1 is standard output, stdout, where it writes its results. Descriptor 2 is standard error, stderr, where it writes error messages and warnings. In a terminal all three are connected to the terminal, so results and errors appear mixed together. The examples write their files into a practice directory, so create it first; then give ls one file that exists and one that does not:

The two lines look alike, but they travelled on different descriptors: the complaint about /etc/nope on stderr and the listing on stdout. ls reports problems as it checks the names and prints the listing afterwards, which is why the error comes first. It exits with status 2 because one of the names could not be accessed.

Redirection changes where a descriptor points. > FILE sends stdout into a file:

Only the error reached the screen; the listing is in the file. > is short for 1>, and 2> does the same for stderr:

/dev/null is a device file that discards everything written to it. Sending stderr there hides messages you expect and do not care about, such as the "Permission denied" lines find prints for directories you may not read (the next lesson uses it for exactly that).

The exit status is still 2: throwing the message away does not change the result. Hide errors only after you have read them once and know they are noise.

Writing to files: >, >> and <

> creates the file if it does not exist and empties it if it does. >> adds to the end instead:

The second > wiped out "first". That is what you want for a report you regenerate, and a disaster for a log you meant to keep. The shell sets up every redirection before it starts the command, so > empties the file even before the command has read a byte. Sorting a file into itself shows the consequence:

The shell opened hosts.txt for writing and cut it to zero bytes, then sort read the now empty file and wrote nothing. The three names are gone, and grep ... file > file or sed ... file > file lose data the same way. Write to a new file and rename it afterwards, or use an option that does this for you, such as sort -o:

If you want the shell to protect existing files, set -o noclobber makes > refuse to overwrite them (>| overrides it for one command). The option lasts until the shell exits or you switch it off with set +o noclobber, as the last line here does so that the rest of the lesson works as shown; put it in ~/.bashrc, covered in the lesson on quoting and aliases, to keep it.

The -bash: line 2: prefix appears because the terminals in this course were recorded by running each command from a script. At an interactive prompt the same error starts with -bash:.

Redirection works in the other direction too. < FILE connects a file to a command's stdin. Many commands accept a file name or stdin, and the difference shows in their output:

Given a name, wc -l opens the file itself and prints the name next to the count. Reading stdin, it never learns the name, so it prints the bare number, which is handier in a script.

Both streams in one file: the order matters

To capture everything a command prints, say for a ticket, send stdout to the file and then point stderr at the same place. 2>&1 means "make descriptor 2 point wherever descriptor 1 points right now", and the shell applies redirections from left to right.

Nothing reached the screen and the file holds both lines. Swap the two redirections and the result changes:

Read it left to right. 2>&1 came first, when descriptor 1 still pointed at the terminal, so stderr was pointed at the terminal. Only then did > move stdout to the file. Bash also accepts &> FILE as a shorter spelling of > FILE 2>&1, and >> FILE 2>&1 appends both.

Pipes, and tee

A pipe, written |, connects the stdout of the command on its left to the stdin of the command on its right. Both commands run at the same time and no temporary file is involved. How many programs are there in /usr/bin?

ls prints one name per line when its output is not a terminal, so wc -l counts programs. A pipe carries stdout only, and errors still go to the screen:

In the first command the error bypassed the pipe and wc counted one line. Redirecting stderr into stdout before the pipe sends both into it, and bash accepts |& as a short form of 2>&1 |. The next lesson combines this with find, and the text-processing lesson builds longer pipelines.

tee is a T-junction for a pipeline: it copies its stdin into a file and passes it on to its stdout. Use it when you want to keep what went through a pipe and still see or process it.

sudo and redirection

Now save the SSH server's effective configuration into root's home directory, /root, which only root may enter. Putting sudo in front looks like it should work:

The message comes from bash, not from sshd or sudo. Your shell runs as deploy, and it opens the file for the redirection before it starts sudo; deploy may not create files in /root, so the command never runs. sudo raises the privileges of the command it starts, never of your shell's redirections. sudo echo "192.0.2.10 db01" >> /etc/hosts fails for the same reason. The fix is to let a program that runs as root open the file, and tee is the usual choice:

sudo tee writes the file as root, and > /dev/null throws away the copy tee would print on your screen. sudo tee -a appends, like >>. Another way is to give the whole command line to a shell that runs as root: sudo sh -c 'sshd -T > /root/sshd-effective.txt'.

Exit status and chaining commands

The first lesson introduced the exit status: 0 means success, anything else failure, and $? holds the status of the last command. Some commands answer a question through it. grep -q WORD FILE looks for lines containing WORD (the next lesson covers grep), prints nothing, and exits 0 when it found a match, 1 when it found none and 2 when it could not read the file:

Three operators join commands on one line and decide from that status what runs next. ; runs the next command whatever happened. && runs it only if the previous command succeeded, and || only if it failed:

The difference matters most after cd. When cd fails, a ; lets the next command run in whatever directory you were already in:

pwd is harmless, but put a delete in its place and it runs in your home directory. Chain steps that depend on each other with &&. In a script, write cd DIR || exit 1, or start the script with set -e, which stops it at the first command that fails (bash(1) lists the exceptions, such as commands tested with && or ||). One more trap: a && b || c is not "if a then b else c", because c also runs when b fails.

A pipeline has one exit status, and it is the status of its last command:

cat failed with 1, but wc counted zero lines successfully, so $? is 0 and a check on it would report success. Bash keeps each command's status in PIPESTATUS, an array (a variable that holds a list; ${PIPESTATUS[@]} prints all of it), which is overwritten by the next command, so read it immediately. set -o pipefail makes a pipeline fail when any command in it fails, and scripts usually set it together with set -e:

Try this

Run cat /etc/hostname /etc/shadow > ~/pipes/report.txt 2>&1 (in the ~/pipes directory from the start of the lesson). Before you open the file, predict how many lines it holds and which one is the error, then check with cat: the host name, then cat: /etc/shadow: Permission denied. Next run cat /etc/hostname /etc/shadow 2>/dev/null | wc -l, then echo ${PIPESTATUS[@]}, and explain both numbers (1, then 1 0). Finally append a line to a file only root can write with echo "checked" | sudo tee -a /root/pipes-check.txt, confirm it with sudo cat /root/pipes-check.txt, and remove it with sudo rm.

Takeaway

Your shell sets up redirections as you, before the command starts: use > file 2>&1 in that order to capture everything, | sudo tee to write where only root may, and && between steps that depend on each other. Judge a command by its exit status, including every command of a pipeline, not by what reached the screen.

Related