Input, output and redirection

printf over echo, results vs errors in scripts, here-documents and read -r.

This lesson is about where a script's input comes from and where its output goes. You will print data safely with printf, write a script that keeps results and error messages apart and reports what happened through its exit status, capture its output in a log line that also works under sh and cron, feed fixed text in with here-documents and here-strings, and read input without read mangling it. These are the details that decide whether a failed cron job leaves an error in its log or leaves nothing. The examples run in ~/bash-ops/io.

Three streams, and printf for output

The Linux essentials lesson "Pipes, redirection and exit status" covered what the shell does with a command's streams: standard input (0), standard output (1) and standard error (2), >, >>, 2>, <, /dev/null, 2>&1 and pipes. In short, > empties the file before the command starts and >> appends, and 2>/dev/null hides a message but not the failure, because the exit status still records it. This lesson is about the script's side: what it prints, on which stream, with which status.

For printing, prefer printf to echo whenever the text includes a value you did not type yourself. echo treats some arguments as its own options, so a value that happens to look like one changes what it does:

echo "$opt" printed nothing at all, not even an empty line: Bash's echo took -n as its "no newline" option. printf takes a format string first and the values after it. In the format, %s means "insert the next value as text" and \n is a newline, and the values themselves are never read as options. The last line shows the format being reused: %-8s pads the first value to 8 characters, and printf repeats the format until it has used all four values, which gives a small aligned table. echo stays fine for fixed messages you wrote yourself.

A script that keeps results and errors apart

The examples use a small status file with one host state pair per line. Save it in the lesson directory as servers.txt:

web01 ok
web02 ok
db01 degraded
cache01 ok

A well-behaved script writes its results to standard output, its complaints to standard error, and says with its exit status which case happened. Here is a small report on servers.txt:

#!/usr/bin/env bash
# report.sh: count the states in a status file with one "host state" pair per line.
# Exit status: 0 report printed, 1 file cannot be read, 2 wrong usage.
 
if [ "$#" -ne 1 ]; then
  printf 'usage: %s FILE\n' "$0" >&2
  exit 2
fi
file=$1
 
if [ ! -r "$file" ]; then
  printf 'report.sh: cannot read %s\n' "$file" >&2
  exit 1
fi
 
printf 'states in %s:\n' "$file"
awk '{ print $2 }' "$file" | sort | uniq -c

Arguments are the words typed after the script's name. Inside the script, $1 is the first, $# is how many there are, and $0 is the name the script was started with; the lesson "Command-line options" covers options properly. [ "$#" -ne 1 ] is a test ("the count is not equal to 1"), and if ... then ... fi runs its body when the test succeeds; both get their own lessons shortly. >&2 on a printf sends that message to standard error. exit 2 means "you called me wrongly" and exit 1 means "I could not do the job", the convention Python's argparse and many Unix tools follow. In every tool of this track 0 means success and 2 means the command could not run as asked; what 1 means is up to each tool, and its header says so. The last line is a pipeline, explained two sections down.

The good run printed the counts and exited 0. The two failures each exited with their own number and printed nothing on standard output: in the last step, report.out is 0 bytes and the message still reached the screen. That separation is what lets the next step in a pipeline or a scheduler trust standard output and act on the exit status.

One log for both streams, also under sh

A scheduled job usually needs one log file with everything in it. The portable way to write that is > file 2>&1, in that order. The Linux essentials lesson showed why: 2>&1 points standard error wherever standard output points at that moment, so in 2>&1 > file the errors stay on the terminal.

Bash also has a shorthand, &> file, which means the same as > file 2>&1. It is Bash-only, and under sh it means something else entirely:

In Bash, &> captured the error. Dash read the same line as two commands: ./report.sh ghost.txt &, which runs the script in the background with its output still on the terminal, and > all.log, a redirection with no command, which just creates an empty file. (The sleep 1 only gives the background job time to print before wc looks at the file.) This is not a lab curiosity: cron runs every crontab line with /bin/sh, so backup.sh &> backup.log in a crontab gives you an empty log forever. Write > file 2>&1 in crontab lines and in anything that might run under sh; inside a Bash script either form works.

Pipes, here-documents and here-strings

The pipe at the end of report.sh works as the Linux essentials lesson showed: only standard output goes through it, and the pipeline's status is that of its last command (the lesson "Strict mode, honestly" shows what pipefail changes in a script). awk '{ print $2 }' prints the second field of each line (the state), sort puts equal states next to each other, and uniq -c counts each run of equal lines.

A here-document feeds lines written in the script itself to a command's standard input. Everything up to a line containing only the marker word (EOF by convention) is the input. With a plain marker, variables in the text are expanded; with the marker in quotes, <<'EOF', the text is passed exactly as written:

app.conf got the real value of $app, which is what you want for a generated config file. start.tmpl kept $app literally, which is what you want when the text is itself a script or a template that another program will expand later. A here-string, <<< "text", is the one-line version: it feeds a single string (plus a newline) to standard input. It is Bash-only.

read took the line from the here-string and split it at the space into two variables, host and state.

Reading input with read -r

read reads one line from standard input into one or more variables. Without -r, it treats a backslash as an escape character and removes it, which silently changes Windows paths, regular expressions and anything else containing \:

Always use read -r. (read also trims spaces at the start and end of the line; the lesson "Loops and reading input safely" adds IFS= to keep them.) A script that asks a question uses -p to print a prompt:

#!/usr/bin/env bash
# ask.sh: ask for a directory, then show exactly what was read.
read -r -p "Directory to archive: " dir
printf 'You entered: [%s]\n' "$dir"

With input from a pipe, read got the whole line, space included, and no prompt appeared: -p prints its prompt (on standard error) only when input comes from a terminal. Started at a terminal, the prompt appears and read waits for a line. The two empty cases show the weakness of this version. When there is no input at all, read fails, but nothing checks it: the script carries on with an empty dir and exits 0. A backup script that did the same would archive the wrong directory, or none, and report success.

Try this

Make ask.sh fail closed. read returns a non-zero status when it reaches the end of input without reading a line, so wrap it: if ! read -r -p "Directory to archive: " dir; then (! inverts the status, so the branch runs when read fails; "Decisions: if, case and guard clauses" covers it), print ask.sh: no input to standard error with >&2, exit 1, and fi. Then check each case:

1. ./ask.sh < /dev/null followed by echo "exit status: $?": expect ask.sh: no input and 1.

2. ./ask.sh < /dev/null 2>/dev/null and the same echo: expect no message, and still 1. The status reports the failure even when the message is thrown away.

3. printf '%s\n' '/srv/app data' | ./ask.sh > answer.txt, then cat answer.txt: expect status 0 and You entered: [/srv/app data] in the file, space included.

Takeaway

Send results to standard output, messages to standard error and the verdict to the exit status; capture both streams with > file 2>&1 in that order, and read input with read -r.

Related