Loops and reading input safely

for over lists and globs, while read loops that keep every line, find -print0.

Most operations work is the same step applied to many things: every host in a list, every log in a directory, every line of a file. This lesson covers the loops that do that and the input problems that break them. You will loop over lists, arguments and globs, handle a glob that matches nothing, write a bounded while loop, read a file line by line without losing its last line, walk file names from find safely even when a name contains a newline, and load a file into an array with mapfile. The examples run in ~/bash-ops/loops.

for: once per item

for NAME in WORDS; do ...; done runs the body once for each word, with the variable set to that word. The words can be written out:

More often the list comes from a glob. To see how loops cope with real file names, make a directory whose names are awkward on purpose: one with a space, one containing a newline, one literally named *, and an empty log:

$'new\nline.log' is ANSI-C quoting, which Bash has and Dash does not: inside $'...', \n stands for a newline. A Linux file name may contain any byte except / and the NUL byte, newlines included, and tools such as ls put quotes round the odd ones so you can see where a name starts and ends. Now loop over the logs:

The shell replaced inbox/*.log with the matching names, sorted, before the loop started, and each name became exactly one item: the space and the newline did not split anything. That holds as long as the body uses "$f" in double quotes. Without the quotes, each value is split and expanded again when it is used:

The file named * expanded into every name in the directory, q3 report.log became two words, and the newline name became two as well. Run rm $f in that loop and the file named * would take every other file with it. The lesson "Quoting, word splitting and globbing" explains exactly which expansions happen; for loops, the rule is short: always write "$f".

A script loops over its own arguments with for arg in "$@". "$@" expands to every argument, each exactly as it was given. At the prompt, set -- fills $1, $2 and so on the way a script receives them:

When a glob matches nothing

A glob that matches no file is not an error in Bash. It is left in place as literal text, and the loop runs once with the pattern itself as the value:

The loop tried to process a file called inbox/*.csv, and wc failed on it. In a cleanup or upload job that is at best a confusing error and at worst an action on a path nobody meant. Bash has an option for this, shopt -s nullglob (Bash-only), which makes an unmatched glob expand to nothing, so the loop runs zero times. shopt -u nullglob turns it off again; the demos end with it so that your shell is back to normal for the next steps:

nullglob has a side effect, though. It applies to every glob in the shell, not just loops, and a command that receives no arguments often falls back to a default:

ls with no arguments lists the current directory, so "list the CSV files" listed everything in it (in the lab only inbox existed at this point; your listing shows whatever your directory holds). If you turn nullglob on in a script, keep it for the loops that need it. The alternative that works everywhere is a guard as the first line of the body, skipping a value that does not exist:

continue skips the rest of this pass and moves to the next item. With nothing else to do, the loop ended.

while: repeat until something changes

while COMMAND; do ...; done runs the body for as long as the command succeeds. A common use is waiting for something, such as a file that another job creates. Here (sleep 1.5; touch ready.flag) & stands in for that other job: the parentheses group two commands, and & runs the group in the background, so the flag appears after one and a half seconds while the loop is already checking:

(( attempt <= 5 )) limits the loop to five checks, break leaves the loop as soon as the file exists, and sleep 1 spaces the checks out. The flag appeared between the second and third check. Give every waiting loop an upper bound. A loop that waits forever for a file that never comes looks exactly like a job that is still working.

Read a file line by line

The standard way to process a file one line at a time is while IFS= read -r line; do ...; done < file. The redirection after done feeds the file to the whole loop, and each read takes the next line. -r keeps backslashes, as the lesson "Input, output and redirection" showed, and IFS= (empty, set for this one command) stops read from trimming spaces at the start and end of the line. Here is a host list. Like many files written by programs, it has no newline after its last line. An editor would add one when it saves, so create the file with printf, which writes exactly the bytes in its format, and check the end of the file with od -c, which prints every byte:

tail -c 5 took the last five bytes: a newline (\n) and then d b 0 1, with no newline after it. (If you cat the file, your next prompt starts right after db01 for the same reason.) Now read it with the standard loop:

db01 is missing, and wc -l agrees with the loop, because it counts newline characters and there are only five. read did put db01 into line, but it reached the end of the file before finding a newline, so it returned 1, and while stopped before running the body. The fix is to also accept a line that was read without a newline:

|| [[ -n $line ]] means "or line is not empty", which is true exactly once, for that last partial line. Here it is in a script that also skips blank lines and comments:

#!/usr/bin/env bash
# Usage: list-hosts.sh FILE
# Prints the host names in FILE. Blank lines and lines starting with # are
# skipped; a last line without a final newline still counts.
if [[ $# -ne 1 || ! -r $1 ]]; then
  printf 'usage: list-hosts.sh FILE (a readable file)\n' >&2
  exit 2
fi
count=0
while IFS= read -r line || [[ -n $line ]]; do
  if [[ -z $line || $line == '#'* ]]; then
    continue
  fi
  count=$((count + 1))
  printf 'host %d: %s\n' "$count" "$line"
done < "$1"
printf '%d hosts\n' "$count"

'#'* is a pattern for "starts with #"; the # is quoted so that it cannot be read as the start of a comment. continue skips those lines, and count counts the rest with count=$((count + 1)). All three hosts are there, including db01. You will see for line in $(cat file) in old scripts: it splits lines at every space and expands a line such as * into file names, so use while IFS= read -r instead.

File names from find: NUL separators

find searches a directory tree and prints one path per line. Lines are the wrong separator for file names, because a name can contain a newline:

inbox/new and line.log are not two files. They are one name cut in half, and a loop that acted on them would act on two paths that do not exist. The one byte a path can never contain is NUL, so tools can separate names with it. find -print0 ends each name with a NUL byte, sort -z sorts NUL-separated items, and read -d '' reads up to the next NUL instead of the next newline:

Each name arrived whole. (The sort -z only makes the order the same on every run; find itself returns names in whatever order the directory stores them.) One problem remains, and it has nothing to do with names:

After the pipe, count was still 0. Each command in a pipeline runs in its own subshell, a copy of the shell, so the loop counted in a copy that disappeared when the pipeline ended. The second form, done < <(find ...), keeps the loop in the current shell. <( ) is process substitution (Bash-only; Dash reports Syntax error: redirection unexpected): Bash runs the command and gives its output a file name, and < reads from it like any file.

This script puts the pieces together: a guard, NUL-separated names, and a counter that survives the loop:

#!/usr/bin/env bash
# Usage: scan-logs.sh DIR
# Prints the line count of every .log file under DIR, then how many it found.
# File names travel NUL-separated, so spaces, * and even newlines are safe.
if [[ $# -ne 1 || ! -d $1 ]]; then
  printf 'usage: scan-logs.sh DIR (an existing directory)\n' >&2
  exit 2
fi
found=0
while IFS= read -r -d '' file; do
  lines=$(wc -l < "$file")
  printf '%5d  %s\n' "$lines" "$file"
  found=$((found + 1))
done < <(find "$1" -type f -name '*.log' -print0 | sort -z)
printf '%d log files\n' "$found"

Every .log file is listed with its line count, the newline name included (it prints on two lines because its name really contains a newline), and the total of 4 is right. A directory that does not exist is refused with status 2.

mapfile: a whole file into an array

Sometimes you need all the lines at once, to count them first or to use them twice. mapfile -t NAME < file (Bash-only) reads every line into an array, a variable that holds a numbered list of values; the lesson "Arrays and parameter expansion" covers arrays properly. -t removes the newline from each line:

${#hosts[@]} is the number of elements, ${hosts[-1]} the last one, and "${hosts[@]}" expands to every element as a separate word, the array version of "$@". mapfile kept db01 even without a final newline. It also kept the comments and the blank line, so filter those out when you use the list. With -d '', mapfile splits on NUL bytes instead, which loads find -print0 output safely:

Use while IFS= read -r to stream through input once, especially large input, because it holds one line at a time. Use mapfile when the list is small and you need it more than once.

Try this

Copy scan-logs.sh and the inbox directory into a new directory. Change the script so that empty log files are skipped with continue and counted separately: test [[ ! -s $file ]], add one to an empty counter, and continue before wc runs. The last line becomes N log files, M empty skipped. Run ./scan-logs.sh inbox and expect:

1. Three files listed: app.log (3), the newline name (1, printed across two lines) and q3 report.log (2). empty.log is not listed.

2. The last line 3 log files, 1 empty skipped and status 0. If both counts come out as 0, check that the loop is still fed with < <( ) and not through a pipe.

Takeaway

Quote the loop variable, guard globs that may match nothing, and bound every waiting loop. Read lines with while IFS= read -r line || [[ -n $line ]], and read file names NUL-separated with find -print0 and read -d '', fed through < <( ) when the loop's variables must survive.

Related