Quoting, word splitting and globbing

The expansion order behind most script bugs, and the habits that prevent them.

Earlier lessons asked you to put double quotes round every expansion, to write -- before file names, and to pass file names NUL-separated. This lesson explains why those habits work. You will see the steps Bash applies to a command line before the command starts, in the order the Bash manual defines, and watch each step change the arguments a command receives. Then you will pass a script's arguments on without damaging them, stop a file named -rf from turning into an option, move file names between commands safely, and split a string on purpose when splitting is what you want. The examples run in ~/bash-ops/quoting.

What happens to a line before the command runs

A command never sees the line you typed. Bash first turns the line into a list of words, and the program receives that list as its arguments. To see the list, use a tiny script that prints how many arguments it got and each one in brackets:

#!/usr/bin/env bash
# args.sh: print how many arguments arrived, then each one in brackets.
printf '%s argument(s)\n' "$#"
for arg in "$@"; do
  printf '[%s]\n' "$arg"
done

Two arguments. The double quotes kept q3 report.log together as one word, and the quote characters themselves were removed before the script started: args.sh never saw them. Create two files for the next steps (ls shows the spaced name in single quotes so you can see where it starts and ends; the quotes are not part of the name):

Between reading the line and starting the command, Bash applies its expansions, always in the same order:

Brace expansion comes first and works on text alone, before any variable has been read. That is why a range cannot take its end from a variable:

app-{web,db}.conf became two words. {1..$n} did not become 1 2 3: when brace expansion ran, $n was still the two characters $n, so there was no range to expand. Step 2 replaced $n with 3 afterwards, too late. Brace ranges only work with values written out (numbers, or single letters as in {a..e}). Tilde expansion, in step 2, happens only when ~ is unquoted at the start of a word:

In quotes, ~/notes stayed literal. Write "$HOME/notes" when the path needs quotes. Word splitting, step 3, cuts only the unquoted results of step 2, never text you typed:

That is the whole reason for every "$var" in this course. Splitting has a second effect that is easy to miss: an unquoted expansion whose value is empty produces no word at all.

Unquoted, $empty vanished and end became the first argument, so a command that expects a file name first reads the wrong thing as its file. This is how [ -n $path ] answered "yes" for an empty path in "Exit status and tests". Quoted, the empty value stays an argument, an empty one. Pathname expansion, step 4, runs on the words that splitting produced, so a pattern that arrives inside a value is expanded as if you had typed it:

The names that pathname expansion produces are not split again, so q3 report.log arrived whole even here. Quoted, the pattern was passed on as text, which is what find -name '*.log' needs. The results of an expansion are also never read again as shell syntax. Quote characters inside a value are ordinary characters, and so is a ;:

Quotes stored inside a variable protect nothing; only the quotes you type around the expansion do. And the ; in line did not end a command. Splitting can only add arguments, never run a second command. Extra arguments are harmful enough (the -rf file below is exactly that), but running text as a command needs something that parses it again, such as eval or bash -c, which is why neither may ever receive text from outside the script.

A few places are not argument lists and are not split: the right-hand side of an assignment, the word after case, and everything inside [[ ]].

copy=$name kept all three spaces. Quotes in those places do no harm, though, and one habit ("quote every expansion") is easier to keep than a list of exceptions.

"$@" and $*: passing arguments on

Inside a script, $@ and $* both stand for all of its arguments. They differ when quoted, and only one form passes arguments on unchanged. set -- sets the arguments at the prompt the way a script receives them; the second one here is a lone *:

"$@" produced the two original arguments, exactly. Unquoted $* (unquoted $@ behaves the same) went through splitting and pathname expansion: q3 report.log fell apart and * became every name in the directory. "$*" joined all the arguments into one word, separated by spaces (strictly, by the first character of IFS). Use "$@" whenever you pass arguments on to another command, and "$*" only to put them into a message, as in printf 'called with: %s\n' "$*".

A file name that looks like an option

The Linux essentials lesson "Quoting, wildcards, history and aliases" showed this problem with a file named -l: once the glob has expanded, ls * reads the name as an option, and -- or ./* fixes it. Quoting cannot help, because the name already arrives as exactly one argument; the trouble is that it starts with -. In a script that deletes files, the same mistake destroys data. To create such a file for the demo, put a path in front of the name (drop/-rf), because touch -rf would read it as options too. Here drop stands for a directory other people can write to, such as an upload area; drop2 and drop3 are copies for trying the fixes:

Someone has left a file named -rf there, and a routine cleanup runs rm *. Do experiments like this only in a scratch directory. The parentheses run the commands in a subshell, a copy of your shell, so the cd inside ends with it and your shell stays in ~/bash-ops/quoting for the next step. cd drop || exit 1 leaves the subshell if the directory is missing, so rm * can never run in the wrong place:

The printf line shows the words rm received: -rf archive report.pdf. rm took -rf as its recursive and force options, deleted the archive directory with everything in it and report.pdf, and exited 0. The only survivor is -rf itself, which rm never treated as a file. Its position is not what makes this work: GNU rm, like most GNU tools, accepts options anywhere among its arguments. There are two fixes. The first is --, which marks the end of the options: every argument after it is an operand (a file name here), whatever it looks like. This is a POSIX convention and nearly every command follows it:

This time rm removed -rf and report.pdf as files and refused archive, because deleting a directory needs -r and nobody asked for it. That error and status 1 are the correct outcome. The other fix from the Linux lesson, rm ./*, gives the same result in drop3, because every word then starts with ./. That form also protects commands for which -- is not enough: find reads any argument that starts with - as the start of its search expression, even after --, so a starting directory named -old must be given as ./-old (the lesson "Command-line options" uses this). Put -- before the file operands of every command whose paths come from a variable or a glob, as in rm -- "$file" or cp -- "$src" "$dst". A path that starts with a directory, such as "$dir"/* with dir=/srv/uploads, is safe for the same reason as ./*.

Lists of file names: xargs and NUL separators

When file names travel from one command to another as text, the separator decides whether they arrive intact. "Loops and reading input safely" read find -print0 output with read -d ''. The other common consumer is xargs, which reads items from standard input and runs a command with them as arguments. By default it splits its input at blanks and newlines, and it treats quote characters as quoting. Three names test that:

names/a b.log became names/a and b.log, neither of which exists, and when xargs reached the apostrophe in it's.log it stopped reading with unmatched single quote. Only c.log was counted, and status 123 is how GNU xargs reports that a command it ran failed. find -print0 and xargs -0 agree on NUL as the separator, the one byte a file name cannot contain:

When you do not need a pipe at all, find can run the command itself. With -exec COMMAND {} + it puts as many names as fit where {} is, with no text in between to parse:

The order is the order in which find met the names in the directory. Every name begins with names/, the starting point you gave find, so none can start with a dash; with find . they begin with ./, which is the ./* fix again.

When you want splitting, ask for it

Sometimes a string really is a list: PATH holds directories separated by colons, and a setting may hold several host names separated by spaces. Split it on purpose with read -a, which stores the words of one line in an array (Bash-only; the next lesson covers arrays), and set IFS for that one command to choose the separator, as "Variables, quotes and command output" showed for NAME=value command:

Each directory of PATH became one element. read splits at the IFS characters (and trims them from the ends of the line), but it never expands * or any other pattern, and that is the difference from an unquoted variable:

The unquoted $hosts split correctly and then its * expanded into every name in the directory, so the loop ran nine times for three hosts. read -ra produced exactly the three words, * included. When you want a string split, use read -ra for words and mapfile for lines, and keep file names NUL-separated. Leaving a variable unquoted gives you splitting and pathname expansion together, and you almost never want both.

Try this

In a new directory, make a directory spool holding a.tmp, b c.tmp, -v.tmp and keep.txt (create the dash name as spool/-v.tmp). Write purge-tmp.sh, which deletes the .tmp files in the directory named by its argument, first the obvious way: cd "$1" || exit 2, then rm *.tmp. Make it executable and check:

1. ./purge-tmp.sh spool then echo "status: $?": expect rm: invalid option -- '.', GNU rm's hint Try 'rm ./-v.tmp' to remove the file '-v.tmp'., status 1 and all four files still there. Work out from the glob which argument rm read as options.

2. Fix it with rm -- *.tmp (and cd -- "$1" for a directory name that starts with a dash). Expect status 0 and only keep.txt left. ShellCheck, from the lesson "Check your script", reports the first version as SC2035 and suggests the same two fixes.

3. Run it once more: expect rm: cannot remove '*.tmp': No such file or directory and status 1. That is the unmatched glob from "Loops and reading input safely"; decide whether an empty directory should be an error for this job, and guard the rm if not.

Takeaway

Quote every expansion so a value stays one argument, and put -- or a ./ prefix before file operands so no argument can become an option. When you want a string split, split it on purpose with read -ra or NUL separators, never by leaving a variable unquoted.

Related