>_SecOpsLog_
CoursesBash for ops, done safelyBash fundamentals

Loops: for, while & reading files

Repeat work and read input line by line.

Beginner18 min · lesson 6 of 14

Doing the same job on a pile of things

You have forty servers to check, two hundred log files to scan, or a list of user accounts to audit. Typing the same command forty times is slow and easy to fumble. A loop is the fix. It is the assembly-line worker of Bash (the command language you type into a Linux terminal): you describe the steps once, hand it a pile of items, and it runs those steps on each item in turn. This lesson covers the loops you will actually reach for in operations work, plus the one file-reading pattern that quietly breaks security scripts when people get it wrong.

for: one run per item in a list

A `for` loop is you reading names off a guest list. For each name you do the same thing: check them in. You write the list of items after the word `in`, and Bash runs the body once with the loop variable set to each item.

~/secopslog — bash
$ for env in dev staging prod; do echo "Deploying to $env" done
Deploying to dev Deploying to staging Deploying to prod

Bash split that list on spaces and ran the body three times. `$env` held `dev`, then `staging`, then `prod`. The words after `in` are the whole list, and they can be anything: hostnames, ports, filenames, usernames.

for over a glob: let the shell find the files

Most of the time you do not want to type the filenames by hand. You want to hand the shell a rule instead of a list: "every file whose name ends in `.log`". That shorthand is a glob (short for global, the old name for filename wildcards). The `*` matches any run of characters, so `*.log` means every name ending in `.log`. The shell expands the glob into the real filenames before the loop starts.

~/secopslog — bash
$ for logfile in *.log; do lines=$(wc -l < "$logfile") echo "$logfile has $lines lines" done
app.log has 3 lines auth.log has 2 lines nginx.log has 1 lines

The shell replaced `*.log` with the three matching files (in alphabetical order) and the loop ran once per file. Notice the double quotes around `"$logfile"`. A filename can contain spaces, and the quotes keep such a name as one item instead of two. Leave them off and a file called `access log.txt` would split and break your script. One sharp edge: if no file matches, plain Bash leaves the text `*.log` unexpanded and the loop runs once with that literal string. Running `shopt -s nullglob` first makes a no-match glob expand to nothing, so the loop runs zero times, which is usually what you want.

Counting and waiting: C-style for and while

Two loops cover the cases where you are not walking a list. When you want to run something a fixed number of times, use the C-style `for`, named after the C programming language it copies. It works like a lap counter at a track: it starts at a number, keeps going while a condition holds, and ticks the number up each time around. Inside the double parentheses those three parts sit in order: where the counter starts, how long to keep going, and how the counter changes each pass.

~/secopslog — bash
$ for (( i=0; i<3; i++ )); do echo "attempt $i" done
attempt 0 attempt 1 attempt 2

When you do not know the count ahead of time and instead want to keep going until some condition flips, use `while`. It works like a bucket under a tap: keep filling while the bucket is not full. Bash checks the condition, runs the body, then checks again, stopping the moment the condition is false.

~/secopslog — bash
$ count=3 while (( count > 0 )); do echo "$count..." (( count-- )) done echo "liftoff"
3... 2... 1... liftoff

`(( count-- ))` subtracts one from `count` each pass. Forget that line and the condition never turns false, so you get an infinite loop. Press Ctrl-C to kill one if it happens.

Reading a file line by line, the right way

This is the pattern people get wrong most often, and the wrong version fails in ways that matter for security. Say you have an allowlist of addresses, where the exact content on each line matters:

allowlist.txt
10.0.0.5
  10.0.0.6
C:\logs\path
*

Here is the correct way to read it. Memorize this line as a single unit:

~/secopslog — bash
$ while IFS= read -r line; do echo "[$line]" done < allowlist.txt
[10.0.0.5] [ 10.0.0.6] [C:\logs\path] [*]

Every line arrived exactly as written: the leading spaces on the second address, the backslashes in the path, and the lone `*`. Read the machinery from right to left. `< allowlist.txt` feeds the file into the loop as its input. `read` pulls one line into the variable `line` and returns success until the file runs out, which is what stops the loop at the end. Two small pieces carry the weight: `IFS=` and `-r`.

`IFS` is the Internal Field Separator, the set of characters Bash treats as gaps between words. Normally `read` uses it to trim leading and trailing spaces and tabs off each line. Setting `IFS=` (empty) for this one command turns that trimming off, so ` 10.0.0.6` keeps its indentation. `-r` means raw. Without it, `read` treats a backslash as an escape character and swallows it. The difference is not cosmetic:

~/secopslog — bash
$ # paths.txt contains the literal line: C:\logs\audit while IFS= read line; do echo "[$line]"; done < paths.txt # no -r while IFS= read -r line; do echo "[$line]"; done < paths.txt # with -r
[C:logsaudit] [C:\logs\audit]

Without `-r`, `C:\logs\audit` came back as `C:logsaudit`. The backslashes vanished. On Windows paths, Active Directory names, or regular expression (regex) patterns stored in a rules file, that silent damage changes what your script matches. Keep `-r` on every read loop.

Now the wrong way, the one you will find in old blog posts and copy-pasted scripts:

~/secopslog — bash
$ for line in $(cat allowlist.txt); do echo "[$line]" done
[10.0.0.5] [10.0.0.6] [C:\logs\path] [allowlist.txt]
for line in $(cat file) is broken and unsafe
Two bugs are visible in that output. First, `$(cat file)` gets split on every space and newline, not only on newlines, so ` 10.0.0.6` lost its leading spaces, and any line containing a space would shatter into separate items. Second, the shell runs filename expansion on the result, so the lone `*` on the last line turned into the filename `allowlist.txt`. Feed such a loop a file an attacker can write to, drop a `*` on a line, and you are suddenly looping over your directory contents instead of the data you meant to process. Always read files with `while IFS= read -r line; do ... done < file`, never with `for` over `cat`.

break and continue: skip one, or stop early

Inside any loop, two commands change the flow. `continue` says "skip the rest of this pass and move to the next item", like passing on a name on your checklist. `break` says "stop the whole loop now and walk away". A common shape reads a host list, skips comment lines, and halts at a marker:

hosts.txt
web-01
# web-02 is decommissioned
web-03
STOP
web-04
~/secopslog — bash
$ while IFS= read -r host; do [[ "$host" == \#* ]] && continue # skip comment lines [[ "$host" == STOP ]] && break # stop at the marker echo "Scanning $host" done < hosts.txt
Scanning web-01 Scanning web-03

The comment line was skipped by `continue`, `web-03` was scanned, and `STOP` ended the loop with `break` before `web-04` was ever read. `[[ "$host" == \#* ]]` is a pattern test: it is true when `host` starts with a `#`. The backslash before `#` keeps the shell from reading it as the start of a comment.

Putting it together: scan a folder of logs

Real work stacks these loops. Here is a script that walks every log file in a folder and, for each one, reads it line by line to count failed logins. The outer loop is a glob over files, and the inner loop is a `while read` over the lines of each file.

scan.sh
#!/usr/bin/env bash
for logfile in logs/*.log; do
  fails=0
  while IFS= read -r line; do
    [[ "$line" == FAIL* ]] && (( fails++ ))
  done < "$logfile"
  echo "$logfile: $fails failed login(s)"
done
~/secopslog — bash
$ bash scan.sh
logs/auth-web01.log: 2 failed login(s) logs/auth-web02.log: 1 failed login(s)

The `fails` counter is reset to zero at the top of each file, counts lines starting with `FAIL`, and gets printed once per file. Because the file is fed in with `< "$logfile"` rather than piped, the whole loop runs in your current shell, so the counter survives to the `echo` after it.

A missing final newline can drop your last line
`read` returns failure when it hits the end of the file without a trailing newline, even though it did place that last chunk into the variable. So a plain `while IFS= read -r line` loop silently skips the final line of any file that does not end in a newline (truncated logs and some exported data do this). Guard against it: `while IFS= read -r line || [[ -n "$line" ]]; do`. The extra test is true whenever `read` pulled in leftover text but reported end-of-file, so that last line still gets processed.
Which loop should you reach for?
Repeating work in Bash
pick by what you are looping over
A list you can name
for x in a b c
envs, ports, users
Files on disk
for f in *.log
glob expands to real names
A fixed count
for (( i=0; i<n; i++ ))
retries, N passes
Until a condition flips
while (( cond ))
wait, drain, count down
Lines of a file
while IFS= read -r line
never for line in $(cat)
Match the loop to the shape of your input.
Quick check
01You must process an allowlist file line by line, and some lines have leading spaces or backslashes that have to be preserved exactly. Which loop is correct?
Incorrect — This splits on all whitespace and then runs filename expansion, so leading spaces are lost and a line containing * can turn into filenames. Wrong and unsafe.
Incorrect — Closer, but without IFS= it trims leading and trailing whitespace, and without -r it eats backslashes, so your exact content is damaged.
Correct — IFS= stops the whitespace trimming and -r keeps backslashes literal, so each line arrives exactly as written.
Incorrect — The read flags are right, but piping cat into the loop runs the body in a subshell, so variables set inside are lost after the loop, and it spawns an extra process for nothing. Redirect with < instead.
02In plain Bash with no special options set, you run `for f in *.csv; do echo "$f"; done` in a directory that contains no files ending in .csv. What happens?
Incorrect — that is the behaviour only after `shopt -s nullglob`; plain Bash does not silently drop an unmatched pattern.
Incorrect — a glob that matches nothing is not an error in plain Bash; it is simply left as-is.
Correct — plain Bash leaves an unmatched glob unexpanded, so the loop runs once with the pattern itself, which is why `shopt -s nullglob` exists to make it expand to nothing instead.
Incorrect — Bash never falls back to matching all files; a non-matching glob stays as the literal pattern.
03A colleague's audit script counts lines with `while IFS= read -r line; do ((n++)); done < export.csv`. It reports one fewer line than the file contains, but only for files from one tool that omits the trailing newline. What is going on, and what fixes it?
Correct — `read` reports end-of-file on an unterminated final line, so `while IFS= read -r line || [[ -n "$line" ]]` is needed to catch that leftover text.
Incorrect — `IFS=` only disables leading/trailing whitespace trimming; it never drops a whole line.
Incorrect — `-r` only stops backslash from being treated as an escape; it has nothing to do with the missing line.
Incorrect — there is no strict mode here, so `((n++))`'s exit status is ignored; the dropped line is an end-of-file behaviour of `read`.
Conditionals: if & caseFunctions & arguments

Related

Project layout and venvsPython for security automationBash internals: subshells, FDs & process substitutionAdvanced scripting for DevSecOpsBash scripting cheat sheetCheat sheetGitOps interview questionsInterview prepDevSecOps & supply-chain security interview questionsInterview prep