Text processing without forks: expansion, awk & sed

Parameter expansion, awk and sed that replace pipelines of cat/grep/cut.

Every time your script runs an external command like cut, grep, or sed, the shell does two quiet things under the hood. It calls fork (the system call that makes a near-identical copy of the current process) and then exec (the call that throws away that copy's program and loads a new one in its place). It is like hiring a temp worker for a single envelope: you fill out the paperwork, brief them, they lick one stamp, and you let them go. One envelope is fine. Put that inside a loop over a two-million-line log file and you have made two million hires. The paperwork, not the stamping, becomes your whole afternoon.

Here is the part most people miss. A large share of everyday text work never has to leave the shell at all, and the rest is usually one awk program standing in for a pipeline of five commands. Fewer processes means faster scripts, fewer moving parts that can fail in the middle of an incident, and a smaller set of external binaries that an attacker could have swapped out on a box you are trying to trust. For a defender pulling fields out of logs at three in the morning, that speed and that predictability are the whole point.

Why a Fork Is Expensive

You do not have to take the cost on faith. Measure it. Here are fifty thousand file paths, and two ways to strip each one down to its filename.

echo and read are builtins: the shell runs them itself, in the same process, with no hire. basename is a separate binary, so Version A forks fifty thousand times. Look at where the time went. Most of it is sys (time spent inside the kernel, the core of the operating system that manages processes and memory), and that is the cost of all the forking, not of the text work. Version B does the exact same job with parameter expansion and never leaves the shell. Same output, roughly eighty times faster.

Parameter Expansion Replaces Most of Coreutils

Instead of sending every small edit out to a hired hand, you do it at your own desk. Parameter expansion is a little language built into the ${...} shape. It trims off prefixes and suffixes, substitutes text, slices out substrings, changes case, and supplies defaults, all in-process. The trimming operators are the ones worth burning into memory. One hash (#) trims the shortest match off the front, two hashes (##) trim the longest. One percent (%) trims the shortest match off the back, two percents (%%) trim the longest.

That single block already ate basename, dirname, a couple of sed substitutions, tr for the lowercasing, and cut for the slice. Those are all coreutils (the GNU Core Utilities, the standard kit of small text commands you reach for by reflex), and not one of them had to run. Now the real work a defender actually does: pulling fields out of a log line. The same operators carve up a structured message without a single fork, and two more forms round it out. Two slashes (${var//old/new}) replace every match, not only the first, and ${#var} gives you the length.

Two habits keep this safe when the text comes from somewhere you do not control, like a log written by whoever is knocking on your SSH (Secure Shell, the encrypted remote-login service) port. Read with read -r so a backslash in the data stays a literal backslash instead of being treated as an escape. And put IFS= (the Internal Field Separator, the set of characters the shell uses to break a line into words) in front of the read, set to empty, so leading and trailing spaces are preserved exactly as they arrived: while IFS= read -r line; do ...; done.

One awk Pass Beats a Five-Stage Pipeline

awk (named after the three people who wrote it: Aho, Weinberger, and Kernighan) is a worker standing at a conveyor belt. Each row arrives already cut into columns, the worker runs your instructions on that row, and keeps a tally sheet that survives from one row to the next. Precisely: awk reads its input one record at a time (a record is a line by default), splits each record into fields on whitespace (field one is $1, field two is $2, and so on), runs your code for every record, and keeps variables and arrays alive across all of them. That memory across records is the trick. It lets a single awk filter, pick columns, sum, and group in one read of the file, where a pipeline would need grep, then another grep, then cut, then sort, then uniq.

That first IP (Internet Protocol address) 203.0.113.44 stands out: it made more than twice the requests of anyone else. It is the same address that showed up in the auth.log line earlier. This is what a defender is looking for, one loud source in a sea of quiet ones, and awk found it in a single read of the file. The -F flag sets the input field separator, which is how you carve up a colon-delimited file like the account database.

The nobody exclusion matters and is easy to forget: on Debian and Ubuntu, nobody has UID 65534, which sails past a >= 1000 test and pollutes your list of real logins. To flip columns around and re-join them with your own separator, set OFS (the output field separator) in the BEGIN block, which runs once before any record is read.

The tally sheet is where awk pulls ahead of any pipeline. Here it filters to failed logins, extracts the source address, counts per address, and applies a threshold, all in one pass over the file. There is no grep, no cut, no sort, no uniq. That is five processes collapsed into one, and the whole log is read exactly once.

sed for Surgical Stream Edits

sed (short for stream editor) is the copy editor with a red pen, moving down a stream of text and making the same kind of correction over and over. It earns its keep for in-stream substitution and line-addressed edits. A few idioms cover most real jobs. First, pick a delimiter other than slash when your pattern contains file paths, so you are not drowning in backslashes.

Second, edit a file in place, but keep a backup while you learn to trust the pattern. With GNU sed (the version on every systemd Linux) the suffix attaches straight to the flag.

Third, an address range scopes an edit to a region of the file. With -n (which switches off sed's habit of printing every line) plus a range that runs from one marker to the next, sed prints only the lines between the two markers, inclusive. One catch: a fullchain file stacks several certificates end to end, and the range happily fires again on each new BEGIN, so on its own it would spill every block. A q (quit) on the closing marker stops sed after the first one. That is how you carve a single certificate out of a bundle.

Addresses can be line numbers too, and they refer to the input line count, not the count after earlier deletions. This trips people up: delete the comment on line one, and the surviving lines still carry their original numbers.

The moment you need arithmetic, fields, or grouping, stop reaching for sed. That is awk's job, and forcing it into sed gives you write-only line noise that the next person (probably you, in six months) cannot read.

When you replace a pipeline with an expansion or one awk, prove the swap did not quietly change the answer. Run the old version and the new one against the same file and diff the two: diff <(old_pipeline access.log) <(new_awk access.log). The <(...) shape (process substitution) hands each command's output to diff as if it were a file, so you compare them without writing anything to disk. If the diff prints nothing, the output is identical and you can ship it. If it prints anything at all, your refactor changed behavior, and you want to know that on your terminal, not from an alert an hour later.

Try this

Work through “sed for Surgical Stream Edits” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: quote every expansion, especially attacker-controlled text. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related