Text processing without forks: expansion, awk & sed

Parameter expansion, awk and sed that replace pipelines of cat/grep/cut.

Every time a Bash script shells out to cut, head, grep or sed inside a loop, it forks a process per iteration — thousands of forks that dominate runtime and add failure modes. A surprising amount of text work can be done inside the shell itself with parameter expansion, and the rest is usually one awk invocation instead of a five-stage pipeline. Fewer processes means faster, more robust, more portable scripts.

Parameter expansion replaces most of coreutils

Bash parameter expansion trims prefixes and suffixes, substitutes, changes case, slices substrings, and supplies defaults — all without a fork. ${var#pat} and ${var##pat} strip from the front (shortest/longest match); ${var%pat} and ${var%%pat} from the back; ${var/old/new} substitutes; ${var:-default} supplies a fallback. Learn these and basename, dirname, and half your sed calls disappear.

path=/var/log/app/server.log
echo "${path##*/}"      # server.log      (basename, no fork)
echo "${path%/*}"       # /var/log/app    (dirname, no fork)
echo "${path##*.}"      # log             (extension)
 
name=server.log
echo "${name%.log}.gz"  # server.gz       (swap suffix)
 
url=HTTPS://Example.COM
echo "${url,,}"         # https://example.com   (lowercase, bash 4+)
echo "${url:0:5}"       # HTTPS           (substring)
 
echo "${MISSING:-fallback}"    # default if unset/empty

One awk beats a pipeline of five

awk is a whole streaming language: it reads records (lines) split into fields, runs your code per record, and keeps state across them. A pipeline like grep X | grep -v Y | awk '{print $3}' | sort | uniq -c is usually one awk program that filters, selects and counts in a single pass — fewer processes, one file read, and no intermediate buffering. It is the right tool the moment you need fields, sums, or grouping.

# sum bytes (field 10) of 200 responses in an access log, one pass
awk '$9==200 { bytes += $10 } END { print bytes }' access.log
 
# top 5 client IPs by request count
awk '{ c[$1]++ } END { for (ip in c) print c[ip], ip }' access.log \
  | sort -rn | head -5
 
# filter + select + default in one program (replaces grep|grep -v|cut)
awk -F: '$3 >= 1000 && $1 != "nobody" { print $1 }' /etc/passwd
 
# join fields with a custom OFS
awk 'BEGIN{OFS=","} {print $1,$3}' data.tsv

sed for surgical stream edits

sed earns its place for in-stream substitution and line-addressed edits. Prefer it over hand-rolled loops for search-and-replace, and know the safe idioms: a non-slash delimiter when the pattern contains paths, -i.bak for an in-place edit with a backup, and address ranges to scope a change. But resist the urge to build a program in sed — once you need fields or arithmetic, switch to awk.

# substitute with an alternate delimiter (no backslash soup for paths)
sed 's#/old/path#/new/path#g' config.ini
 
# in-place with a backup file (portable-ish; GNU sed)
sed -i.bak 's/DEBUG/INFO/g' app.conf
 
# only within a line range, delete matching lines
sed '/^#/d; 10,20 s/foo/bar/' input.txt
 
# print just the block between two markers
sed -n '/BEGIN CERT/,/END CERT/p' bundle.pem