Text processing
grep, sed, awk, sort, uniq, cut, tr.
A pipe, written as a single vertical bar (|), is a conveyor belt. One command drops its output onto the belt, the next command picks it up, does one small job, and passes it along. None of these tools tries to do everything. grep finds lines. cut grabs a column. sort puts things in order. Bolt them together with pipes and you can pull a clean answer out of a million-line log without writing a program. In security and operations work, the log is usually where the story lives: who signed in, what broke, and which machine is rattling the doorknob.
grep: find the lines that match
grep is a highlighter. You hand it a word or a pattern, it reads a file one line at a time, and it gives back only the lines that contain a match. The name looks like a cat walked across the keyboard, but it is an old editor command spelled out: g/re/p, 'globally search for a regular expression and print'. A regular expression (a small pattern language where, for example, a dot stands for any single character) is what makes grep more than a plain word search. The flags you reach for constantly: -i ignores upper/lowercase, -n adds line numbers, -r searches a whole folder and everything under it, -v flips the meaning to show lines that do NOT match, and -c prints a count instead of the lines. -E turns on the fuller pattern syntax, so you can write 401|403|500 to match any of several codes at once.
Say you are chasing a database outage in an application log. You want every line that mentions a refused connection, and you do not care whether someone typed it 'connection' or 'Connection'. -i ignores the case difference and -n tells you where each hit is.
The same tool is a security scanner. Before code ships, sweep the whole tree for secrets that must never be committed: an AWS access key ID (an Amazon Web Services credential that always begins with the letters AKIA), or any line that assigns a password or an api_key. -r walks every folder underneath, and -E lets you match several patterns in one shot.
-c gives you a number instead of a screenful. But be careful what the number counts.
845. Keep that number. awk will shortly say 842, and the difference is the lesson: grep counted every line where the three characters 4, 0, 1 appear together, including a few where 401 sat inside a byte count or a timestamp. grep matches text, not meaning.
cut: grab one column
cut is a pair of scissors for columns. You tell it what separates the fields with -d (the delimiter, the character sitting between columns) and which field or fields you want with -f. The file /etc/passwd, the list of every account on the system, is a perfect target: its fields are separated by colons.
root:x:0:0:root:/root:/bin/bashdaemon:x:1:1:daemon:/usr/sbin:/usr/sbin/nologinsync:x:4:65534:sync:/bin:/bin/syncwww-data:x:33:33:www-data:/var/www:/usr/sbin/nologindeploy:x:1000:1000:Deploy User:/home/deploy:/bin/bash
Field 1 is the username and field 7 is the login shell (the program that runs when the account logs in). A shell like /bin/bash means a person can get a command prompt; /usr/sbin/nologin means the account is fenced off. So 'which accounts can actually log in?' is one line, and it is a real check you run when auditing a box you inherited.
cut has one weak spot: it splits on a single character, so columns lined up with runs of spaces confuse it. Log lines are usually separated by single spaces though, so pulling the address (the first field: the IP address, short for Internet Protocol address, the number that names a machine on a network) out of a web server's access log is clean.
sort and uniq: order, then count
Here is where it clicks together. sort puts lines in order (add -n to sort by number instead of alphabetically, so 9 comes before 80, and -r to reverse the order). uniq collapses repeated lines into one, and with -c it writes a count in front of each. The catch, and it is the single most common mistake with these tools, is that uniq only compares each line to the one directly above it. It collapses runs, not scattered duplicates. So you almost always sort first, to shove identical lines next to each other. The famous 'top talkers' pipeline stacks all of this to answer 'which address hit us most?'
One address at 842 requests when the next is at 317 is worth a look. Maybe it is a busy load balancer (a server that spreads incoming traffic across several machines). Maybe it is someone knocking. The next tools tell you which.
tr: swap or delete characters
tr (short for translate) works one character at a time, like a rubber stamp that replaces every letter of one kind with another. Give it two sets and it maps the first onto the second. The classic is forcing text to uppercase.
The one that saves your afternoon is -d, which deletes characters instead of swapping them. Files edited on Windows end each line with two invisible characters, a carriage return and a line feed (together, CRLF), where Linux uses only the line feed. That stray carriage return breaks shell scripts with baffling errors like 'bad interpreter: /bin/bash^M'. cat -A makes it visible as a ^M at the end of each line, and tr -d strips it out.
The ^M is gone and the script runs. tr also has -s, which squeezes runs of a repeated character down to one. That is how you rescue cut when a file uses uneven spacing: tr -s ' ' first to collapse the gaps, then cut.
sed: find and replace in a stream
sed (the stream editor) runs down the text with a red pen, applying the same edit to every line, without ever opening the file in a window. Its bread and butter is substitution, written s/old/new/. By default it changes the first match on each line; add a g on the end (for 'global') to change all of them. It prints the result to the screen and leaves the file untouched, which is exactly the safe way to preview a change before you commit to it.
sed can also pick out lines by number. -n tells it to stay quiet and print nothing on its own, and the p command prints only what you ask for, so 'show me lines 2 through 4' reads like this.
Before you paste a log into a chat or a ticket, scrub the secrets out of it. With -E for the fuller pattern syntax, one substitution can black out anything shaped like an AWS key so you do not leak a live credential into a support thread.
To change the file itself instead of printing to the screen, add -i (in place). Write it as -i.bak and sed saves the original with a .bak extension first, so you have something to revert to.
awk: column-aware processing
awk (named for its three inventors, Aho, Weinberger, and Kernighan) is a small language for working with columns, closer to a spreadsheet than to a single command. It splits every line into fields for you: $1 is the first field, $2 the second, on up, and $0 is the whole line. You can print fields, test them, and add them up. The shape is condition { action }: for each line, if the condition is true, run the action. Remember grep counting 845 lines with '401' in them? awk can match the status field exactly. In this access log the HTTP status code sits in field 9. (HTTP, HyperText Transfer Protocol, is the language browsers and servers speak. The status code is the short number the server sends back to report how a request went; 401 means 'unauthorized', a failed login.)
842 real failures, not 845, and now you can see who owns them: one address failed to log in 840 times. That is not a person who forgot a password. That is a machine guessing them, a brute-force attempt (trying password after password until one works), and you found the source in a single line.
awk can also do arithmetic across lines, which is how you spot data leaving the building. Field 10 is the response size in bytes. Total it per address, sort, and a machine quietly pulling far more data than the rest stands right out.
10.0.1.5 barely showed up in the request count, yet it pulled eight megabytes while everyone else moved kilobytes. That is the kind of quiet outlier worth a question, and often the first visible sign of data being copied out.
uniq -c on it without sorting first. The same address shows up hundreds of times, scattered all through the file. The counts come out wrong: each address appears many times over, each with a tiny count. What went wrong?grep -c 401 access.log and it reports 845, but awk '$9 == 401' access.log | wc -l on the same file counts 842. Why does grep report more?grep -c counts matching lines, not matches, so a line with two 401s still counts once.awk '{bytes[$1] += $10} END {for (ip in bytes) print bytes[ip], ip}' access.log | sort -rn | head puts 10.0.1.5 on top with 8.2 MB, even though that address barely appeared in the request-count ranking. What is the most reasonable read?Keep the failed-login line where you can grab it fast: awk '$9 == 401 {print $1}' access.log | sort | uniq -c | sort -rn | head. It turns a wall of log into a short list of addresses ranked by how hard they failed. When one address sits at 840 tries and the next is at 2, you are not looking at a forgetful user. That is a machine guessing passwords. Your next move: check whether any of those 840 attempts returned a 200 rather than a 401 (a 200 means a guess landed), then block the address at the firewall before it keeps hammering the login.
Try this
Work through “awk: column-aware processing” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: grep reads your pattern as a regex, not plain text. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.