Text and regex for logs & output

Extract fields without fragile string slicing.

A regular expression (a written pattern that describes what text should look like, nearly always shortened to 'regex') works like the Find box in a text editor, except what you type can describe a shape instead of an exact word. Instead of hunting for the literal string 203.0.113.42, you describe 'four numbers joined by dots' and catch every address in the file at once. For a DevSecOps engineer (someone who builds security checks straight into the pipeline that ships software), that is the difference between reading a log by eye and writing a tool that chews through ten thousand lines a second.

A pattern is a tiny search language

Python comes with regex support built in, in a standard-library module called re (short for 'regular expressions'). You hand it a pattern, it hands you back matches. A small set of building blocks covers most log work. \d means any digit. \w means any letter, digit, or underscore. \s means any whitespace, like a space or a tab. And a plain dot means any character at all. Put a count after a piece and you say how many you want: \d+ is one or more digits, and \d{1,3} is between one and three of them. Wrap a piece in round brackets, (like this), and you tell the engine to remember whatever matched so you can pull it out later. That last move is what turns a smear of text into labeled fields you can actually use.

Compile the pattern once, reuse it forever

Every time Python runs a pattern, it first turns your pattern text into a small internal program, the way a recipe gets prepped before anyone starts cooking. re.compile() does that prep once and hands back a reusable object. Saving a search filter in your email works the same way: you write the query once instead of retyping it every morning. If you call re.search(pattern, text) inside a loop over a million log lines, Python has to look the pattern up on every pass. It keeps a small cache of recent ones, but you should not lean on that. Compile the pattern once, above the loop, and every line after that reuses the finished version. On a big file that is a real speed win, and the code reads better because the pattern now has a name you chose.

search, match, findall, finditer: four ways to look

These four look almost the same and behave very differently. match only checks the very start of the string, as if your pattern had an invisible 'must begin right here' pin on its front. search scans the whole string and stops at the first hit. findall returns every hit as a plain list of strings and forgets where each one sat. finditer returns every hit as a match object you can question, including its exact position. For counting and pulling fields out of logs, finditer is usually the one you want, because each result carries both its captured groups and its spot in the line.

import re
 
text = "user=root ip=10.0.0.9 user=deploy ip=192.168.1.5"
 
pat = re.compile(r"ip=(\d+\.\d+\.\d+\.\d+)")
 
print("match  :", pat.match(text))
print("search :", pat.search(text))
print("findall:", pat.findall(text))
print("finditer:")
for m in pat.finditer(text):
    print("   ", m.group(1), "at", m.span())

The first line shows match returning None, because the string opens with user=, not ip=, and match refuses to look past the front door. search found the first ip= further along and stopped there. findall handed back both addresses as a flat list. finditer returned the same two plus their character positions (the span), which is what you need if you want to blank them out in place before sharing a log with someone outside your team.

Pulling fields out of a real log line

A capture group is a labeled bucket. You wrap part of the pattern in round brackets, and whatever matches drops into that bucket for you to grab afterward. Plain groups are numbered 1, 2, 3 by the order their opening bracket shows up, which gets confusing the moment you have more than two. Named groups fix that: write (?P<ip>...) and the bucket is called ip instead of 'group 3'. Below is a line straight from an SSH (Secure Shell, the standard encrypted way to log in to a machine over the network) server's auth.log, the file Linux writes every login attempt into. The pattern next to it pulls out three things: the timestamp, the username someone tried, and the IP (Internet Protocol, the numeric address every machine on a network is reachable at) address the attempt came from.

import re
 
line = "Feb 10 13:45:11 web01 sshd[24800]: Failed password for invalid user admin from 203.0.113.42 port 54021 ssh2"
 
pattern = re.compile(
    r"(?P<ts>\w{3}\s+\d+\s[\d:]+)\s"        # timestamp: month day time
    r"\S+\ssshd\[\d+\]:\s"                   # host + process
    r"Failed password for (?:invalid user )?"
    r"(?P<user>\S+)\s"                       # username that was tried
    r"from (?P<ip>\d+\.\d+\.\d+\.\d+)"       # source IP address
)
 
m = pattern.search(line)
if m:
    print("timestamp:", m.group("ts"))
    print("user:     ", m.group("user"))
    print("ip:       ", m.group("ip"))

The (?:invalid user )? piece earns its keep quietly. (?:...) is a group that matches without making a bucket, and the trailing question mark means zero or one of it. A bad-username attempt reads 'Failed password for invalid user admin', while a real account that fumbled its password reads 'Failed password for root'. One pattern handles both shapes without a second line of code.

VERBOSE and MULTILINE: patterns you can actually read

That pattern was already dense, and production ones get worse. A cramped pattern reads like a sentence printed with every space and comma stripped out. Two flags give you the spacing and the margin notes back. re.VERBOSE tells the engine to ignore the whitespace you put inside the pattern and to allow # comments, so you can spread the pattern over several lines with a note beside each part. The catch: a real space now has to be written as \ (backslash-space) or \s, because ordinary spaces get thrown away. re.MULTILINE changes what ^ and $ mean. Normally ^ is the start of the whole string; with re.MULTILINE it matches the start of every line, and $ matches the end of every line. Together they let you write a pattern you can read and run it across a whole block of log at once. Here is a small tool that counts failed SSH logins per source address, the first thing you would reach for to catch someone trying passwords in bulk.

import re
from collections import Counter
 
log = """\
Feb 10 13:45:11 web01 sshd[24800]: Failed password for invalid user admin from 203.0.113.42 port 54021 ssh2
Feb 10 13:45:13 web01 sshd[24801]: Failed password for root from 203.0.113.42 port 54044 ssh2
Feb 10 13:45:14 web01 sshd[24802]: Accepted password for deploy from 198.51.100.7 port 51992 ssh2
Feb 10 13:45:19 web01 sshd[24803]: Failed password for root from 203.0.113.42 port 54070 ssh2
Feb 10 13:45:22 web01 sshd[24804]: Failed password for invalid user oracle from 45.83.64.9 port 33110 ssh2
Feb 10 13:45:25 web01 sshd[24805]: Failed password for root from 45.83.64.9 port 33144 ssh2"""
 
fail = re.compile(r"""
    ^.*Failed\ password\ for\            # keep only failed-login lines
    (?:invalid\ user\ )?                # optional 'invalid user' prefix
    (?P<user>\S+)\ from\                # the username that was tried
    (?P<ip>\d+\.\d+\.\d+\.\d+)          # the source IP address
""", re.VERBOSE | re.MULTILINE)
 
counts = Counter(m.group("ip") for m in fail.finditer(log))
for ip, n in counts.most_common():
    print(f"{n:>3}  {ip}")

Three failures from 203.0.113.42, two from 45.83.64.9, and the one successful Accepted login is left out because the pattern only matches Failed password lines. That sorted count is a working brute-force detector in a handful of lines, and it is the kind of thing you would run over a freshly rotated log every few minutes.

The trap: catastrophic backtracking (ReDoS)

A regex engine hunting for a match is like a person walking a maze. When a pattern can match the same stretch of text in more than one way, the engine may try route after route before it gives up. Most of the time that is cheap. But some patterns turn the maze into one with a doubling number of paths, so adding a few characters to the input multiplies the work wildly. Feed such a pattern a string that almost matches, and it can spin for seconds, minutes, or for all practical purposes forever, on a single line. That is a ReDoS (Regular expression Denial of Service): an attacker sends one short, carefully shaped string and pins a CPU (central processing unit, the chip that does the actual computing) core at full load.

The classic warning sign is a quantifier wrapped around something that already repeats, like (a+)+. Watch the time blow up as the input grows two characters at a step.

import re, time
 
evil = re.compile(r"^(a+)+$")           # nested quantifier: one group inside another
 
for n in (24, 26, 28):
    text = "a" * n + "!"                 # n copies of 'a', then a char that can't match
    start = time.perf_counter()
    evil.search(text)                    # forced to try every possible split
    print(f"{n} chars: {time.perf_counter() - start:5.2f}s")

Twenty-four characters finished in under a second; twenty-eight took over twelve. Every two extra characters roughly quadrupled the work. Your exact numbers will differ by machine, but the curve is the point: the cost runs off the top of the page. A string of forty a's would outlast your patience, and remember that the input came from outside your system, chosen by someone who wants it to hurt.

The fixes are dull and they work. Do not nest quantifiers: ^a+$ does the same job in linear time and clears forty thousand characters in a fraction of a millisecond. Be specific instead of greedy where you can, so a bounded \w{1,64} or [^\s]+ beats an open .* sitting next to another star. Test any new pattern against a long, almost-matching string before you ship it, so you meet the cliff in a test run and not in production. Never point a hand-written pattern at attacker-controlled text without a timeout or a vetted library behind it. When you need regex on untrusted input at scale, the RE2 engine (Google's linear-time regex library, available for Python as the google-re2 package) is built so it can never backtrack, so it cannot blow up this way; it rejects the handful of pattern features that would force backtracking and runs everything else in guaranteed linear time. A validator that can be hung on demand is a denial-of-service switch you have handed to strangers.

import re, time
 
safe = re.compile(r"^a+$")             # no nested quantifier -> linear time
text = "a" * 40000 + "!"               # 40k almost-matching characters
 
start = time.perf_counter()
safe.search(text)
print(f"took {time.perf_counter() - start:.4f}s on {len(text)} chars")

Same input size that would have frozen the nested pattern, and the safe one barely registers. The only change was removing the nesting. That is the whole trick: give the engine one clear way to match instead of thousands.

Related