Regular expressions for logs and tool output

Named groups, readable patterns, ReDoS and validating with ipaddress.

Logs and tool output are text with a structure you know but Python does not. This lesson shows how to pull fields out of that text with the re module, how to write a pattern a colleague can still read next year, and where regular expressions stop being enough. You will parse Ubuntu 26.04 sshd log lines with named groups, defend the parser against a user name crafted to mislead it, validate addresses with ipaddress instead of a pattern, and watch a badly shaped pattern freeze on 40 characters of input, then fix it with a construct the standard library has had since Python 3.11. The lab works in ~/ps-regex; unpack the lesson files in your home directory, or create the files as they appear.

Patterns and the ways to look

A regular expression (regex) is a pattern that describes text: \d means a digit, + means "one or more of the previous item", and parentheses make a group whose text you can read back. Write patterns as raw strings, r"...", so that Python passes each backslash to re unchanged instead of treating \d as a string escape. re.compile() turns a pattern into an object you can name and reuse. The module also caches the patterns you pass as strings, so compiling is mainly about readable code, not speed.

import re
 
line = "sshd-session[2214]: Failed password for root from 203.0.113.42 port 54044 ssh2"
port = re.compile(r"port (\d+)")
 
print("match:    ", port.match(line))
print("search:   ", port.search(line))
print("fullmatch:", re.fullmatch(r"\d+", "54044"), re.fullmatch(r"\d+", "54044 ssh2"))
print("findall:  ", re.findall(r"\d+", line))
for m in port.finditer(line):
    print("finditer: ", repr(m.group(0)), "group(1):", repr(m.group(1)), "span(1):", m.span(1))

Each method answers a different question. match() only tries the very start of the string, and the line starts with sshd-session, so it found nothing: None. search() scans for the first place the pattern fits and returns a match object; span=(63, 73) is where port 54044 sits. fullmatch() succeeds only when the whole string fits, which is what you want for validating one value: "54044" fits \d+, "54044 ssh2" does not.

findall() returns every match as a list of strings, and its result is a warning: \d+ found 203, 0, 113 and 42 as separate numbers and the 2 of ssh2. A pattern matches a shape wherever the shape occurs, so describe the context as well (port (\d+)), not just the value. finditer() returns one match object per hit. group(0) is the whole match, group(1) the first group, and span(1) the position of just that group, which is what you need to mask or replace only the value.

Named groups and VERBOSE on a real log

Ubuntu 26.04 writes /var/log/auth.log with RFC 3339 timestamps, and the OpenSSH process that handles a login is called sshd-session. The lab uses a copy of such a log with test addresses, so the output is the same on every machine. Save it as auth.log. Its eighth line is crafted, and its last line comes from an IPv6 address:

2026-09-27T08:14:02.118204+00:00 web01 sshd-session[2211]: Failed password for invalid user admin from 203.0.113.42 port 54021 ssh2
2026-09-27T08:14:05.402911+00:00 web01 sshd-session[2211]: Failed password for invalid user admin from 203.0.113.42 port 54021 ssh2
2026-09-27T08:14:09.771380+00:00 web01 sshd-session[2214]: Failed password for root from 203.0.113.42 port 54044 ssh2
2026-09-27T08:15:12.006517+00:00 web01 sshd-session[2230]: Accepted publickey for deploy from 198.51.100.7 port 51992 ssh2
2026-09-27T08:16:40.330128+00:00 web01 sshd-session[2241]: Failed password for invalid user oracle from 203.0.113.9 port 33110 ssh2
2026-09-27T08:16:44.918733+00:00 web01 sshd-session[2241]: Failed password for root from 203.0.113.9 port 33144 ssh2
2026-09-27T08:17:03.551902+00:00 web01 sshd-session[2250]: Failed password for deploy from 198.51.100.7 port 52010 ssh2
2026-09-27T08:18:21.204471+00:00 web01 sshd-session[2262]: Failed password for invalid user root from 10.9.9.9 from 192.0.2.50 port 40022 ssh2
2026-09-27T08:19:30.640015+00:00 web01 sshd-session[2270]: Failed password for root from 999.1.2.3 port 40100 ssh2
2026-09-27T08:20:02.884120+00:00 web01 sshd-session[2281]: Failed password for root from 2001:db8::5 port 40200 ssh2

Here are the two lines before the last, without the timestamp, host and process fields, and the kind of pattern people write first, run on the first of them:

This is the crafted user name from the Bash lesson "Arrays and parameter expansion" and from "Python for Bash users: values, collections and loops", now against a regular expression: the client chooses the user name it sends, and it may contain spaces. Someone who logs in as root from 10.9.9.9 produces a line with two froms, and the first-draft pattern took 10.9.9.9 as the source address. A tool that blocks addresses after failed logins would block a host the attacker named and leave the real source, 192.0.2.50, alone. The field sshd writes itself is at the end of the line, so anchor the pattern there:

import re
import sys
from collections import Counter
 
FAILED = re.compile(
    r"""
    \ sshd-session\[[0-9]+\]:
    \ Failed\ password\ for
    \ (?:invalid\ user\ )?        # present when the account does not exist
    (?P<user>.+)                  # the name the client sent: it may contain spaces
    \ from\ (?P<ip>[0-9.]+)       # .+ is greedy, so this is the LAST " from "
    \ port\ (?P<port>[0-9]+)\ ssh2$
    """,
    re.VERBOSE | re.ASCII,
)
 
 
def main(argv: list[str]) -> int:
    counts = Counter()  # a dict that starts every missing key at 0
    users = {}
    unparsed = 0
    with open(argv[0], encoding="utf-8", errors="replace") as log:
        for line in log:
            if "Failed password" not in line:
                continue
            m = FAILED.search(line.rstrip("\n"))
            if m is None:
                unparsed += 1
                continue
            counts[m["ip"]] += 1
            users.setdefault(m["ip"], set()).add(m["user"])
    for ip, n in counts.most_common():
        print(f"{n:>3}  {ip:<15} users: {', '.join(sorted(users[ip]))}")
    if unparsed:
        print(f"fail_count: {unparsed} line(s) did not match", file=sys.stderr)
    return 0
 
 
if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

re.VERBOSE lets a pattern span lines: unescaped whitespace is ignored and # starts a comment, so a literal space is written \ . Put that \ at the start of a line: a backslash at the end of a line escapes the newline, not a space, and the pattern then silently matches nothing. (?P<user>...) is a named group, read back as m["user"]. (?:...) groups without capturing, and the ? after it makes invalid user optional. .+ is greedy: it first takes the rest of the line, then gives characters back until from , an address, a port and ssh2 fit before $, the end of the line. That makes the address the last from on the line, which is the one sshd wrote.

Counter, from the standard collections module, is a dictionary that treats a missing key as 0, so counts[ip] += 1 works on the first hit where a plain dict raised KeyError in the basics lesson; most_common() returns the pairs sorted by count. users.setdefault(ip, set()) returns the set stored for ip, creating an empty one on first use.

The crafted line is now counted against 192.0.2.50, with the odd user name visible in the report. One line did not match: the log's last line has an IPv6 source, 2001:db8::5, and [0-9.]+ accepts only IPv4 characters. Counting unparsed lines on stderr is what made that visible; a parser that skips lines silently would have hidden an attacker who arrived over IPv6. The fix is to widen the group to [0-9A-Fa-f:.]+ and let the ipaddress module in the next section decide what is an address.

The pattern uses [0-9] and the re.ASCII flag for a reason. For str patterns, \d matches any Unicode decimal digit and \w any Unicode letter or digit:

\d+ accepted the Arabic-Indic digits ٨٠, and int() turned them into 80. A port check built on \d lets that text through a validator that looked strict. re.ASCII limits \d, \w and \s to ASCII; [0-9] says the same thing explicitly.

A match is not validation: ipaddress

fail_count.py reported 999.1.2.3 as an address, because [0-9.]+ checks characters, not meaning. Before an extracted value reaches a block list, a firewall rule or a query, parse it with a module that knows the format. For addresses that is ipaddress:

import ipaddress
import sys
 
JUMP_HOSTS = ipaddress.ip_network("198.51.100.0/24")
 
addresses = []
for text in sys.argv[1:]:
    try:
        addr = ipaddress.ip_address(text)
    except ValueError as err:
        print("rejected:", err)
        continue
    if addr in JUMP_HOSTS:
        print(f"skipped:  {addr} is in {JUMP_HOSTS}")
        continue
    addresses.append(addr)
 
print("sorted as text:     ", sorted([str(a) for a in addresses]))
print("sorted as addresses:", [str(a) for a in sorted(addresses)])

ip_address() returns an IPv4Address or IPv6Address object, or raises ValueError. It refused 999.1.2.3 and also 010.0.0.1: some parsers read a leading zero as octal, so that text means 8.0.0.1 to them and 10.0.0.1 to others, and Python refuses the ambiguity. ip_network() describes a range, and in tests membership, which is how the jump host was skipped without string tricks.

The last two lines finish a job the basics lesson left open. Sorted as text, 203.0.113.42 comes before 203.0.113.9 because 4 sorts before 9. Address objects compare as numbers, so sorted(addresses) puts .9 first. Python refuses to compare an IPv4 with an IPv6 address, so sort the two families separately.

Catastrophic backtracking (ReDoS)

Python's re engine is a backtracking engine: when one way of matching fails, it goes back and tries another. For most patterns the alternatives run out quickly. A quantifier inside a repeated group is the exception, because the text can be split between the inner and the outer repetition in a number of ways that grows exponentially with its length. On text that almost matches, the engine tries them all before it answers "no". This is regular-expression denial of service (ReDoS): a short string that keeps a CPU busy. The demonstration uses tiny inputs and always runs under timeout 5:

import re
import sys
import time
 
# Meant to accept host names separated by single spaces: "web01 web02".
# The group repeats, and so does \w+ inside it: a nested quantifier.
HOSTS = re.compile(r"^(\w+\s?)+$")
 
for arg in sys.argv[1:]:
    n = int(arg)
    text = "a" * n + "!"  # almost matches: one bad character at the end
    start = time.perf_counter()
    HOSTS.search(text)
    print(f"{n:>3} characters: {time.perf_counter() - start:6.3f} s")

Each two extra characters multiplied the time by about four, so 24 characters took around a second. At 40 characters the same curve predicts hours, and timeout stopped the process after five seconds with exit status 124. Your timings will differ; the shape of the curve will not. Python's re has no timeout of its own, and a service that runs this pattern on a request field can be stalled by anyone who can send one.

The warning sign is (\w+\s?)+: a group that repeats, containing \w+, which also repeats, and nothing forces each repetition of the group to start somewhere definite. Since Python 3.11 the standard library has two fixes for exactly this shape. An atomic group (?>...) keeps whatever its contents matched and never lets the engine go back inside it, and a possessive quantifier such as ++ does the same for a repetition. The third fix needs no new syntax: rewrite the pattern so that every repetition must start with a space, which leaves only one way to split the text.

import re
import time
 
FIXES = {
    "atomic group": r"^(?>\w+\s?)+$",
    "possessive": r"^(\w+\s?)++$",
    "de-nested": r"^\w+(\s\w+)*\s?$",
}
NESTED = re.compile(r"^(\w+\s?)+$")
SAMPLES = ["web01 web02", "web01 ", "web01  web02", "web01;id"]
LONG = "a" * 40_000 + "!"
 
print(f"{'nested':<13}", [bool(NESTED.search(s)) for s in SAMPLES])
for name, pattern in FIXES.items():
    fixed = re.compile(pattern)
    start = time.perf_counter()
    fixed.search(LONG)
    elapsed = time.perf_counter() - start
    answers = [bool(fixed.search(s)) for s in SAMPLES]
    print(f"{name:<13}", answers, f"40,001 characters: {elapsed:.4f} s")

All three fixed patterns accept and reject the same four samples as the original, including the trailing space and the refusal of a double space and a ;, and each handled 40,001 characters in a few milliseconds or less. Check that a rewritten pattern still accepts what the old one accepted, as the sample list does here, before you replace it. For patterns that run on untrusted input at scale, the scripting-adv lesson on hostile input adds time budgets and engines that never backtrack.

Try this

Write check_user.py NAME... in ~/ps-regex. An account name is word characters in parts joined by single dots, like svc.backup. Print valid or invalid and the name for each argument, and exit 1 if any name was invalid. Break it first: use (\w+\.?)+ with fullmatch() and re.ASCII, confirm it accepts svc.backup, then run it under timeout 5 on 40 as followed by ! ("$(python3 -c 'print("a" * 40 + "!")')") and see exit status 124. Fix it with \w+(?:\.\w+)* and verify: svc.backup and deploy are valid, svc..backup and -rf are invalid, the exit status is 1, and the 41-character string is reported invalid at once.

Takeaway

Anchor patterns on the parts of a line the program wrote, not on text a client controls, and hand every extracted value to a real parser such as ipaddress before acting on it. When a group that repeats contains a repeat of its own, rewrite it, or make it atomic or possessive, and test it on a long almost-matching string under a timeout.

Related