Parsing at scale: generators, ReDoS & structured logs

Streaming huge inputs, catastrophic-regex safety, and JSON logging.

A busy web server can write tens of gigabytes of access logs in a single day. Your triage script has to read all of it, find the handful of lines that matter, and do it without falling over. Three quiet decisions settle whether that script survives contact with real data: how much of the file it holds in memory at once, whether the patterns it matches can be turned against it, and whether what it prints can be read by the next tool in the chain. Get those wrong and your tool either runs out of memory, hangs forever on one crafted line, or spits out text nobody can query. This lesson fixes all three.

Read one line, not the whole file

Reading a file comes in two shapes. You can photocopy the whole book and carry the stack around, or you can read one page, act on it, and turn to the next. Python's f.read() and f.readlines() are the photocopier: they pull every byte into memory before you touch a single line. That's fine for a 2 MB config file. It's a disaster for a 50 GB log on a box with 8 GB of RAM (random-access memory, the fast working memory a program runs in). The process keeps growing until the OOM killer (the out-of-memory killer, the part of Linux that kills a process when the machine runs out of memory) steps in and ends it, usually at the worst possible moment.

A generator is the read-one-page-at-a-time reader. It's a function that hands you a single item, pauses, remembers exactly where it stopped, then picks up again when you ask for the next one. The yield keyword is what turns an ordinary function into that kind of pausable producer. Iterating a Python file object already works this way: each turn of the loop reads the next line and nothing more. Write your filters and transforms as generators too, and you can chain them into a pipeline where nothing is computed until you pull results out the far end. Every stage stays lazy. Only one line sits in memory at a time, so the cost of reading the file stays flat no matter how large it grows.

from collections import Counter
 
def lines(path):
    with open(path) as f:
        yield from f                      # one line at a time; never loads the file
 
def parse(rows):
    for line in rows:
        parts = line.split()
        if len(parts) >= 9:               # skip short / malformed lines
            yield {"ip": parts[0], "status": parts[8]}
 
def failed(records):
    for r in records:
        if r["status"][0] in ("4", "5"):  # 4xx client errors, 5xx server errors
            yield r
 
# nothing above has run yet; the Counter is what pulls lines through the pipeline
top = Counter(r["ip"] for r in failed(parse(lines("access.log"))))
for ip, n in top.most_common(5):
    print(f"{n:>6}  {ip}")

Point it at a real access log and the shape of an attack falls right out of the data.

Two source IP addresses (Internet Protocol addresses, the numeric label that identifies each machine on a network) account for over eleven thousand 4xx and 5xx responses between them, while the normal internal hosts sit in the low hundreds. Pull the request paths for those two addresses and you will almost always find them pounding a single endpoint, usually something like /login. That is the fingerprint of credential stuffing or a brute-force sweep (an attacker running many username and password guesses from a script). You found it by reading a multi-gigabyte file in a few kilobytes of memory.

Here is the same count run two ways under tracemalloc (Python's built-in memory tracer), one version holding the whole file in a list, the other streaming it line by line.

Both loops give the identical answer. The read-all version held every line in a list and peaked at about 2.4 megabytes for this small sample. The streaming version stayed at 34 kilobytes. The ratio on a tiny file is not the lesson. The lesson is that the streaming number does not move when the file grows to 50 GB, while the read-all number keeps climbing until the machine dies.

Catastrophic backtracking, a regex that fights back

A regular expression (regex, a compact pattern language for matching text) usually feels instant. Some patterns hide a trap, though. Think of a guard at a door holding a checklist who, when the last item fails to match, refuses to give up: he walks back and re-tries every earlier item in every possible combination before admitting the visitor does not fit. For certain pattern shapes, the number of combinations doubles with every extra character of input. That behavior is called catastrophic backtracking, and the denial-of-service attack built on it is ReDoS (Regular expression Denial of Service, freezing a program by feeding it one carefully chosen string).

The warning sign is a quantifier nested inside another quantifier, like (\w+\s*)+. The inner \w+ (one or more word characters) and the outer + can carve the same run of text apart in exponentially many ways. Add one trailing character that makes the whole match fail, and the engine explores all of those ways before it gives up. This script times Python's built-in re engine against strings of the letter a with a single ! stuck on the end to block the match.

import re, time
 
bad = re.compile(r"^(\w+\s*)+$")          # nested quantifier: the trap
 
for n in (16, 20, 24, 28):
    s = "a" * n + "!"                     # the trailing ! forces a full backtrack
    start = time.perf_counter()
    bad.search(s)
    print(f"n={n:>2}  {time.perf_counter() - start:8.3f}s")

Four more characters, roughly fifteen times slower, at every single step. By n=40 you are into hours of work on one string. Now remember where log lines and HTTP (Hypertext Transfer Protocol, the request format the web runs on) header fields come from: whoever is talking to your server. If that same regex parses a User-Agent value (the header where a browser or client announces what it is), an attacker sends one 40-character request and your parser stops answering. One request. The built-in re module has no timeout. It spins until someone kills the process.

The repair has three parts. First, remove the nested quantifier. Rewrite ^(\w+\s*)+$ as ^[\w ]{1,200}$: one character class, an explicit upper bound, no nesting. Second, anchor and length-cap anything you run on untrusted input, so a match cannot wander off looking for combinations. Third, when you genuinely need a complex pattern on hostile input, run it under the third-party regex module, which takes a timeout= argument and raises TimeoutError when a single match runs past its budget.

import re
 
# SAFER: one character class, hard length cap, anchored, no nesting
good = re.compile(r"^[\w ]{1,200}$")
good.match("a" * 100000 + "!")            # returns None in well under a millisecond
 
# For a genuinely complex pattern on attacker-controlled input,
# give the match a wall-clock budget it cannot exceed:
import regex                              # pip install regex  (third-party)
try:
    m = regex.match(pattern, untrusted, timeout=0.5)
except TimeoutError:
    raise ValueError("input rejected: regex exceeded time budget")

Nested quantifiers are not the only offender. Overlapping alternations like (a|a)*, an optional group followed by a required one, and most hand-written email or URL (Uniform Resource Locator, a web address) validators all backtrack badly. Do not eyeball a pattern and call it safe. Test every regex that touches untrusted input with a handful of long adversarial strings and a stopwatch, and treat any pattern you cannot bound in length as a liability rather than a convenience.

Structured logs a machine can read

There are two ways to label a moving box. You can scrawl "kitchen stuff, some fragile" on the side in marker, or you can stick on a label with fields: room=kitchen, fragile=yes, count=12. The marker note reads fine to one person unpacking one box. The fields let a warehouse system sort ten thousand boxes without anyone reading a word. Log lines face the same fork in the road.

A tool that prints "Blocked 203.0.113.7 after too many failures" forces every downstream reader to grep (scan text line by line for a pattern) and guess. A tool that prints one JSON (JavaScript Object Notation, a simple key/value text format machines parse natively) object per event lets a log pipeline filter, count, and alert on exact fields. Attach context as fields, never by gluing values into an English sentence. Send logs to stderr (standard error, the secondary output stream) and keep stdout (standard output, the primary stream) for your tool's real result, so a caller can pipe the data onward while the logs go somewhere else. And never write a secret into a log line. The examples below pipe that output through jq (a small command-line tool for slicing and filtering JSON).

import logging, json, sys
 
class JsonFormatter(logging.Formatter):
    def format(self, rec):
        payload = {
            "ts":     self.formatTime(rec, "%Y-%m-%dT%H:%M:%S%z"),
            "level":  rec.levelname,
            "logger": rec.name,
            "msg":    rec.getMessage(),
        }
        if hasattr(rec, "context"):
            payload.update(rec.context)   # merge structured fields
        return json.dumps(payload)
 
h = logging.StreamHandler(sys.stderr)     # logs to stderr, not stdout
h.setFormatter(JsonFormatter())
log = logging.getLogger("triage"); log.addHandler(h); log.setLevel(logging.INFO)
 
log.info("auth failures over threshold",
         extra={"context": {"src_ip": "203.0.113.7", "count": 7965, "action": "blocked"}})

Because src_ip and count are real fields, the next stage can pick out exactly the events it cares about instead of parsing prose. Notice the numbers stayed numbers: count is 7965, not "7965", so a query can compare it against a threshold. That only works because you built the object rather than formatting a string.

That line ships straight into Elasticsearch or Loki (log stores that index fields, so you can search and alert on them) and becomes a rule: page me when count crosses a threshold from one source. No fragile text matching, no regex over your own logs, and the fields carry through every hop of the pipeline.

The value is still attacker data

Streaming safely and matching safely buy you nothing if the last step hands the value to something that will execute it. A parsed src_ip came from the log, which came from the network, which came from an attacker. Where it travels next is where the real damage happens. Treat every parsed field as hostile right up to the moment it is used.

Try this

Work through “The value is still attacker data” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: parse carefully, then don't inject. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related