Parsing logs at scale

Generators and streaming over read-it-all.

An `auth.log` (the file where a Linux machine records logins, sudo, and other authentication events) on a busy jump host behaves like the conveyor belt at baggage claim, not like a book. Bags keep arriving. The belt never stops, and it only ever gets longer. Each line is one event, appended to the end and never edited. Your job is to stand at the belt and pull off the bags you care about, without trying to lift the whole belt into your arms.

A log is an append-only text file: new events go on the end, and old lines are never rewritten. Parsing means taking one line of free-form text, like `Failed password for root from 203.0.113.47 port 54321 ssh2`, and turning it into labeled fields (a timestamp, a username, a source IP) that code can filter, count, and compare. "At scale" has a specific meaning here: the file no longer fits in RAM (random-access memory, the fast working memory your running programs live in). The habits you built on a 5 megabyte sample stop working, sometimes catastrophically.

The Failure That Stalls An Incident

Here is the classic mistake, and it lands mid-incident when you can least afford it. An analyst writes `open(path).read()` or `.readlines()`, tests it on a small extract pulled off the box, and it works. Then they point the same code at the live 11 gigabyte log while an attack is in progress. Forty minutes later the process dies with a one-word message: `Killed`. No results, no partial output, triage frozen. The OOM killer (out-of-memory killer, the part of the kernel that frees memory as a last resort by terminating a process) picked your Python as the fattest target and shot it.

The second failure is quieter and worse. A parser reads the whole file, chokes on a single malformed byte 80 percent of the way through, and throws an exception. Or, subtler still, it silently miscounts and hands you a confident wrong answer. Streaming fixes both at once: memory stays flat no matter how big the file gets, and you decide on purpose what happens to garbage input instead of being surprised by it.

Streams, Not Slurps

Reading a file line by line is like drinking from a water fountain instead of trying to swallow the whole reservoir. Python hands you the fountain for free, as long as you never ask for the reservoir. When you call `open()`, you get back a buffered reader. Iterate over it with a `for` loop and Python pulls the file off disk in small pieces (roughly 8 kilobytes at a time; `io.DEFAULT_BUFFER_SIZE`, the fallback size Python reaches for, is 8192 bytes), decodes one line, hands it to you, and forgets it. Peak memory is a few kilobytes whether the file is 5 megabytes or 500 gigabytes.

A generator extends that same laziness to code you write. Any function with a `yield` in it is a generator: instead of building a list and returning it all at once, it produces one item, pauses, and resumes only when asked for the next. Chain a few generators together and you have a pipeline where a single line flows from disk, through your parsing, into your counter, and out, before the next line is ever read.

That last command counted 96 million lines and peaked around 10 megabytes of memory, because iteration only ever holds one chunk at a time. Notice `errors='replace'`. Real logs are full of bytes that are not valid UTF-8 (the standard encoding that maps raw bytes to characters and back): writes torn in half when `logrotate` rolled the file, binary junk from a daemon that crashed mid-line, deliberately malformed strings sent by a client probing you. The default, `errors='strict'`, raises `UnicodeDecodeError` on the first such byte, so a single bad byte at the 80 percent mark throws away everything the scan already read. `replace` swaps each undecodable byte for the replacement character U+FFFD and keeps reading. You count the damage instead of dying from it.

A Real Parser: SSH Brute-Force Triage

The real task: count failed SSH (secure shell, the encrypted remote-login protocol) logins per source IP (internet protocol address, the number identifying a machine on the network) across the live log and every rotation of it. `logrotate` (the tool that keeps logs from growing without bound) renames yesterday's `auth.log` to `auth.log.1`, then compresses older ones to `auth.log.2.gz`, `auth.log.3.gz`, and so on. A complete count has to read the plain files and the gzip-compressed ones the same way. The pattern below uses named groups, written `(?P<ip>...)`, so each captured field comes out in a dictionary keyed by name (`m['ip']`) instead of by a brittle position number.

"""Count failed SSH logins per source IP across rotated auth logs."""
import gzip
import json
import re
import sys
from collections import Counter
from pathlib import Path
 
FAILED = re.compile(
    r"^(?P<ts>\w{3} [ \d]\d [\d:]{8}) (?P<host>\S+) sshd\[\d+\]: "
    r"Failed password for (?:invalid user )?(?P<user>.+?) "
    r"from (?P<ip>[0-9a-fA-F:.]+) port \d+"
)
 
 
def read_lines(path: Path):
    """Yield decoded lines from a plain or gzip log, one at a time."""
    opener = gzip.open if path.suffix == ".gz" else open
    with opener(path, mode="rt", encoding="utf-8", errors="replace") as fh:
        yield from fh
 
 
def main() -> int:
    paths = sorted(Path("/var/log").glob("auth.log*"))
    hits, malformed = Counter(), 0
    for path in paths:
        for line in read_lines(path):
            m = FAILED.match(line)          # match() anchors at position 0
            if m:
                hits[m["ip"]] += 1
            elif "Failed password" in line:
                malformed += 1              # format drift: surface, never drop
    for ip, count in hits.most_common(20):
        print(json.dumps({"ip": ip, "failed_logins": count}))
    print(f"unparsed 'Failed password' lines: {malformed}", file=sys.stderr)
    return 1 if malformed else 0
 
 
if __name__ == "__main__":
    raise SystemExit(main())

Every choice in there is a decision about scale or safety. The pattern is compiled once when the module loads, not rebuilt per line. `match()` anchors at position 0, so a line that does not start correctly is thrown out after a few characters, where `search()` would scan the whole line looking for a match anywhere. The `(?:invalid user )?` swallows both wordings sshd uses, one for real accounts and one for accounts that do not exist. `gzip.open` in `rt` (read-text) mode makes the compressed rotations look identical to plain files to the rest of the code. And `Counter` grows with the number of unique IPs (thousands of keys) not with the number of lines (tens of millions), so memory stays flat.

Eleven gigabytes read in under five minutes, with a peak footprint of about 15 megabytes (that `Maximum resident set size` line). Roughly 40 megabytes per second per core is what regex-bound CPython gives you, and that number is a planning tool: it tells you before you start whether an ad-hoc scan finishes over a coffee or needs a different engine entirely.

Every Log Line Is Attacker Input

Every field the regex pulls out was typed by whoever connected to the box, and some of them connected specifically to mess with you. Usernames show up carrying SQL (structured query language, the language databases speak) fragments, ANSI escape sequences (control codes a terminal obeys, which can move your cursor or repaint the screen when you `cat` the file raw), and log-injection payloads. That last one is the sharp one: an attacker picks a username that reads like a whole log line, hoping a lazy `grep` later reports a login that never happened, a forged `Accepted password for root` sitting inside a field they controlled.

Two rules follow. Never feed an extracted field into a shell command or an SQL query by pasting it into a string. Never `print()` a raw field to a terminal. Emit through `json.dumps`, which escapes quotes, newlines, and control characters for you, so a booby-trapped username lands in your output as harmless text. That is why the script writes NDJSON (newline-delimited JSON, meaning one JSON object per line, where JSON is the plain-text format for structured data). It is the common tongue of security tooling: `jq`, DuckDB, and most SIEMs (security information and event management platforms, the systems that centralize and search logs) read it directly, and it streams as naturally as the input did.

One more safety habit. When something downstream reads `offenders.ndjson`, do not write to that path directly. Write to a temporary file and then `os.replace()` it into place, which swaps the name atomically. A run killed halfway then leaves the old complete file untouched, instead of a truncated one that happens to parse as "only three offenders today".

Where One Script Stops

Know the ceiling of what you built. This is one process on one core, and regex matching holds the GIL (global interpreter lock, the mechanism that lets only one thread execute Python code at a time), so adding threads buys you nothing. The cheap parallel win is `multiprocessing.Pool` with one worker process per rotated file, since each file is independent. Past tens of gigabytes a day of continuous parsing, stop scripting the pipeline at all: a log shipper like Vector or Filebeat feeding columnar storage is the right layer, and your Python moves up to enrichment and detection logic. Better yet, push the producers toward structured JSON logging, so parsing becomes `json.loads` (or `orjson`, several times faster), and the fragile regex only survives for legacy sources that cannot be changed.

Treat the lines your parser could not read as evidence, not noise. The script counts every `Failed password` line the regex rejected and exits nonzero when any exist. A sudden spike in those rejects means one of two things: the log format drifted after a package upgrade, or someone is tampering with what gets written. Both need a human to look. Parsers that silently drop what they cannot read are how a detection quietly rots, until the day it matters and it is already dead.

The parser works, but the log paths, the top-20 cutoff, and the exit-nonzero threshold are all baked into the source. Fine for this one host, useless for a teammate's. The next lesson turns this into a real command-line tool with `argparse`: flags for the input paths and thresholds, reading from standard input so it composes with a pipe, and help text good enough that someone can run it at 3 a.m. without opening the file.

Related