Parse auditd logs with Python before they hit your SIEM

Raw audit records are unreadable and expensive to index. Normalize them with 80 lines of Python and cut ingest cost.

auditd is the kernel's flight recorder — it can tell you exactly who ran sudo, what process wrote to /etc/shadow, and when. The catch is the format: a single logical event is spread across several type= records, each a wall of key=value pairs. Python turns 'who touched this file last week' from an afternoon of grep gymnastics into a function you run in a second — and structured JSON output cuts SIEM ingest cost dramatically.

This note parses raw audit logs into merged events, answers a concrete identity question, and shows the watch rules that make sure the events exist in the first place. For host logging fundamentals, pair with Linux hardening; for shipping parsed events onward, see Python for security automation.

Group records into events

The whole trick is the event id — the number after the colon in audit(timestamp:ID). Split each line into a dict, pull that id, and collect every record that shares it. Now one event is one Python object instead of three scattered lines. Handle quoted values carefully — audit fields use both key=value and key="value with spaces" forms.

import re, collections, json
 
FIELD = re.compile(r'(\w+)=(?:"([^"]*)"|(\S+))')
EVENT = re.compile(r'audit\((\d+\.\d+):(\d+)\)')
 
def parse(path):
    events = collections.defaultdict(dict)
    for line in open(path):
        m = EVENT.search(line)
        if not m:
            continue
        eid = m.group(2)
        for k, q, u in FIELD.findall(line):
            events[eid][k] = q or u
    return events
 
if __name__ == "__main__":
    for eid, e in parse("/var/log/audit/audit.log").items():
        print(json.dumps({"id": eid, **e}))

Answer a real question

With events assembled, questions become one-liners. 'Which non-root user edited a file we watch under the identity key, and with what binary?' — filter on the key and read the fields you kept. The sentinel auid=4294967295 means 'unset' in audit-speak; filter it out when you care about human actors.

For fleet-wide questions, point the parser at forwarded logs in /var/log/remote/ or stdin from journalctl -u auditd -o export. The merge logic stays identical — only the input path changes.

UNSET = "4294967295"
 
for eid, e in parse("/var/log/audit/audit.log").items():
    if e.get("key") == "identity" and e.get("auid") not in (None, UNSET):
        print(f"uid={e['auid']} ran {e.get('exe')} on {e.get('name')}")

Make sure the events exist

None of this works if the events were never recorded. Add watch rules so auditd emits on the paths and syscalls you care about — a write to /etc/passwd should always leave a trace tagged with a key you can filter on. Load rules from /etc/audit/rules.d/ and verify with auditctl -l after reboot. Without -k identity on the rule, your parser cannot filter efficiently and every query becomes a full-file scan.

# watch the identity files for write + attribute change
-w /etc/passwd  -p wa -k identity
-w /etc/shadow  -p wa -k identity
-w /etc/sudoers -p wa -k identity

Where this goes next

Emit each parsed event as one JSON line and you have built the bridge to everything downstream — ship it to Loki or Elasticsearch and the same 'who touched /etc/shadow' question becomes a saved dashboard query across your whole fleet, not one host at a time. The Python for security automation course covers log parsing, APIs, and CLI tools with real error handling.

Related posts