The audit pipeline

Collect, normalize, and where it belongs.

A detection pipeline is a delivery route for evidence. Something happens on a machine (a file gets opened for writing, a program launches, an ordinary account turns into root), a sensor writes it down, and that note has to travel from the host to a place you can search. It has to arrive without being lost, altered, or buried under a million boring notes along the way. Get the route right and you can watch one attacker move across fifty servers from a single screen. Get it wrong and the one record that mattered is sitting on a disk the intruder already wiped.

Three jobs make up the route. Collect the events on each host. Normalize them into one shape, so a detection rule never has to know which sensor produced them. Ship them off the host to a store the person on that host cannot reach. Every real decision here is about where each job runs: on the host itself, on a nearby aggregator (one middle machine that gathers events from many hosts), or in the central system. That placement is what sets your bill and whether the evidence lives long enough to be useful.

Collect: auditd is the baseline, eBPF and osquery fill the gaps

auditd (the Linux Audit daemon, a background service that records security-relevant events straight from the kernel) is the sensor you already have. You feed it rules. Each rule can carry a key, a short label you make up, so you can pull those records back later by name.

When someone writes to that file, the kernel does not produce one record. It produces several that share a timestamp and a serial number: one for the system call, one for the file path, one for the working directory, one for the full command line. Read them back by key with ausearch.

Two fields carry the weight. syscall=257 is openat, the call that opened the file. auid=1000 is the audit login id, the account that first authenticated at the start of the session. That login id sticks to every later action even after the person runs sudo and becomes root, which is why the same record shows auid=1000 and uid=0 side by side. It names the human behind the action, even when that action ran as root. That density is also the problem. A machine, not a person, has to parse this, and every sensor writes its own dialect.

auditd is good at 'watch these files and these system calls', and it turns heavy and noisy the moment you ask it to watch everything. eBPF (extended Berkeley Packet Filter, a way to run small, safety-checked programs inside the kernel itself) is the richer sensor. It can see every process start with its full family tree at low cost. Falco and Tetragon are built on it. bpftrace lets you try it in one line.

auditd and eBPF are streams you tail as events happen. osquery is a ledger you interrogate. It presents the live state of the machine as database tables and lets you run SQL (Structured Query Language, the everyday language for asking a database questions) against them, which is the right tool for 'is anything wrong on this box right now'. This query finds running programs whose executable has been deleted from disk, a classic sign of malware that unlinks its own binary to hide.

Normalize: one shape for every sensor

You now have three sensors speaking three languages. auditd writes space-separated key=value text. The eBPF tools write their own JSON. osquery returns table rows. Your detection rules should not have to care about any of that. Normalizing means rewriting every event into one agreed schema before it goes anywhere, the way a mailroom drops every incoming letter into the same standard envelope with sender, recipient, and date in fixed spots. A common target is ECS (Elastic Common Schema, a published set of field names like process.pid, user.name, and event.action). The raw auditd record above becomes this.

{
  "@timestamp": "2026-07-13T14:22:31.123Z",
  "host":    { "name": "web-03" },
  "event":   { "kind": "event", "module": "auditd", "action": "passwd_change", "outcome": "success" },
  "process": { "pid": 1442, "ppid": 1201, "name": "vim", "executable": "/usr/bin/vim.basic" },
  "file":    { "path": "/etc/passwd" },
  "user":    { "id": "0", "name": "root", "audit": { "id": "1000" } }
}

A forwarder does this rewriting on or near the host. Vector (an open-source tool for building log and event pipelines) reads the raw log, parses each line into fields, keeps only what you asked for, and reshapes the rest before anything leaves the machine.

# /etc/vector/vector.yaml - read auditd, keep keyed records, reshape, ship
sources:
  auditd_log:
    type: file
    include: ["/var/log/audit/audit.log"]
    read_from: end
 
transforms:
  parse:
    type: remap
    inputs: ["auditd_log"]
    source: |
      .audit = parse_key_value!(.message, field_delimiter: " ", key_value_delimiter: "=")
 
  keep_keyed:                 # edge filter: only records that matched a keyed rule
    type: filter
    inputs: ["parse"]
    condition: '.audit.key != null && .audit.key != "(null)"'
 
  to_ecs:                     # reshape survivors into the common envelope
    type: remap
    inputs: ["keep_keyed"]
    source: |
      . = {
        "@timestamp": now(),
        "host":    { "name": get_hostname!() },
        "event":   { "kind": "event", "module": "auditd", "action": .audit.key },
        "process": { "name": .audit.comm, "executable": .audit.exe, "pid": .audit.pid },
        "user":    { "id": .audit.uid, "audit": { "id": .audit.auid } }
      }
 
sinks:
  aggregator:
    type: socket
    inputs: ["to_ecs"]
    mode: tcp
    address: "10.20.0.10:9000"
    encoding:
      codec: json

Three moves matter here. parse turns the flat text into named fields. keep_keyed throws away every record that did not match a keyed rule, right at the source. to_ecs reshapes the survivors into the shared envelope. Notice that the filter runs before the sink. That ordering is the whole game in the next section.

Filter at the edge, not after ingest

Filtering after events reach your central system is like paying to ship every piece of junk mail across the country and then binning it on arrival. You already paid the postage. A SIEM (Security Information and Event Management system, the central tool where analysts store and search logs) bills by volume, so the cheapest event is the one you dropped closest to where it was born. Before you decide what to drop, measure. auditd will tell you exactly where your volume comes from.

time-change fired 184,203 times, and almost none of it earns central storage. You have two good places to kill it: scope the rule so the kernel stops generating it, or drop it at the forwarder before it ships (the keep_keyed filter above does exactly that). Every raw record you choose not to send is money saved and one fewer thing burying the record that matters. Filtering at ingest instead is how teams end up with a six-figure bill, and then with retention cut so short they cannot investigate the breach they find two months late.

Ship off-host, because local logs lie

A logbook that sits in the same room as the person you are watching is a logbook they can rewrite. Once an attacker has root on a host, every file on that host is theirs, including your audit log and the timestamps on it. Off-host shipping means the record leaves the building before anyone on the host thinks to burn it. If auditd is your only sensor, its own remote plugin is the path with the fewest moving parts. It sends each record over TCP (Transmission Control Protocol, the network transport that resends anything dropped along the way), so a record either lands on the far end or the sender finds out it did not. Turn it on with two config files.

active = yes
direction = out
path = /usr/sbin/audisp-remote
type = always
format = string

remote_server = 10.20.0.10
port = 60
transport = tcp
network_failure_action = syslog

The aggregator is another host running auditd with tcp_listen_port = 60 set in /etc/audit/auditd.conf, so it accepts incoming records and writes them to its own log. Set name_format = hostname in each sender's own auditd.conf so every shipped record carries a node= field you can filter on. Then prove the whole route works instead of assuming it does. Change the watched file on the host, and go look for the record on the aggregator.

It arrived on the aggregator, tagged web-03. That is a working pipeline. If that record had shown up only in the host's local log and nowhere central, you would have shipped nothing, and you would have built a logbook the attacker fully controls.

Make that touch-and-check the last step of every change you make to a rule, a forwarder config, or a firewall between the host and the aggregator. A pipeline you have not watched a real event travel through end to end is a pipeline you are hoping works, and the morning you find out it does not will be the morning you needed it most.

Try this

Work through “Ship off-host, because local logs lie” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: only one program can own the audit stream. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related