The audit pipeline

Collect, normalize, and where it belongs.

Detecting everything above requires visibility, and visibility is a pipeline problem: collecting the right events on each host, normalizing them into a consistent shape, and getting them somewhere safe to search — all without drowning in volume or cost. The foundation is the audit data the previous course produced (auditd rules) plus richer sources you will add (eBPF, osquery), and the architecture decision is where each stage of collection, filtering, and analysis belongs.

Filter at the edge, not after ingest

The single most important cost-and-signal decision is where you filter. Shipping every raw event to a SIEM and filtering there is how security teams get six-figure ingest bills and drown in noise. Instead, drop the known-benign high-volume events at the host or a nearby aggregator before they are shipped, and normalize the rest into a consistent schema so detections are simple to write. Every raw auditd record you choose not to ship is money saved and one less thing burying the event that matters.

Ship off-host, because local logs lie

The recurring theme of this course — a compromised host’s local data cannot be trusted — makes off-host shipping non-negotiable for security telemetry. Events must reach a central store the attacker on that host cannot reach or edit, ideally in near-real-time so the record exists before an intruder thinks to cover their tracks. This is also what lets you correlate across the fleet (the same technique appearing on three hosts) and investigate a host that has since been wiped or is untrustworthy. The pipeline’s destination is a source of truth outside the blast radius.

# ship audit events off-host in near-real-time (audisp/rsyslog/vector → central store)
# /etc/audit/plugins.d/au-remote.conf  or a forwarder like Vector/Fluent Bit:
#   - read auditd + journald + eBPF events
#   - drop known-benign high-volume noise HERE (at the edge)
#   - normalize to a common schema, then forward to the SIEM/data lake