The audit pipeline
From kernel events to a SIEM, with auid.
The Linux audit framework records security-relevant events inside the kernel: a file opened, a program run, a rule changed, a login. This lesson follows a record from the kernel to disk and off the host. You wrote rules in linux-hard/auditd, so the rules here are a quick recap; the new ground is detection engineering. You will attribute activity by two people and two services with auid, the login identity that survives sudo and that the rest of this course relies on, see what the kernel does when the audit daemon falls behind or is not running, and prove that records reach another consumer.
How an audit record travels
Three pieces cooperate. The kernel's audit subsystem checks its rules as each system call finishes and builds records. One user-space daemon, auditd, registers with the kernel over a netlink socket (a kernel-to-process message channel), receives the records and writes /var/log/audit/audit.log. Its built-in dispatcher hands each event to plugins that forward it elsewhere. Other readers get only a read-only multicast copy, which is what systemd-journald's audit socket uses.
Where auditd comes from differs by platform. A default Ubuntu Server 26.04 does not have it; the package is in the Ubuntu archive.
RHEL 10 installs the audit package and enables auditd at boot.
Its unit also refuses a manual stop, so systemctl restart auditd fails there; "auditd rules that matter" in Linux hardening shows the refusal and the two ways around it. You rarely need a restart on either platform: rules load with augenrules --load, and most auditd.conf settings reload with sudo auditctl --signal reload. The rest of the lab runs on Ubuntu with auditd installed (sudo apt-get install auditd).
Rules, keys and augenrules
The examples need something to watch: a small application's credentials file, readable by a developer group, and a developer account, ap-ana, that logs in over SSH with a key. Create them first.
The file is root:ap-dev, mode 640, and known_hosts holds this host's key so the SSH login below is accepted. Rules live in files under /etc/audit/rules.d/. Write this one with sudoedit: it watches the credentials file, one privileged program, everything the web server's account runs, and a starter set of persistence locations.
## Reads of the application credentials file-a always,exit -F arch=b64 -F path=/etc/ap-demo/db.env -F perm=r -F key=secrets_read## A privileged program run by someone who logged in (not by a service)-a always,exit -F arch=b64 -F path=/usr/bin/chage -F perm=x -F auid>=1000 -F auid!=unset -F key=privileged## Every program the web service account runs (what a web shell would do)-a always,exit -F arch=b64 -S execve -F uid=www-data -F key=svc_exec## Persistence locations, a starter set: cron, local systemd units, the preload file-a always,exit -F arch=b64 -F dir=/etc/cron.d -F perm=wa -F key=persist-a always,exit -F arch=b64 -F dir=/var/spool/cron -F perm=wa -F key=persist-a always,exit -F arch=b64 -F dir=/etc/systemd/system -F perm=wa -F key=persist-a always,exit -F arch=b64 -F path=/etc/ld.so.preload -F perm=wa -F key=persist
The grammar (-a always,exit, -F arch=b64, path= or dir= with perm=, -S, and key=, the label you search on) is the one "auditd rules that matter" in Linux hardening taught. Two rules are about identity. The chage rule's auid>=1000 auid!=unset limits it to people who logged in, so it never fires for a process a service started. That blind spot is where a compromised web application lives, so the svc_exec rule covers it from the other side: every execve by the web server's account (uid= accepts a name). That account runs few programs, and a shell or curl under it is the classic web-shell signal.
The persistence rules are a starter set, not a catalogue. /etc/ld.so.preload lists libraries the dynamic linker loads into every dynamically linked program, so a write to it is almost never legitimate. Left out, and worth adding where they matter to you: /etc/crontab and /etc/cron.{hourly,daily,weekly,monthly}, user systemd units (/etc/systemd/user, ~/.config/systemd/user, made persistent with loginctl enable-linger), /usr/lib/systemd/system, each account's ~/.ssh/authorized_keys (one rule per account you care about; all of /home is noisy), shell start-up files such as /etc/profile.d and ~/.bashrc, /etc/ld.so.conf.d, /etc/sudoers.d and /etc/pam.d. linux-hard/auditd adds the identity, sudoers and module rules.
Load it with augenrules, as in the hardening lesson, which also shows how the merge works and how auditctl -l expands each rule.
auid: the person behind the action
Every process carries a login UID (loginuid; auid in audit records) and a session ID. pam_loginuid sets both once, when a person logs in: on Ubuntu it is in the PAM stacks for sshd, login and cron. Children inherit them, and sudo or su do not change them. Processes that systemd starts for a service never had a login, so theirs are unset (4294967295). Compare a login shell, the same shell through sudo, and a process started as a service.
deploy is 1001 in session 90, and sudo keeps both. The systemd-run process belongs to no login at all. Now create some ordinary activity. ap-ana logs in over SSH, reads the credentials file and runs chage to check her password age. Then deploy reads the same file through sudo, a service reads it, deploy drops a placeholder file into /etc/cron.d, and a process runs as www-data, standing in for a web application.
ap-ana's SSH login got its own loginuid (1002) and a new session (91). Ask ausearch for the secrets_read events in plain sentences. -ts sets the start of the search window; here it is the moment the rules were loaded (11:03:07), and -ts recent would mean the last ten minutes.
Each line is one event. "ap-ana successfully opened-file" is her own read. "deploy, acting as root" is the sudo read: the record's auid is deploy and its uid is root. "system, acting as root" is the service read, whose auid is unset. The first line shows that loading the rule was itself recorded, so someone deleting your rules leaves a trace too. That pairing of auid and uid is how you turn a root action into a named person. One caveat: loginuid_immutable 0 unlocked in auditctl -s (next section) means a root process may still change its own loginuid; --loginuid-immutable in a rules file makes it write-once, which auditctl(8) warns can trouble some container setups.
Reading records: ausearch and aureport
An event is several records sharing one timestamp and serial number. Here is the SYSCALL record of deploy's sudo read in its stored form. --input-logs makes ausearch read the log files when its input is not a terminal, as in a script or this lab; typed at a shell you can leave it out.
msg=audit(1790506988.633:16132) is the time in seconds since 1970 and the event's serial number. arch=c00000b7 is the audit constant for aarch64 and syscall=56 is openat there; an x86_64 server records arch=c000003e and syscall=257 for the same call, so never hard-code numbers from one architecture. exit=3 is the file descriptor returned. auid=1001 uid=0 euid=0 is deploy acting as root, ses=90 ties it to that login, and exe= is the uutils path Ubuntu 26.04 uses for head. -i translates the numbers into names, as in the persistence write.
The PATH record with nametype=CREATE names the new file, proctitle shows the full command (tee /etc/cron.d/ap-demo), and the SYSCALL record shows O_WRONLY|O_CREAT with auid=deploy uid=root. The privileged key caught only a person.
ap-ana's chage run matched because her auid is 1002; the same program launched by a service would not, because of auid!=unset. The svc_exec line is the web-shell view: "system, acting as www-data", a program run by the service account with no login behind it. aureport --summary counts events per key, and those counts include the rule-load records (four persist rules plus one write makes 5; one svc_exec rule plus one run makes 2). Measuring volume per key like this is how you find the rule that floods your pipeline before you ship anything.
Backlog, failure, and the gap when auditd is down
Records wait in a kernel queue until auditd reads them. auditctl -s shows that queue and what happens when it overflows.
backlog_limit 8192 comes from Ubuntu's base rules file (-b 8192; the kernel's own default is 64) and backlog 0 is the current queue, empty at that moment. When the queue is full, the process whose system call produced the record is made to wait, up to backlog_wait_time, for auditd to catch up; only then is the record dropped and lost increased. That wait is counted in kernel ticks (jiffies): auditctl(8) gives the default as 60*HZ, and this kernel runs at 1000 ticks a second.
So 60000 is a minute. The waiting process is your application: when auditd cannot keep up (a slow or full disk, a starved daemon, a rule that matches every execve), every program whose calls match a rule stalls, and the host looks hung. backlog_wait_time_actual totals that waiting and belongs next to lost on your dashboard: one rising means applications are slowed, the other that records are gone. A larger -b and fewer noisy rules avoid both; a shorter --backlog_wait_time turns stalls into drops sooner.
failure decides what the kernel does about a lost record. 1 (printk, the default and Ubuntu's setting) writes a message to the kernel log and carries on; 0 says nothing; 2 panics the machine. auditctl(8) suggests 2 for secure environments, which means a backlog overflow, even one caused by a stuck auditd, takes the host down. Use it only where policy says a host must stop rather than run unrecorded; everywhere else keep 1 and alert on lost. At the disk, disk_full_action = SUSPEND makes auditd stop writing but keep running, so events are lost to disk until space returns. q_depth is the dispatcher queue for plugins, and overflow_action what happens when a plugin cannot keep up.
The journal is not a second copy. journald can read the kernel's read-only audit multicast group through systemd-journald-audit.socket, but the unit is disabled on both platforms.
What does reach the journal is the kernel log. Stop auditd, read the watched file, and look.
With no daemon registered, the kernel printed the record into its log (rate-limited), where journalctl -k shows it. Record 16250 is not in audit.log after auditd came back (serial numbers restart at every boot, so the search starts at the rules-load time): it was printed and dropped. The kernel's audit=1 boot parameter (linux-hard/auditd) changes that, holding up to audit_backlog_limit records in memory until auditd starts; this host was booted without it (grep found nothing and exited 1). A default Ubuntu server, with no auditd at all, works this way all the time: AppArmor's messages, in audit format, live in the kernel log.
Forwarding records off the host
An attacker who gains root can edit a log on the same host, so records should leave it quickly. There are three common routes.
The first is auditd to auditd. The audisp-remote plugin (package audispd-plugins on both platforms) sends events to an auditd on a collector that listens with tcp_listen_port in its auditd.conf. Its transport = TCP is clear text, so use its Kerberos option or an encrypted network path. Set name_format = hostname on each sender so every record carries a node= field.
The second is a local plugin, af_unix or syslog, feeding a log shipper or rsyslog's forwarding (linux-hard/logging), which brings that tool's TLS. The third is a shipper that reads audit.log directly. Whatever you choose, prove it with a canary. Enable the af_unix plugin, which publishes every event on a Unix socket for a local reader.
auditctl --signal reload re-read the plugin configuration and started the plugin, which created a socket only root can read. auditctl -m injects a user message into the audit stream, a harmless canary; nc -dU reads the socket like a shipper would.
The canary came out of the plugin as a USER record with auid=1001, followed by the names auditd resolved when it received the event (Ubuntu's log_format = ENRICHED). auditd's state report shows the plugin and its queue.
One active plugin, a queue of 2000 (q_depth) and no overflow. On a real pipeline, run the same canary after every change to rules, plugin, shipper or firewall, and look for it at the collector, not on the host. To switch the plugin off again, set active = no and reload.
audit.log: a record dropped from a full backlog, one produced while auditd was stopped, one a stuck plugin never sent. Alert on lost and backlog_wait_time_actual rising in auditctl -s, on plugin queue overflow detected in the state report, and on a canary that does not arrive, rather than on the absence of events.Try this
With the demo file, account and rules file above loaded, run echo test | sudo tee /etc/cron.d/audit-test, then sudo systemd-run --wait --quiet touch /etc/cron.d/audit-test2. sudo ausearch -k persist -ts recent --format text should show one line with "your-account, acting as root" and one with "system, acting as root". Stop auditd (sudo systemctl stop auditd; on RHEL, sudo service auditd stop), run sudo cat /etc/ap-demo/db.env, start auditd, and confirm the read is in sudo journalctl -k -g secrets_read but not in sudo ausearch -k secrets_read -ts recent. Clean up: remove the test files, the rules file and /etc/ap-demo, run sudo userdel -r ap-ana and sudo groupdel ap-dev, then sudo augenrules --load; sudo auditctl -l should print "No rules".
Takeaway
Keep rules in rules.d with keys, load them with augenrules, and read auid before uid in every record. Watch the lost and backlog_wait_time_actual counters and send a canary through the whole pipeline after every change, because lost records leave no gap you can see in audit.log.
persist key shows a tee process with auid=unset uid=root creating a file in /etc/cron.d. What does auid=unset tell you?audit=1. Your rules stay loaded. What happened to the events those rules matched while auditd was down?