auditd rules that matter

Watch identity, privilege and module events.

The Linux audit system records security events from inside the kernel: which account changed /etc/shadow, who ran sudo, what loaded code into the kernel. A program that has been taken over can lie in its own log; it cannot stop the kernel from reporting the system calls it made, as long as a rule asks for them. In this lesson you install auditd where it is missing, write a short ruleset for the events worth recording, load it the way the system does at boot, prove that every rule fires, decide what happens when the audit log fills up, and weigh locking the rules with -e 2. Building detections on these records, and the login UID in depth, belong to the audit pipeline lesson in linux-det.

Installing auditd, and what it records by default

Ubuntu Server does not install auditd. On an untouched 26.04 install the unit does not exist, and a simulated installation shows the three packages it would bring:

sudo apt install auditd installs and starts it. RHEL 10 installs and enables auditd by default (audit 4.0.3; Ubuntu 26.04 ships 4.1.2). The lab machine already has it:

Two units do the work in audit 4. auditd.service is the daemon that writes records to /var/log/audit/audit.log. audit-rules.service loads the rules at boot by running augenrules --load. The packaged rules file sets only a few parameters, so the kernel holds no rules yet.

auditctl -s reads the kernel's side of the audit system. enabled 1 means auditing is on and the rules can still be changed. backlog_limit 8192 is the queue of records waiting for auditd to collect them. backlog_wait_time 60000 says what happens when that queue is full: a process that causes a record waits for room. auditctl(8) gives the kernel default as 60*HZ, so the number counts kernel clock ticks: 60000 is 60 seconds on Ubuntu's kernel (CONFIG_HZ=1000) and ten minutes on RHEL 10's (CONFIG_HZ=100), which ships the same line. Only after that wait does failure 1 apply: the kernel gives up on the record, logs the problem to the kernel log and carries on, and lost counts the records dropped that way.

The wait is the part that hurts. If auditd stalls (a slow or full disk, or a plugin stuck behind a log forwarder), every process that triggers a rule waits on each audited call, and the host slows to a crawl before any record is lost. Watch backlog, backlog_wait_time_actual and lost together, keep rules narrow, and on latency-sensitive hosts consider a shorter --backlog_wait_time in the rules file. Without rules auditd still receives the events that programs send on their own, such as logins and sudo sessions reported through PAM, but nothing about files or system calls.

Rules worth having

A rule has one of two shapes. A filesystem rule names a file with -F path= or a directory tree with -F dir=, plus the kinds of access to record with -F perm=: r read, w write, x execute, a attribute change such as chmod or chown. A system call rule names the calls with -S.

Both shapes start with -a always,exit, which records the event when the call returns, so the record says whether it succeeded. Both carry -F key=, a label you search by later. -F arch=b64 selects 64-bit programs: system call numbers differ between architectures and between 64- and 32-bit programs, and b64 means aarch64 on this lab machine and x86_64 on most servers. The older -w path -p wa form still works, but auditctl(8) in audit 4 marks it deprecated because of its performance; this course writes every rule in the -F path=/-F dir= form.

The package ships sample rules to borrow from, including the DISA STIG set (30-stig.rules), module loading (43-module-load.rules) and the -e 2 lock (99-finalize.rules), in /usr/share/doc/auditd/examples/audit-rules/ on Ubuntu and /usr/share/audit-rules/ on RHEL. On x86_64, where 32-bit programs can still run, either pair every system call rule with an arch=b32 twin or load 21-no32bit.rules, which records any 32-bit system call at all.

One path needs care before you copy anything:

On Ubuntu 26.04 /usr/bin/sudo is a symbolic link, through the alternatives system, to sudo-rs in /usr/lib/cargo/bin/sudo. When a program runs, the kernel records the file it actually executes, so a rule must name the resolved path. (/usr/bin/su is a real file.) This is the rules file:

## SecOpsLog: identity, privilege and kernel-module events.
## Syscall form with -F path= (auditctl(8) deprecates the -w form).
## arch=b64 means 64-bit programs on both x86_64 and aarch64.
 
## Accounts, password hashes and groups
-a always,exit -F arch=b64 -F path=/etc/passwd -F perm=wa -F key=identity
-a always,exit -F arch=b64 -F path=/etc/shadow -F perm=wa -F key=identity
-a always,exit -F arch=b64 -F path=/etc/group -F perm=wa -F key=identity
-a always,exit -F arch=b64 -F path=/etc/gshadow -F perm=wa -F key=identity
 
## Who may become root (sudo-rs reads /etc/sudoers-rs first if it exists)
-a always,exit -F arch=b64 -F path=/etc/sudoers -F perm=wa -F key=sudoers
-a always,exit -F arch=b64 -F path=/etc/sudoers-rs -F perm=wa -F key=sudoers
-a always,exit -F arch=b64 -F dir=/etc/sudoers.d/ -F perm=wa -F key=sudoers
 
## sudo and su started by a person who logged in (auid 1000 and up)
-a always,exit -F arch=b64 -F path=/usr/lib/cargo/bin/sudo -F perm=x -F auid>=1000 -F auid!=unset -F key=priv
-a always,exit -F arch=b64 -F path=/usr/bin/su -F perm=x -F auid>=1000 -F auid!=unset -F key=priv
 
## Kernel modules loaded or removed, by anyone
-a always,exit -F arch=b64 -S init_module,finit_module,delete_module -F key=modules

Each block answers one threat. An intruder who wants to come back adds an account, a password hash or a group membership (identity), or drops a line into /etc/sudoers.d (sudoers). priv records every start of sudo and su. Its filter uses the audit UID, auid: the account a person logged in as, set at login and kept through sudo and su, so a record of root's actions still names the person. auid>=1000 -F auid!=unset limits the rule to people; services and boot-time processes have no login and their auid is unset. The module rule has no such filter on purpose, because a module load is worth recording whoever triggers it: a kernel module runs with full kernel privileges and is where a rootkit hides. Comments are lines starting with #; augenrules strips them.

--check compares the files in rules.d with the compiled /etc/audit/audit.rules without changing anything. --load merges every *.rules file in version order, writes audit.rules and loads it: the "No rules" line comes from -D clearing the old set, and a status block is printed for each setting line (-b, -f, --backlog_wait_time). auditctl -l shows what the kernel really holds, and it does not look like the file. audit 4 expands each -F path= or -F dir= rule into the list of system calls that can write to or change the attributes of a file, and prints auid!=unset as auid!=-1. Rules added with auditctl alone are lost at the next load or reboot; only files in rules.d persist.

Prove that each rule fires

A rule you have never seen fire is a guess. Trigger each one on purpose. Here deploy adds a user and drops an empty file into /etc/sudoers.d:

ausearch -k finds events by key, -ts recent limits them to the last ten minutes, and --format text turns each event into a sentence. --input-logs makes ausearch read the log files even when it does not run in a terminal (a script, cron, or this lab); at an interactive prompt it does that anyway. "deploy, acting as root" is the pair that matters: the auid is deploy, the effective user root. useradd writes each file under a new name and renames it into place, which is why the events say "renamed". The full record shows every field:

-sc openat picks the event where install created the file. Read it from the bottom. The SYSCALL record says what happened: syscall=openat with O_CREAT, success=yes, auid=deploy uid=root, the program (exe is the uutils install on Ubuntu 26.04) and the key. The PATH records name the directory and the new file, and PROCTITLE has the full command line. -i translated numbers into names; the raw record stores arch=c00000b7 (aarch64) and syscall=56, the aarch64 number for openat. An x86_64 server records arch=c000003e and different numbers, which is why rules name the architecture.

Every sudo in this lab, including the ones that ran ausearch, left a priv event. The text summary names the path the user typed, /usr/bin/sudo, while the rule matched the binary it resolves to. The module summary describes both events as "loaded-kernel-module"; the records' syscall fields say finit_module and delete_module. Piping raw events from ausearch into aureport -k --summary counts them by key, a quick way to spot a key that suddenly fires hundreds of times. -ul deploy (login UID) keeps only events from deploy's logins.

Now the common failure. Older guides watch /usr/bin/sudo with -w /usr/bin/sudo -p x. Add that watch at runtime with auditctl (runtime rules are lost at the next augenrules --load or reboot), run sudo, and search:

auditctl warns that old-style watches are slower, loads the watch, and it never fires. A watch attaches to the file at that path, here the symbolic link, while execve executes the file the link resolves to, and that is the object the kernel records. Rules on commands must name the real file, so check readlink -f before you write them, and prove each rule with a test run as above. The same trap applies to any command managed by alternatives.

When the log fills up, and locking the rules

The records land in /var/log/audit/audit.log, readable by adm on Ubuntu (log_group) and by root only on RHEL. By default auditd keeps five files of 8 MiB and deletes the oldest when it rotates (ROTATE), so a busy server may hold only hours of history. When free space on the filesystem drops below 75 MiB it warns through syslog, and below 50 MiB, or when the disk is full, it suspends writing: the daemon keeps running, but new records are not written. The CIS RHEL 10 Level 2 server profile goes the other way: keep_logs (never delete) and halt or single when space runs out, trading availability for a complete record. SecOpsLog advice: ship records off the host (set active = yes in /etc/audit/plugins.d/syslog.conf to hand them to the forwarding from the previous lesson, or use a dedicated collector), give /var/log/audit its own filesystem with room for the retention you need, and choose halt only where policy says a missing record is worse than downtime. Changes to auditd.conf take effect with sudo auditctl --signal reload.

-e 2 makes the rules immutable until the next reboot. auditctl(8) says any attempt to change the configuration is then "audited and denied": augenrules --load exits with "Audit system is in immutable mode - exiting with no changes", and auditctl refuses with "The audit system is in immutable mode, no rule changes allowed". That stops an intruder with root from quietly removing the rule that would record them, and it is required by the CIS RHEL 10 Level 2 server profile (audit_rules_immutable). The cost is that every rule change needs a reboot, and so does the rollback of a bad rule. SecOpsLog advice: run a new ruleset without -e 2 until you have watched each rule fire, then add 99-finalize.rules with the line uncommented, and schedule rule changes with reboots. The lab machines cannot be rebooted, so the lock is not demonstrated here.

To roll back the whole ruleset while it is not locked, delete the file and load again:

On RHEL 10

RHEL's unit sets RefuseManualStop=yes, so systemctl restart auditd is refused (exit status 4); on Ubuntu it works and also reruns audit-rules.service. On RHEL reconfigure the daemon with auditctl --signal reload, which records a DAEMON_CONFIG event naming who did it. The older service auditd restart still works there: the audit package ships a legacy-actions script that stops the daemon with a signal and starts it again through systemctl, which also reloads the rules. Rule files, augenrules and ausearch are the same; on RHEL /usr/bin/sudo is the real classic sudo binary, so the priv rule names that path.

Try this

On your Ubuntu lab machine, install auditd, create the rules file above and load it with sudo augenrules --load. Add a user, drop a file into /etc/sudoers.d, load and unload the tcp_bbr module, and find each event with sudo ausearch --input-logs -k KEY -ts recent --format text. Then change the priv rule to watch /usr/bin/sudo instead of the resolved path, reload, run sudo true, and confirm that the key finds nothing. Put the correct path back and confirm the event appears.

Takeaway

Record the few events an intruder cannot avoid (identity files, sudoers, privileged programs by their real path, module loads), prove each rule fires before you rely on it, and ship the records off the host. Lock the rules with -e 2 only once you can schedule a reboot for every change.

Related