File integrity monitoring with AIDE

Baseline, detect change, update safely.

File integrity monitoring records what the important files on a server look like while it is known to be good, and later tells you exactly which ones changed. In this lesson you build an AIDE baseline on Ubuntu 26.04, catch a planned change and an unplanned one, decide which is which, accept the planned change with the update workflow, run the check on a schedule on Ubuntu and on RHEL 10, and keep the baseline somewhere an intruder cannot rewrite it. Logs and audit rules tell you who did what; integrity monitoring tells you what the files are now, even when nobody logged the change.

How AIDE works

AIDE (Advanced Intrusion Detection Environment) reads a configuration that says which paths to watch and which attributes to record for each: permissions, owner, group, size, inode, link count, timestamps, extended attributes and cryptographic hashes of the contents. aide --init walks those paths and writes a database. aide --check walks them again and reports every difference from the database. A cryptographic hash changes completely when one byte of the file changes, and nobody can practically craft a different file with the same hash, so a replaced binary cannot hide from a hash comparison.

Both platforms ship AIDE 0.19.2, with different packaging. On Ubuntu, aide plus aide-common bring a configuration that covers the whole filesystem (/etc/aide/aide.conf plus about 240 rule snippets in /etc/aide/aide.conf.d), the aideinit wrapper and a daily systemd timer. On RHEL, /etc/aide.conf watches /boot, /usr, /root and a list of /etc files, and nothing is scheduled. Installing aide on Ubuntu with its recommended packages also pulls in a mail program and a mail server (postfix); on a server that must not gain a listener, install with --no-install-recommends or check ss -tlnp afterwards. The lab machine has AIDE without a mail server.

Build the baseline

A baseline is only as trustworthy as the system it was taken from, so build it right after a clean installation or rebuild, before the server is exposed. On Ubuntu, aideinit runs aide --init with the packaged configuration and, with -y -f, installs the result as the live database without prompting. It logs the details to /var/log/aide/aideinit.log.

About two and a half minutes on this small, idle server, for about 160,000 entries; a large server takes much longer, and AIDE is I/O- and CPU-heavy while it runs, so schedule it and give it a low priority. The database is 37 MiB and readable only by root and by _aide, the system user Ubuntu's daily job runs as. aide.db is the baseline that checks compare against; aide.db.new is what the last run wrote.

Catch a change and triage it

Two things now happen on the server. An administrator installs a small package, a planned change. Someone also appends a line to an admin script in /usr/local/bin and drops a hidden file next to it, which nobody planned.

The first attempt to check fails with exit status 17, a configuration error. Ubuntu's aide has no built-in configuration path: the packaged wrappers pass --config /etc/aide/aide.conf, and so must you. (RHEL's aide reads /etc/aide.conf by default.)

Each summary line is a file type (f file, d directory) followed by one position per attribute, in the order YlZbpugamcinHAXSECF given in aide.conf(5). A + in every position means a new file and - a removed one. For a changed file, the third character (the size position, after the link-name position, which is blank for a regular file) is =, < or > for size unchanged, smaller or larger, and each letter that appears names an attribute that changed: m and c the modification and change times, i the inode, n the link count, H the content hashes. f >.... mc..H on the admin script means it grew and its contents changed. The detailed section shows old and new values side by side.

The exit status is a bit mask: 1 for added files, 2 for removed, 4 for changed, summed: 5 here means added and changed files and nothing removed, and 7 would mean all three. Ubuntu's configuration watches the whole filesystem, so even this short window produced noise: journal stream files under /run, apt's caches and the audit log. Tuning that noise with local rules in /etc/aide/aide.conf.d is part of adopting AIDE. The package database explains most of the rest.

dpkg -S knows /usr/bin/hello and nothing about the two files in /usr/local/bin, which is normal for local files but means only you can vouch for them. The script's new last line fetches and runs something from another host: this is the unplanned change, and on a real server it starts an incident, not an update. Keep the report, preserve the files and follow your incident process. Here the response is short: copy both files into a root-only evidence directory (with cp -a, which keeps their modes and times), restore the script's known-good content (on a real server, from configuration management or a backup; here that means deleting the appended line), and remove the hidden file.

ls -l does not list the hidden copy (ls -la would); the AIDE report further down shows both files in the evidence directory. The evidence copy belongs off the host as soon as your incident process allows; a copy under /root is only a first step. dpkg --verify (Ubuntu) and rpm -Va (RHEL) are useful second opinions for package-owned files, because they compare them with the package's own checksums.

The daily job, and accepting planned changes

Ubuntu's aide-common ships dailyaidecheck.timer, enabled when the package is installed. Its behaviour is set in /etc/default/aide.

The timer fires once a day between 01:50 and 03:50 (a random delay of up to two hours spreads the load across a fleet). COMMAND=update makes the job run aide --update, which checks against aide.db and also writes a fresh aide.db.new. COPYNEWDB=no means it never promotes that file, so every change is reported again each day until a person reviews it. After the response, run the job by hand:

The job succeeded, wrote its report to /var/log/aide/aide.log and a new aide.db.new, and warned that it could not mail the report, because this server has no mail program. The report still compares with the old baseline, so it lists the package and the system noise again, and the response itself shows up. Read the lines for the lab's paths: the hidden file is gone from the report; the script is listed with its inode and times changed but no H (sed -i wrote a new file with the old contents); and the evidence directory appears as added. Each of those is explained by something you did. By default the report is also mailed to root. Decide how it leaves the host: a mail client plus a mail server that listens only on localhost, or SILENTREPORTS=yes and /var/log/aide/aide.log collected by the log forwarding from the logging lesson. Once every line of the report is explained, promote the new database:

Promoting is a deliberate copy. Never promote without reading the report: whatever is on disk at that moment, a backdoor included, becomes the new normal and future checks stop reporting it. On any distribution the same step is aide --update followed by moving the new database into place. To stop Ubuntu's daily check, sudo systemctl disable --now dailyaidecheck.timer; to remove AIDE, sudo apt purge aide aide-common.

On RHEL 10

RHEL's configuration is narrower, the database is gzip-compressed, and /etc/aide.conf is readable by root only. There is no wrapper: aide --init, then rename the new database.

NORMAL is RHEL's rule for /usr and /boot: permissions, owner, inode, links, size, hashes and more, but not modification or change time (R+sha512-m-c), so a file that is only touched is not reported. 30 seconds for 56,000 entries, compared with two and a half minutes for 160,000 on Ubuntu's whole-filesystem configuration. Installing a package shows the RHEL report format:

The package's seven files are the planned change. The RPM database and the linker cache change with every package transaction, so expect them after each patch window. The report ends with the database's own hashes, which AIDE also prints at init time: record them off the host (below). RHEL ships no schedule. Red Hat's documentation suggests a daily cron entry at 04:05; a systemd timer does the same and turns a difference into a failed unit, because aide --check exits non-zero when it finds one. It also exits non-zero when the check itself breaks: aide(1) lists 1 to 7 for differences and 14 to 25 for errors (17 is the configuration error above), and an out-of-memory kill is a failure again. "Someone changed /usr/bin" and "the check did not run" must not look the same to whoever gets the alert, so the unit's comment says how to tell them apart.

[Unit]
Description=AIDE file integrity check
Documentation=man:aide(1)
 
[Service]
Type=oneshot
# aide --check exits 1-7 when it finds differences (1 added, 2 removed,
# 4 changed, summed) and 14-25 when the check itself failed. Both mark this
# unit failed; an OnFailure= alert unit tells them apart by ExecMainStatus.
ExecStart=/usr/sbin/aide --check
Nice=19
IOSchedulingClass=idle

[Unit]
Description=Daily AIDE file integrity check
 
[Timer]
OnCalendar=*-*-* 04:05:00
RandomizedDelaySec=30min
Persistent=true
 
[Install]
WantedBy=timers.target

Exit status 5 (the added and changed tree files) left the unit failed, which is the hook for alerting. systemctl show -p ExecMainStatus is what an alert unit reads: 1 to 7 means "review the report", anything else (or a Result such as signal or oom-kill) means "the check is broken, fix it today". Alert as well when the timer has not run for more than a day (systemctl list-timers shows its last run), because a check that silently stops running is exactly what an intruder wants. Note the memory peak: hashing every file with every compiled-in algorithm is not free, so on a small server add MemoryMax= to the service. Rolling back is systemctl disable --now on the timer and deleting the two unit files.

Protect the baseline

An intruder with root on the host can rewrite the database, the configuration or the aide binary, and every later check comes back clean. Red Hat's guidance is to keep the database, the configuration and the binary on read-only media. In practice: after each promotion, copy the database and its sha256sum (or the hashes AIDE prints for the database at the end of every run) to a system this server has no credentials for (pulled by that system, rather than pushed with a key stored here), and when you need a check you can trust, run it with the off-host copies. A hash stored on the same server proves nothing, because whoever can replace the database can replace the hash.

AIDE is not a Ubuntu default; Ubuntu packages it with a daily timer, and RHEL packages it without one. The CIS benchmarks for RHEL 10 and for the older Ubuntu 24.04 release (as encoded in the SCAP Security Guide content) require AIDE to be installed, a database built and a periodic check scheduled; the next lesson shows these rules failing on an untouched Rocky server. The ongoing cost is review: package updates make the report long, so plan to review and promote after every patch window.

Try this

On your Ubuntu lab machine, create a small script in /usr/local/bin and build a baseline with sudo aideinit -y -f. Append a comment line to the script, create a second, empty file next to it, and install a small package such as hello. Before running sudo aide --config /etc/aide/aide.conf --check; echo $?, predict the exit status and which summary letters the script's line will show; then compare. Use dpkg -S to sort the report into package-owned and local files, remove the empty file and the comment line, run sudo aide --config /etc/aide/aide.conf --update, read the report, and promote it with sudo cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db.

Takeaway

Build the baseline on a clean system, keep a copy and its hash off the host, and run the check daily with the result going somewhere a person will read. Promote a new database only after every changed line in the report is explained by a change you know about.

Related