File integrity monitoring

AIDE: know when the system changes.

A museum guard walks the halls at closing time and photographs every painting in its frame. The next morning, before the doors open, the guard compares each frame against last night's photo. A canvas nudged a centimeter left, a smudge that was not there, a frame that hangs a little differently: anything that moved gets flagged before a single visitor walks in. File integrity monitoring (FIM, keeping watch on whether important files change) is that guard for your system.

Hardening and auditing tell you about access: who logged in, who ran which command. FIM answers a different question, whether the files themselves changed. The idea is small and it holds up. While the system is known-good, you record a fingerprint of every file that matters: the contents hashed, plus permissions, owner, group, size, and timestamps. Later, on a schedule, you fingerprint everything again and compare. A modified system binary, a stray file dropped into /usr/sbin, an /etc config that shifted overnight, all of it stands out against the record. On Linux the usual tool for this is AIDE (Advanced Intrusion Detection Environment).

A cryptographic hash is a short, fixed-length fingerprint computed from a file's bytes with an algorithm like SHA-256 (Secure Hash Algorithm, 256-bit output). Change one byte and the fingerprint changes completely, and you cannot run it backward to build a different file that produces the same fingerprint. That one-way property is what makes a baseline worth trusting. An attacker who swaps out a binary cannot make the replacement hash to the old value, so the swap shows up.

Take the baseline

Two files do the work. AIDE writes a freshly built database to /var/lib/aide/aide.db.new, and it reads a live baseline from /var/lib/aide/aide.db when it runs a check. Think of aide.db.new as the photo you developed today and aide.db as the print locked in the guard's album that tomorrow's comparison uses. You build the new one, look it over, then copy it into place as the baseline.

aideinit is the Debian and Ubuntu wrapper around aide --init. It walks every path in the config, hashes what it finds, and writes the result to aide.db.new. On a busy server that first pass records over a hundred thousand files and takes a minute or two. Nothing is a live baseline yet. You promote the new database into place with a copy:

What AIDE actually records

The config file, /etc/aide/aide.conf, is a list of rules: which paths to watch and which properties to record for each. You do not check the same thing everywhere. A system binary should never change its contents, so you hash it. A log file grows all day by design, so you allow its size to increase but still watch its owner and permissions. Here is a distilled config that shows the moving parts:

# /etc/aide/aide.conf  (distilled to show the moving parts)
 
# Where the baseline lives. database_in is read at check time;
# database_out is written at init/update time.
database_in  = file:/var/lib/aide/aide.db
database_out = file:/var/lib/aide/aide.db.new
 
# Attribute groups: which properties to record per file.
#   p permissions   u owner    g group      s size
#   m mtime   c ctime   i inode   n link count   b blocks
#   sha256 / sha512   content hashes
#   S  "size may grow" (append-only logs, not a full hash)
Binlib    = p+i+n+u+g+s+b+m+c+sha256+sha512
ConfFiles = p+u+g+s+m+c+sha256
GrowLog   = p+u+g+n+S
 
# Selection lines: path  then  group to apply.
/usr/bin    Binlib
/usr/sbin   Binlib
/bin        Binlib
/sbin       Binlib
/usr/lib    Binlib
/boot       Binlib
/etc        ConfFiles
/var/log    GrowLog
 
# Exclusions (leading !): things that churn by design.
!/var/log/journal/.*
!/var/cache/.*
!/var/spool/.*
!/etc/mtab$

Each letter is one property. p is permissions, u and g are owner and group, s is size, m and c are the modification and change timestamps, i is the inode (the on-disk record that holds a file's metadata), and sha256/sha512 are content hashes. A capital S means 'size is allowed to grow', which is how you watch a log without treating every new line as an alarm. The lines starting with ! are exclusions: logs, caches, and spool directories change every second, and watching them would drown a real alert in noise. On Debian and Ubuntu the real /etc/aide/aide.conf is assembled from snippets in /etc/aide/aide.conf.d/ by update-aide.conf, but the rules it produces read exactly like the ones above.

Catch the change

With a baseline in place, a check re-fingerprints everything and reports the differences. This is where the guard compares frames to photos.

Read this like a defender. Two things happened. A brand new file, /usr/local/bin/.sysupd, showed up where nothing was before, and its leading dot keeps it out of a plain ls. And /usr/bin/curl grew by 16 KB and its SHA-256 changed, so its actual bytes are different from the baseline. In the compact lines, the letters after the file-type flag tell you what moved: on curl you see s, m, c, and C, meaning its size, both of its timestamps, and its content hash all changed; the row of plus signs on the added file means every attribute is new, because the file did not exist when you took the baseline. The left column of the detailed block is the baseline, the right column is what sits on disk now.

That exit code is the part a machine can act on. AIDE returns a bitmask: 1 means new files were found, 2 means files were removed, 4 means files changed. Here 5 is 1 plus 4, so new files and changed files, nothing removed. Zero means the filesystem matched the baseline. This is the core reason FIM catches things a person cannot. Many rootkits (hidden toolkits an intruder installs) work by replacing ls, ps, and netstat with tampered copies that lie about what is on the box and which processes are running. Ask the tampered ls and it says the directory is empty. AIDE never asks it. AIDE reads the raw bytes of ls itself and hashes them, so the replacement shows up as a changed binary no matter how well it hides its own tracks.

The update problem

There is one honest complication. Package updates change real system binaries. Patch curl for a security fix and the next check will flag /usr/bin/curl and its library, correctly, because they genuinely changed. If you cannot tell an approved patch from an intruder, the tool is useless. So the workflow is: every time you make an intended change, verify the flagged files match what you did, then re-record the baseline. aide --update does the compare and the rebuild in one pass, writing a new database while it reports the diffs.

Notice what those two changed files are: the curl binary and the curl library, exactly what an upgrade of the curl package touches. They match the action you took, so you promote the new database and move on. This turns FIM into a ledger where every change has to be explainable. A flagged change that lines up with a patch, a config edit, or a deploy is expected. A flagged change to a system binary that nobody can account for is one of the strongest single signals of compromise you will get on a host.

Run it on a schedule, and alert on the exit code

A check you run by hand once a month catches almost nothing. The aide package already installs a daily job at /etc/cron.daily/aide (cron is the classic Unix job scheduler) that runs a check and emails the report, configured through /etc/default/aide. On a modern host you can drive it explicitly with a systemd timer instead (systemd is the service manager that starts and supervises everything on the box, and a timer is its built-in scheduler). Two small unit files do it:

[Unit]
Description=AIDE file integrity check
Documentation=man:aide(1)
# On a difference, aide exits non-zero and this unit is marked failed.
# Point OnFailure= at your own alerting unit to get paged.
# OnFailure=aide-alert.service
 
[Service]
Type=oneshot
Nice=19
IOSchedulingClass=idle
ExecStart=/usr/bin/aide --config /etc/aide/aide.conf --check

[Unit]
Description=Run the AIDE integrity check daily
 
[Timer]
OnCalendar=*-*-* 03:00:00
RandomizedDelaySec=15m
Persistent=true
 
[Install]
WantedBy=timers.target

That last command is your proof the schedule took: the timer exists and has a next run time. The design does real work here. Type=oneshot means the service runs the check to completion and exits. Because aide --check returns non-zero the moment it finds a difference, the unit is marked failed on any change, and a failed unit is a clean hook to alert on. Add OnFailure=your-alert.service and systemd fans that failure out to whatever pages you, so a changed binary becomes a notification instead of an email nobody reads. Nice and IOSchedulingClass keep the hashing pass from starving the rest of the machine at 3am.

Protect the baseline

The database is the one file an intruder most wants to edit. If they can quietly rewrite aide.db to hold their backdoor's current hash, every check comes back clean and your guard is now vouching for the break-in. A lock on the front door is worthless if the burglar can rewrite the guard's photo album. So the baseline, the config file, and the aide binary all need to live somewhere the attacker cannot reach. Keep a copy off the host, and record a separate hash of the database first so you can later prove the local copy was not swapped. The copy itself goes out with scp (secure copy, which moves a file over an encrypted network link):

FIM is detective, not preventive. It does not stop the change; it tells you after the fact that a change happened. Its whole value rests on the record being beyond the attacker's reach, the same reason serious detection work ships logs and evidence off the box: something a compromised host can rewrite is not evidence. Store the database and its hash on a system the web server has no credentials to touch, and compare against that trusted copy any time you doubt the local one.

Run your first real check the morning after you build the baseline, before anyone else has touched the box, and read it line by line even though it should be empty. A clean report you have looked at with your own eyes is the reference every later report is measured against, and knowing it was clean once is what lets tomorrow's single changed hash mean something.

Try this

Work through “Protect the baseline” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: baseline a clean host, or you baseline the break-in. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related