Linux forensic artefacts

Timestamps, logins, logs and shell history.

Once the volatile state is captured (linux-det/triage), the disk and the logs tell you what happened and when. This lesson reconstructs one small, real sequence from the traces it left on an Ubuntu 26.04 server, with the Rocky Linux 10.2 differences alongside. In the lab, a throwaway account det-art-sam failed one password over SSH, logged in with a key a few seconds later, created a file, appended to it, backdated it with touch, and ran a few commands before logging out. By the end you can read a file's four timestamps and spot a backdated one, find a login in the authentication log and the journal, know which login records each platform still keeps, read shell history knowing what it leaves out, and merge the sources into one timeline sorted by time.

Four timestamps, and which ones touch can move

Every file carries four timestamps. The modify time (mtime) moves when the contents change. The access time (atime) moves when the contents are read, within limits set by the mount options. The change time (ctime) moves whenever the inode changes: permissions, owner, size, link count, or any write to the timestamps themselves. The birth time (btime) is set once, when the file is created. ls -l shows only the modify time, which is the one an intruder is most likely to have faked.

ls -l says January 2025. stat tells the rest. Access and Modify both read 2025-01-15 09:22:00.000000000; the run of zero nanoseconds is itself a hint, because a program that writes a file almost never lands on a whole second. Birth says the file was created today at 11:05:14, and Change says something rewrote its inode four seconds later. That is exactly what touch -d "2025-01-15 09:22" did. touch calls utimensat(2), which lets the file's owner (or root) set atime and mtime to any value, and the kernel sets ctime to the current time as a side effect. No system call sets the birth time; the kernel only reports it, through statx(2). So an mtime older than the birth time marks a file whose times were set on purpose. Unpacking an archive, cp -p and rsync -t produce the same pattern legitimately, so treat it as a lead and use the ctime to find the moment in the logs and the shell history.

ctime and birth raise the cost of backdating without making it impossible: root can step the system clock back around an edit, or edit the inode directly while the filesystem is unmounted. The Rocky lab shows the same four fields on XFS, RHEL's default root filesystem. Access time is the weakest of the four, and you can damage it yourself.

The filesystem is mounted relatime, the kernel default since Linux 2.6.30 on both platforms. Under relatime the kernel updates atime on a read only when the old atime is not newer than mtime or ctime, or is more than a day old. The backdated atime met both conditions, so the investigator's own cat moved it from 2025 to now, and the 2025 value is gone. Run stat before you open anything, and when atime matters, work on a copy or a read-only image.

Searching a time window

With an anchor time (the first alert, or the first suspicious log line), ask the filesystem what changed after it. find -newermt compares modify times, -newerct compares change times. Here the anchor is 11:05:05, just before the failed password.

The modify-time search misses det-art-notes.txt, because its mtime claims 2025. The change-time search finds it, because touch could not move ctime. It also lists .bash_history and .cache/motd.legal-displayed, which Ubuntu's pam_motd creates at an account's first interactive login. On a real host, widen the search to each filesystem (find / -xdev ...) and expect noise from logs and caches; ctime also moves on a chmod or a rename, so confirm each hit. Birth time is not searchable this way: GNU find on Ubuntu 26.04 reports that it has no way to read birth times and rejects -newerBt, while stat can, because it asks the kernel through statx.

The login in the logs

The lab host refuses passwords on port 22: sshd -T prints the effective setting, and the Ubuntu cloud image's own drop-in sets it.

On the Rocky lab, cloud-init writes the same line into /etc/ssh/sshd_config.d/50-cloud-init.conf. For the demonstration the logins went to a second sshd on 127.0.0.1:2240 that accepts passwords; its log lines have the same form. On Ubuntu, rsyslog writes authentication events to /var/log/auth.log with RFC 3339 timestamps, readable by the adm group (deploy is a member, so no sudo). Because RFC 3339 timestamps sort as plain text, awk can select everything after the anchor with a string comparison, as long as every line carries the same UTC offset (check that before you trust a string comparison on a host whose time zone may have changed).

Read it in order. pam_unix(sshd:auth): authentication failure is PAM rejecting a password for user=det-art-sam from rhost=127.0.0.1; Failed password is sshd's own record of the same attempt, and Connection closed ... [preauth] means the client gave up before authenticating. Three seconds later Accepted publickey records the key's SHA256 fingerprint, which you can match against the account's authorized_keys with ssh-keygen -lf. session opened and systemd-logind's New session ... type 'tty' mark an interactive terminal session (the second, manager session is the user's own systemd instance). Disconnected and session closed end it seventeen seconds later. Two process IDs appear: the privileged monitor logs authentication and PAM, its unprivileged child logs the disconnect, so tie lines together by the client port (37296) rather than by PID alone.

The journal holds the same events, persistently on both platforms, and journalctl -t sshd-session selects them with no text parsing. auth.log and the journal are separate copies, so a careless edit to one shows up as a disagreement with the other. Agreement proves less: on Ubuntu rsyslog receives these lines from journald, and root can rewrite auth.log and rotate or vacuum the journal in the same minute. Only a copy already off the host settles it. RHEL writes the same lines to /var/log/secure, readable by root only, in the classic syslog format with no year and no time zone.

Login records: what each platform still keeps

Traditionally four binary files recorded logins: wtmp (every login, logout and boot, read by last), btmp (failed logins, read by lastb), utmp (who is logged in now, read by who) and lastlog (each account's latest login). Their fixed 32-bit time fields run out in 2038, and Debian, and Ubuntu with it, stopped shipping the readers.

last is not installed on Ubuntu 26.04. The util-linux changelog says why: last moved to the wtmpdb package, which is not installed by default and records logins through its own PAM module once installed, and lastb is gone, with syslog and the journal named as the place for failed attempts. The files tell the rest. There is no /run/utmp, so who prints nothing; loginctl list-sessions shows current sessions from systemd-logind instead. wtmp and lastlog were still written at the login time, and lslogins from util-linux reads them: last login on pts/0 from 127.0.0.1. btmp is empty. The failed password never reached it, so on this server the authentication log and the journal are the only record of failures. RHEL 10 keeps the classic tools, and its sshd records the failure.

last shows the session on pts/0 from 127.0.0.1 and lastb shows the failed attempt as ssh:notty. Two options matter for evidence: without -w both tools truncate long user names, and --time-format iso prints seconds and the time zone instead of minutes. All of these files are plain files that root can edit or truncate, so treat them as corroboration for the logs, not as proof on their own.

Shell history, and what it leaves out

These are the eight lines det-art-sam typed in the key session, one every two seconds. The fourth starts with a space, and id was typed twice.

echo "first note" > det-art-notes.txt
echo "second note" >> det-art-notes.txt
touch -d "2025-01-15 09:22" det-art-notes.txt
 cat /etc/hostname
id
id
uname -r
exit

The history holds six of them. linux-ess/shelltips covered why: Ubuntu's default ~/.bashrc sets HISTCONTROL=ignoreboth, which drops lines that begin with a space and a line identical to the one before, and RHEL's /etc/profile sets ignoredups only, so the Rocky history keeps the leading-space line. What matters for evidence is the rest. There are no timestamps; bash writes them only when HISTTIMEFORMAT was set in the shell that saved the file. Bash saves history when the shell exits, so the file's change time marks the end of the session, and the order of lines is the order of commands.

History is also the easiest trace to suppress. A shell killed with SIGKILL saves nothing, and unset HISTFILE or a ~/.bash_history linked to /dev/null silence every later session. An account that logged in interactively but has no history, or a history file whose change time does not match the session end in the log, is a finding to record, not an absence of evidence.

One timeline

Each source answers part of the question. Merged and sorted, they answer it together. The journal already prints ISO timestamps; stat --printf prints each changed file's birth (%w) and change (%z) time, and the first sed puts a T between date and time so the two formats sort together. The last sed trims each line to the time of day for reading.

The story reads top to bottom: the failed password at 11:05:08, the key login at 11:05:11 with its terminal session, the notes file born at 11:05:14 and re-stamped at 11:05:18, and the history file written at 11:05:28, the same second as the disconnect. The history fills the gap between birth and change: the file was created, appended to, then backdated. Tag each line with where it came from and how an intruder with root could have altered it, because every source here lives on the host under investigation. The journal can at least prove that its own files were not edited, if you set it up to.

journalctl --verify passed, but there is no fss sealing key in the journal directory, so the check covered internal consistency only: the file is well formed, not unedited. Forward Secure Sealing is set up with journalctl --setup-keys, which stores a sealing key on the host and prints a verification key that you keep somewhere else. With Seal=yes in journald.conf (the default once a key exists), persistent journal files are sealed at a fixed interval, 15 minutes by default, and journalctl --verify --verify-key=... then detects alteration of sealed data. It cannot stop root from deleting the files, and entries in the current interval are not yet sealed. Neither platform sets up a key by default. Forwarding logs off the host as they are written (linux-hard/logging) remains the stronger control.

Try this

On a lab host, run date -u '+%F %T' and note the time. Create a throwaway user with a key: sudo useradd -m -s /bin/bash art-try, ssh-keygen -t ed25519 -N '' -f ~/art-try-key -q, sudo install -d -m 700 -o art-try -g art-try ~art-try/.ssh and sudo install -m 600 -o art-try -g art-try ~/art-try-key.pub ~art-try/.ssh/authorized_keys. On a host hardened with AllowGroups or AllowUsers (linux-hard/ssh), sshd refuses this account until you add it to an allowed group for the exercise. Log in with ssh -t -i ~/art-try-key art-try@localhost, run echo test > probe.txt, then touch -d "2024-06-01 12:00" probe.txt, then id with a leading space, then exit. Now reconstruct it: sudo stat ~art-try/probe.txt should show Modify on 2024-06-01 with Change and Birth from the last few minutes; sudo find ~art-try -newermt "<noted time>" should miss probe.txt while -newerct finds it; journalctl -t sshd-session --since "<noted time>" should show Accepted publickey and session opened for the user; and sudo cat ~art-try/.bash_history should show the echo and touch lines but not the leading-space id on Ubuntu (on RHEL it appears). Remove the user with sudo userdel -r art-try and delete the key pair when you are done.

Takeaway

Anchor on one fact and prefer the evidence an intruder cannot simply set: ctime and birth over mtime and atime, the authentication log and journal over login records, and a copy that has left the host over anything still on it.

Related