Linux forensic artifacts
Where the evidence actually lives.
A break-in leaves marks in more than one place. The jimmied window, the muddy print on the carpet, the drawer left hanging open, the sticky note where the burglar wrote down the safe combination. A Linux host is the same. After you have captured the volatile state (the contents of memory, the live network connections, the running processes), the disk still holds a detailed account of what happened, scattered across a handful of very different places. The skill is knowing which place answers which question, then reading them in the right order. The logs record who came in and what the system did. The file timestamps record when things changed. The persistence spots record how the intruder planned to get back in. The per-user traces record what they typed. No single source is the whole story, and the timeline is the thread that ties them together.
Every file wears three clocks, and has a birthday
A library book carries a few different stamps. There is the date it was last checked out (someone touched it), the date someone last wrote inside it (its contents changed), and a separate record the librarian keeps of when its catalog card was edited (its metadata changed). And there is the day it first joined the collection. A file on Linux carries the same four marks. The modify time (mtime) moves when the file's contents change. The access time (atime) moves when it is read. The change time (ctime) moves when the inode changes (the inode is the filesystem's internal record card for a file, holding its permissions, owner, size, and pointers to the data). The first three are the classic MAC times (modify, access, change). The fourth is the birth time (btime, also called crtime), the moment the file was created. The stat command shows all four.
Attackers know you build timelines from these, so they timestomp: they backdate a file to blend in. The usual move is touch -t 202501150922 file, or touch -r /bin/ls file to copy an old timestamp from a real system binary. Here is the catch. touch only sets the access and modify times. It cannot set the change time, because ctime updates to the exact moment any metadata edit happens, including the touch itself. It cannot set the birth time either. So in the output above, Modify says January, but Change and Birth both say 03:14 on July 3. That mismatch, an old modify time sitting right next to a very recent change time, is the fingerprint of a backdated file. If your kernel is old enough that stat prints Birth as a dash, read the creation time straight from the inode with debugfs -R 'stat <262147>' /dev/sda1. The discipline is to compare all four, and to never trust ls -l (which shows only mtime) on its own.
Anchor on one fact, then pivot
Timelining sounds grand, but the method is easy to state. Start from the one thing you are sure of, then fan outward. Say the network sensor flagged a login at 03:14. That is your anchor. Now ask the disk what else happened in that window: which files were born or changed between 03:10 and 03:20? find can pull exactly that slice.
Every line there is a lead: a new binary in /usr/local/bin, an edited authorized_keys, a fresh systemd timer with its matching service, a new per-user crontab (an entry telling cron, the old Unix job scheduler, to run a command on a schedule). All touched in a six-minute window around one suspicious login. Notice the flag is -newerct, not -newermt: you match on change time, not modify time. That is deliberate. A backdated file hides from a modify-time search, but its change time still points at the real moment it landed, which is why update-cache turns up here even though its mtime claims January. The tradeoff is that ctime is noisier (a chmod or a rename bumps it too), so you confirm each hit rather than trust it blindly. The -xdev flag keeps find on the root filesystem, so it does not wander into other mounts like /proc and /sys (the kernel's virtual filesystems), and 2>/dev/null hides the permission-denied noise. To do this across a whole disk image instead of a live host, the Sleuth Kit (an open-source forensics toolkit) walks the filesystem into a body file, and mactime sorts every timestamp it found into one stream.
The macb column shows which of the four times landed on that line (modify, access, change, birth), so a macb line means all four coincided, the signature of a file that was created right then. Look at update-cache, though. Here it shows up with only its birth and its change time, because its modify and access times were stomped back to January and sort themselves off near that fake date. The new authorized_keys, timer, and service each read macb, all four at once, because they really were born at 03:14. For the full picture, log2timeline/Plaso (a tool that stitches together a 'super timeline') folds filesystem times, journal entries, shell history, and more into one sorted stream. And you always cross-check against the off-host log store, the copy the attacker could not reach, so even a host with edited local logs can be put back in order.
The journal keeps its own receipts
On a modern Linux running systemd (the manager that starts and supervises the services on most current distributions), most logs no longer live in plain text. Its logging service (systemd-journald, the background program, or daemon, that gathers messages from the kernel and every running service; the kernel is the core of the operating system that talks straight to the hardware) writes them into a binary database under /var/log/journal, which you read back with journalctl. That binary format is an advantage for a defender. The journal can seal itself as it writes, using Forward Secure Sealing (FSS, a scheme that cryptographically stamps each chunk of the log with a rotating key so any later edit or deletion stands out). You set it up once with journalctl --setup-keys, which prints a verification key for you to copy somewhere safe off the box. From then on journalctl --verify walks the journal and reports whether the seals still hold.
[Journal]Storage=persistentSeal=yesCompress=yesForwardToSyslog=noSystemMaxUse=2G
Two things to notice. The seals all pass, so the journal itself was not edited. But --list-boots shows the previous boot ending at 03:30:58 and a new one starting at 03:31:44, a reboot minutes after the 03:14 login. A reboot in the middle of an incident is worth a hard look, because a common reason is loading a malicious kernel module (which needs a fresh boot) or flushing evidence out of memory. This is also why Storage=persistent matters. If the journal only lived in /run/log/journal (memory-backed, the fallback when /var/log/journal does not exist) that reboot would have erased every log line before it, and you would have nothing.
Who logged in, and the silences that follow
Three files answer who was here, and they work like a hotel front desk. The guest book of everyone who checked in (wtmp), the rejected-visitor log of everyone turned away at the door (btmp), and the list of who is in the building right now (utmp). They are binary, so you read them with tools: last reads wtmp (successful logins and reboots), lastb reads btmp (failed attempts, root only), and who or w read utmp (current sessions). Alongside them sits the door camera itself, /var/log/auth.log on Debian and Ubuntu (or /var/log/secure on Red Hat systems), where sshd (the OpenSSH server that handles remote logins over SSH, the Secure Shell protocol for connecting to a machine over an encrypted link), sudo, su, and PAM (Pluggable Authentication Modules, the framework Linux uses to check logins) write every authentication event in plain text.
Read together they tell a story. A burst of failed logins for invalid users (a brute-force attempt, hammering common account names), then an accepted publickey login for deploy from the same address, then a sudo to root a minute later. The last output backs it up and adds a detail: deploy's 03:14 session reads '- down', cut off by the 03:31 reboot you already spotted in the journal. Note the SSH key fingerprint on the accepted line (SHA256:...). OpenSSH records that fingerprint at the default LogLevel INFO, so you can line it up against a specific public key in the account's authorized_keys and pin the login to one key. Turn the setting up to LogLevel VERBOSE in sshd_config and it logs more of the handshake, including the fingerprints of keys a client offers that do not end up working. Be careful, though. wtmp, btmp, and utmp are plain binary files that any root user can rewrite with utmpdump (which dumps them to text and can load edited text straight back) or truncate to nothing, so treat them as leads to corroborate against auth.log and the off-host record, not as ground truth.
The last per-user trace is the notepad the intruder scribbled on: shell history. Bash keeps it in the file named by the HISTFILE variable (usually ~/.bash_history), and it is trivial to read, which is exactly why attackers attack it.
That symlink is the whole story. Someone ran ln -sf /dev/null ~/.bash_history, so every command from every future session for deploy is written straight into the void, while backup's history is intact. Other common wipes are unset HISTFILE, export HISTFILESIZE=0, and kill -9 $$ to kill the shell before it flushes history on exit. Know too that .bash_history has no timestamps by default. Bash only writes #<epoch> marker lines into it when the HISTTIMEFORMAT variable was set at the time the commands were recorded, so do not expect times in the file unless the environment was configured for them.
Which points at a rule worth holding onto: missing evidence is itself evidence. Attackers clean up after themselves. They wipe .bash_history, delete their tools, and edit or truncate local logs to cover their tracks. A suspiciously empty history, a journal with a gap right around the incident window, a login file truncated to zero bytes, a system binary that no longer matches the package it shipped in, none of that is 'nothing to see here'. It is a strong signal that a capable adversary was present and took the time to hide. Record the absence as a finding, stamp its timestamp against the rest of your timeline, and fall back on the off-host copy that still holds the record they deleted locally.
Trust what the attacker could not reach
When you stack all of this up, weight each source by one question: could the attacker touch it? That gives you a hierarchy. Off-host telemetry (data the machine measured and shipped somewhere else as it happened) sits at the top: the audit and eBPF records you sent elsewhere (eBPF is a way to run small sandboxed programs inside the kernel to watch events), the central syslog server (a separate box that collects log lines as they are written), the network sensor's flow data (records of which machine talked to which, and when). The attacker never had hands on it, so it is your most trustworthy account. A memory image captured early comes next, taken with a tool like LiME or AVML (utilities that dump the contents of RAM, the computer's fast working memory, to a file for analysis). Disk artifacts follow, valuable but editable. And dead last is the live host's own tools (ps, ls, netstat), because a rootkit (malware that hides itself and the attacker's processes from the system's own commands) can make them lie to your face. The part of that ranking that puts memory ahead of disk echoes the order of volatility, a long-standing forensics guideline (RFC 3227) that says to grab the most fleeting evidence first, before it evaporates.
Memory outranks disk for a concrete reason. A tool the attacker deleted from disk is often still running, and a running program keeps living in RAM and in /proc (the kernel's live, in-memory view of every process). You can catch it there, and even recover the deleted file.
The NLINK of 0 and the (deleted) tag are the giveaway: the file is gone from the directory tree, but the process holds it open, so the bytes are still in the kernel and readable through /proc/20915/exe. Copy them out before the process exits and you have the attacker's binary for analysis even though nothing remains on disk.
So when you write up the timeline, tag every fact with where it came from and how much you trust it. The line the network sensor recorded outranks the line the host's own logs show, and both outrank a silence on the host where a record used to be. Do that consistently and the timeline stops being a pile of artifacts and turns into an argument: this account, from this source, at this second, is why you believe what happened. That is the deliverable an incident responder, a lawyer, or the next engineer on the rotation can actually act on.
Try this
Work through “Trust what the attacker could not reach” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: access times lie by default. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.