Live triage: contain, do not destroy
The first hour, done right.
A detection fires at 2 a.m. A process you have never seen is talking to an address in another country. Every instinct says do the fast thing: kill it, reboot, reimage, close the ticket. That reflex is the most expensive move in incident response, because it treats a crime scene like a dirty kitchen. You do not mop a crime scene. You photograph it, bag what is perishable, and only then decide what to touch. The first hour on a live compromised host is about keeping the scene intact while it is still warm. The order you work in decides whether you can ever answer the questions that actually matter: how did they get in, what did they touch, are they still here, and are they on your other machines too.
The rule fits on a sticky note and is hard to hold under pressure. Contain without destroying. Volatile evidence (the running processes, the open network connections, the contents of memory) vanishes on reboot and thins out every minute you wait. Preserve it first, contain the host second, investigate third. Get that order backwards and you evict the attacker from one box while burning the only record of what they did.
Order of volatility: grab what evaporates first
Evidence has a shelf life. Some of it is carved in stone: a file written to disk is still there after a reboot. Some of it is a footprint in wet sand at the tide line: a network connection, a process that exists only in memory, gone in minutes and certainly gone once the power blinks. Order of volatility, first written down in RFC 3227 (Request for Comments 3227, an internet engineering guideline for collecting digital evidence), says collect the fastest-fading things first. In practice you grab the perishable state up front: the live process table and the open network connections, which a handful of commands capture in seconds, then the full contents of memory (RAM, the working memory the running system lives in), which takes minutes to copy off, then the disk, which can wait because it survives a reboot. On a host you suspect is rootkitted (running attacker code that lies to the tools you would normally trust), reach for a static toolkit you brought with you, write your output to external media, and check everything against logs the host cannot edit.
Start by writing down the time in a format nobody can argue about, then snapshot the live state that a reboot would erase.
Follow the connection back to its binary
A snapshot is only useful if you read it. The socket table is where a lot of intrusions give themselves away, because malware that phones home has to open a connection, and that connection has a process behind it. TCP (Transmission Control Protocol, the connection-oriented traffic most services use) sockets map straight to a PID (process ID, the number the kernel uses to track a running program).
One line here does two suspicious things at once. It is beaconing out to a random internet address on port 443, and it calls itself kworker. Real kworker entries are kernel worker threads. They have no program file on disk, so their /proc/<pid>/exe link points at nothing. This one points at a file that was deleted while the program kept running, sitting in /dev/shm (a temporary filesystem, tmpfs, that lives in RAM and is world-writable). Deleting your own binary while staying resident is a classic way to hide from anyone scanning the disk.
The bytes are gone from the directory listing but still mapped in memory, so you can copy the program straight out of /proc and hash it before the process ever dies.
Image memory before you touch the disk
Memory holds the things that never hit the disk: keys typed into a process, decrypted payloads, injected code, the real command line of a program that rewrote its own arguments to hide. It is also the first thing you lose. AVML (Acquire Volatile Memory for Linux, a single static binary) captures it without compiling a kernel module on the suspect host, which matters when you do not trust that host's compiler or headers. It writes LiME format (Linux Memory Extractor format), the standard input for later analysis in a tool like Volatility (an open-source memory-forensics framework).
The logs might be living in RAM too
Here is the detail that turns 'do not reboot' from a slogan into a hard rule. On most modern Linux systems, logging is handled by journald (the logging service that ships with systemd, the software that starts and supervises everything else on the machine). Where journald keeps its journal comes down to one setting.
[Journal]Storage=auto#Compress=yes#SystemMaxUse=#RuntimeMaxUse=# Storage=auto -> persist to /var/log/journal IF that directory exists,# otherwise keep the journal in /run/log/journal, which is tmpfs (RAM).
Storage=auto means: save logs to /var/log/journal if that directory exists, otherwise keep them in /run/log/journal, which is tmpfs, which is RAM. Plenty of cloud and container base images ship without /var/log/journal, so their entire log history is volatile. It reads fine right now with journalctl, and it evaporates the instant the machine reboots. Check where you actually stand, then serialize the journal to stable storage before anything can take it away.
Isolate without pulling the plug
Containment is about stopping the spread while keeping the patient alive. Pulling the network cable works, but it can tip off malware that watches for isolation and wipes itself, and it cuts your own ability to pull evidence off the box. A cleaner move is to quarantine at the firewall: drop everything except the one host you collect from. On a modern system that firewall is nftables (the in-kernel packet filter framework, successor to iptables). Point the host at your forensic workstation and nothing else.
The attacker's command-and-control channel (C2, the connection the malware uses to take orders) is now cut off, your collector still reaches the box over SSH (secure shell), and the machine stays up with its memory and process state intact. Loading the whole table in one shot with nft -f matters here: the drop policy and the collector-allow rules commit together, so there is never a split second where the box locks out the session you are working from. Priority -400 sits below every standard nftables hook priority, so this table runs ahead of any ordinary filter rule, and nothing else gets to accept the traffic you meant to drop.
Treat this host firewall as a fast first move, not the last word. A rule you set with nft can be pulled right back out by anyone who has root on the box, and on a compromised machine that may be the attacker. Cut it off somewhere root cannot reach: a quarantine VLAN (virtual local area network) on the switch, or a locked-down security group in the cloud. Authoritative isolation happens above the host, not on it.
Your own actions are evidence too
A chain of custody is the paper trail that lets someone else trust your evidence: who collected what, when, and that it has not changed since. Start recording before your first real command, so the record includes your own keystrokes and their output. Do not log in as the account that may be compromised, and lean on your static tools rather than the host's, because subverted binaries can hand you clean-looking lies.
When collection is done, freeze a manifest. Hash every finished artifact with SHA-256 (a cryptographic checksum, so any later change is obvious) and store the list beside the evidence.
Once the artifacts are copied, hashed, and off the box, and the host is cut off above the operating system, you have the one thing every later step depends on: a frozen picture of the machine exactly as the attacker left it. Now you can load the memory image into Volatility, diff the process tree against a known-good baseline, and trace the intrusion across the rest of your fleet without ever wondering what you wiped. Rebuild the server whenever you are ready. The scene is already saved.
Try this
Work through “Your own actions are evidence too” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: reboot on reflex and the case is gone. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.