The incident response loop
From detection to lessons learned.
A building fire follows a script. The alarm goes off, someone finds the source, fire doors close to box it in, the crew puts it out, the space gets rebuilt and reopened, and afterwards a fire marshal writes up what went wrong so the next one is smaller. Nobody invents that sequence while smoke fills the hallway. Incident response (IR, the work of handling a security breach from the first alert to the final write-up) is the same idea applied to a compromised machine. Under pressure you follow a rehearsed loop instead of improvising, because improvising is how you tip off the attacker, destroy the evidence, or clean one host while they sit quietly on three others.
The loop has six phases, and it really is a loop, not a straight line. Preparation happens before anything fires. Detection and analysis asks two questions: is this real, and how far does it reach. Containment stops the bleeding. Eradication removes the attacker and every foothold they planted. Recovery restores service from something you actually trust. Lessons learned feeds what you found back into preparation, so the next incident is smaller or never happens at all. Skip a phase and you pay for it later. Rush the scoping step before you eradicate, and the attacker walks right back in.
Preparation, the bookend that decides everything
You cannot review security footage you never recorded. Preparation is having the telemetry, the tooling, and a written plan in place before an alert ever fires. The single setting people miss on Linux is whether logs survive a reboot. On modern systems the logging service is journald (the built-in log collector that ships with systemd, the software that boots Linux and supervises its background services). With its default Storage=auto, if the directory /var/log/journal does not exist, journald keeps everything in memory under /run and throws it away on the next boot. During an incident you often reboot. Make the journal persistent, and cap its size so it cannot fill the disk.
[Journal]Storage=persistentSystemMaxUse=2GForwardToSyslog=yes
Persistent local logs help, but a root-level attacker can delete /var/log at will, so the copy that actually counts is the one already shipped off the box to a central store the attacker cannot reach (ForwardToSyslog=yes hands each line to the local rsyslog daemon, which you point at a log server elsewhere). The rest of preparation is the same shape: the Linux audit daemon (auditd, the kernel-level recorder of security events) running everywhere, osquery (a tool that lets you query a machine's live state as if it were a database) deployed across the fleet, and a runbook that names who does what and lists the phone numbers. The time to install the smoke detector is not during the fire.
Detection and analysis: is it real, and how big
An alert fires. Your egress monitoring (the alarms watching traffic leave your network) flags one host holding a connection to an IP address that belongs to a known Tor exit node (Tor is an anonymity network attackers hide behind). First question, always: real or noise. Start with who is actually on the box, using the login history in /var/log/wtmp (the file that records every login and logout).
There it is. The root account is logged in interactively over a pseudo-terminal (pts, a live shell session) from an external address, and that address is the one your egress alert flagged. In the hardening lesson you set PermitRootLogin no in the SSH (secure shell, the encrypted remote-login service) config, so a direct root login from the internet means either that control was changed or the config was bypassed. Both are alarming. Now follow that session to what it did. The ss command lists network sockets; with root it shows which process owns each one.
Three connections, two of them a problem. One inbound SSH session from the flagged IP (that is the attacker's shell), and one outbound connection from a process called kworkerd talking to the same IP on port 443. That name is a costume. Real kernel workers are kernel threads, they show up in square brackets like [kworker/0:1], and they never own a network socket. A user-space process named kworkerd holding a TCP connection is malware wearing a kernel worker's uniform. This is the command-and-control channel (C2, the attacker's remote control server). Find the real binary behind PID 8210 (its process ID) by reading its executable link in /proc, the kernel's live view of every running process.
The (deleted) tag is the attacker being careful. They ran the binary from a hidden directory under /tmp and then unlinked it from disk so a file listing shows nothing. But the kernel keeps a deleted file alive as long as a process holds it open, and /proc/8210/exe still points straight at the open handle. So you can recover the malware as evidence before you touch anything: cp /proc/8210/exe /evidence/kworkerd.bin pulls the whole binary back out through that link. One running process is a symptom, though, not the disease. How does it survive a kill or a reboot? Sweep for everything created since the login time.
[Unit]Description=Kernel Worker Daemon[Service]ExecStart=/tmp/.cache/kworkerdRestart=always[Install]WantedBy=multi-user.target
This is the persistence. The attacker registered their malware as a systemd service, a unit that systemd starts and keeps running. Restart=always is the important line: it tells systemd to relaunch the process the instant it dies. Think of the running process as a weed's leaf and the unit file as its root. Kill the PID and systemd grows the leaf right back. To stop it for good you have to pull the root.
Scope before you eradicate
A cleaner who wipes one infected room while leaving the other two has not cleaned the house. You now hold a handful of indicators of compromise (IOCs, the concrete fingerprints of this attack): the C2 address 185.220.101.51, the file /tmp/.cache/kworkerd, the unit name kworkerd.service, and the login source. Before you remove anything, answer one question across every host you run: where else is this? osquery lets you ask it as a database query, and a fleet manager runs that same query against thousands of machines at once.
You thought you had one compromised host. You have three, and one of them is a database server. Run the same short set of queries for each IOC (the file path, the unit name, the login source) until you have the complete footprint. Only then do you act, everywhere at once. If you clean web-03 first, the attacker on db-01 watches their access disappear, learns exactly what you can see, and comes back through a quieter door you have not found yet.
Containment without destroying the evidence
You write things down in two kinds of places: on a whiteboard and in a filing cabinet. Memory (RAM, the fast working store a computer wipes on every reboot) is the whiteboard. It holds the attacker's keys, decrypted payloads, the deleted binary, and every live connection, and it is gone the instant the machine restarts. Disk is the filing cabinet, and it survives. So the order of operations is fixed: capture the volatile whiteboard first (the socket list, the process list, the /proc/PID/exe copy you already took), and only then isolate the host in a way that keeps it powered on. Cut the C2 channel with the modern Linux firewall, nftables.
The policy stays accept so you do not sever your own forensic access, and the single drop rule silences this beacon. Treat that as a stopgap. A prepared attacker keeps fallback addresses, so real containment is moving the host onto a quarantine network segment that can reach only your forensic collector and nothing else. Keep it running. A live machine you control, still holding its memory, is worth far more than a powered-off box.
Eradication and recovery
Pull the root, not the leaf. Stop and disable the unit, delete the file (after copying it to evidence), then mask the name so the same trick cannot be replanted. Masking points the unit at /dev/null, which is the equivalent of gluing the light switch in the off position: systemd will refuse to start anything registered under that name.
Do not mistake a cleaned host for a trustworthy one. You found one service, but you cannot prove you found everything a root-level attacker touched, and there is no command that proves a negative. Real eradication means rebuilding the machine from known-good media (a fresh image you trust), restoring data from a backup that predates the intrusion, patching the entry vector so the same door is shut, and rotating every credential the attacker's session could have reached: the SSH keys, any password typed at that shell, and every token or secret readable from a root prompt. Recovery is bringing the rebuilt host back into service and watching it harder than usual for a return.
Lessons learned
An airline does not respond to a crash by finding someone to fire. It runs a blameless investigation to find the cause, because blame makes people hide the details, and the details are the whole point. Run the review the same way. The root-cause question here is blunt: how did root log in over SSH from the internet when your config forbids it? A leaked key, a changed setting, a bypass. Then ask the question that actually pays off: what detection would have caught this hours earlier? Turn that answer into a rule that ships. The attacker persisted by writing a systemd unit, so wire a tripwire onto exactly those paths with auditd.
# Alert on new or changed persistence locations-w /etc/systemd/system/ -p wa -k unit_persist-w /etc/cron.d/ -p wa -k cron_persist-w /etc/crontab -p wa -k cron_persist
Never trust a detection you have not seen fire. Prove it works by simulating the exact action the attacker took and confirming the audit log caught it.
Delete the test unit, and the exact path the attacker used to survive now has a sensor on it that pages you the moment it happens again. That is what the loop buys you. Every incident hands you a map of where your visibility was blind, and you close one of those blind spots for good before you close the ticket. An incident you responded to but never turned into a detection is one you have quietly agreed to run again.
Try this
Work through “Lessons learned” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: eradicating before scoping is the classic own-goal. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.