Detections that survive contact

Behaviour over indicators, and testing rules.

A detection is worth building only if it still fires after the attacker changes their tools. Most do not. A rule that names a file, a hash or an address goes dark the moment the intruder renames a binary or rents a new server, which costs them seconds. A rule keyed on what the technique has to do (run a program out of a scratch directory, read a private key, write to a startup path) costs them far more, because they cannot drop the action without giving up the goal. In this lesson you write two audit rules for the same tool, one keyed on its path and one on its behaviour, watch the first go blind when the tool is renamed while the second keeps catching it, and then measure each rule against ordinary admin work so you ship the one that is both durable and quiet. auditd is the collector here; it is not installed on a default Ubuntu server (linux-det/threatmodel), and the rule grammar and the auid login-identity field belong to linux-det/auditpipe, so they are used here without being re-taught.

The cheap signal and the expensive one

Security engineer David Bianco drew this trade-off as the Pyramid of Pain. At the base sit the indicators an attacker changes almost for free: hash values (trivial), IP addresses (easy), domain names (simple). In the middle sit network and host artefacts, such as a distinctive file name or path, which are annoying to change, and above them the attacker's tools, which are challenging to replace. At the top sit tactics, techniques and procedures, the way the intrusion is carried out, which are tough to change without abandoning the attempt. The pyramid tells you where to spend effort: a rule built on the base is stale before it ships, while a rule built near the top keeps working through a rename, a rebuild and a new server.

Copying a legitimate utility to a new name or location to slip past monitoring is a technique of its own: T1036.003 (Masquerading: Rename Legitimate Utilities) in MITRE ATT&CK, the public catalogue of attacker behaviour. A rule that watches for the name nc or the path /usr/bin/socat is defeated by exactly that move, so write the rule against the move instead.

Phase one: a rule keyed on the path

Start the way an indicator feed would have you start: one audit rule for one executable, identified by its exact path. The rules file also carries a collection rule that records every program started from a login session; it is there only so you can measure noise later. -F exe= matches the executable's path, -F perm=x selects execution, and arch=b64 scopes the rule to 64-bit system calls. On an x86_64 server that can also run 32-bit programs, a 32-bit execve does not match a b64 rule, so ship each rule with an -F arch=b32 copy there (auditctl(8), arch=). The scratch directory was created first with sudo install -d -m 1777 /var/tmp/det-lab: world-writable with the sticky bit, like /var/tmp itself.

## Indicator: this one executable, by its exact path
-a always,exit -F arch=b64 -F exe=/var/tmp/det-lab/sysmon -F perm=x -F key=det_ioc
 
## Collection: every program started from a login session
-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=unset -F key=det_all

The lab loads the file with augenrules --load, the same way boot does, and auditctl -l shows what the kernel now holds. auditctl has added -S execve for the perm=x rule and prints auid!=unset as auid!=-1, its numeric form.

Now stand in for a foothold. An unprivileged account drops a harmless tool into the world-writable scratch directory and runs it. The tool is a copy of GNU id (/usr/bin/gnuid on Ubuntu 26.04, a standalone binary; the default id is part of the uutils multi-call binary, which refuses to run under a name it does not recognise). It prints one line and does nothing else, but to the kernel it is a program executed from /var/tmp.

The indicator rule caught it. ausearch --format text turns the raw record into one readable sentence: who executed what, and when. Now do the thing that costs the attacker nothing, copy the same bytes to a new name and run them, then ask the rule again.

The output has not changed. The rule still shows only the sysmon run; the kswapd0 execution, identical bytes doing the identical thing a second later, is invisible to it because the rule names a path and the path changed. (The name kswapd0 imitates a kernel thread, which is a common disguise.) The execution was recorded by the collection rule, so the evidence exists; the detection simply did not look for it.

Phase two: a rule keyed on the behaviour

Replace the indicator with a rule about the behaviour: any program executed from under the scratch directory, whatever it is called. -F dir= puts a recursive watch on a directory tree. A foothold has to stage its tools somewhere it can write, and /var/tmp, /tmp and /dev/shm are the usual places; the lab watches its own subdirectory so the transcript stays reproducible, while a production rule would watch all three.

## Behaviour: any execution from the world-writable scratch tree, whatever the file is called
-a always,exit -F arch=b64 -F dir=/var/tmp/det-lab -F perm=x -F key=det_behav
 
## Collection: every program started from a login session
-a always,exit -F arch=b64 -S execve -F auid>=1000 -F auid!=unset -F key=det_all

The two rules are tested one at a time for a reason worth knowing. auditd writes each event once, under the key of the first rule that matches (auditctl(8): "the event triggers on the first matching rule"). With both the path rule and the directory rule loaded, the sysmon run would be filed under whichever came first and the other rule would look blind. Rule order is part of the design, which is also why suppression (never) rules go at the top of a rules file.

Both names are caught. The rule does not care what the file is called because it keys on where it runs from, and no rename moves the program out of that directory; to slip the rule the attacker has to stage somewhere else, and the callout below lists where. The raw record behind the second hit shows the fields a detection should rely on.

exe=/var/tmp/det-lab/kswapd0 is the path the program ran from and syscall=execve is the execution; the lab is aarch64, and an x86_64 server records arch=x86_64 in this interpreted view (c000003e in the raw log). auid=deploy ties the event to the login behind it. comm=kswapd0 is the least trustworthy field in the line: the kernel sets it at execve from the name of the file that was executed, so a rename changes it, and a running program can change it again with prctl(PR_SET_NAME). argv[0], which audit shows in the PROCTITLE record, is whatever the launcher chose. Either way comm=nginx can be a lie. Anchor on what the attacker cannot choose freely: the path a file runs from, the parent-and-child chain (a database daemon should never be the parent of a shell), and the system calls actually made.

A rule you have not measured is a guess

A durable rule is useless if it drowns you. The opposite failure to "never fires" is "fires all day until nobody reads it", and you only find out which one you built by measuring. Run ordinary admin work with the behaviour rule loaded, then count each key since the rule was loaded.

The collection rule, which records every program a logged-in user starts, logged over a hundred events from a few seconds of routine work: that is what an "alert on every execution" rule would page you with. The behaviour rule logged two, the sysmon and kswapd0 runs, and nothing from the admin session, because nobody on this host legitimately runs programs out of that directory. Low volume on normal activity is what makes a rule shippable. If det_behav had fired on routine work, you would have found a real workflow that stages in a scratch directory, and you would fix the workflow or add a narrow exception before the rule paged anyone at 2 a.m. Run the measurement on a host doing the work you are protecting, not an idle one, because the false positives that matter come from the legitimate activity a rule has to live alongside.

Try this

On a lab host with auditd, repeat both phases with your own scratch directory (sudo install -d -m 1777 /var/tmp/lab). Put each rule in /etc/audit/rules.d/70-lab.rules and load it with sudo augenrules --load, as the lesson did. Load only -a always,exit -F arch=b64 -F exe=/var/tmp/lab/tool -F perm=x -F key=byname, copy /usr/bin/gnuid to /var/tmp/lab/tool, run it, and confirm sudo ausearch -k byname --format text shows it. Copy it to /var/tmp/lab/kworkerd, run that, and confirm byname still shows only tool. Then replace the rule with -F dir=/var/tmp/lab -F perm=x -F key=bybehav, run both copies again, and confirm bybehav shows both. Finally run id, ls and systemctl status a few times and check with sudo aureport -k --summary -ts recent that bybehav did not grow. If you load both rules at once, expect the first one in the file to take every event it matches.

Takeaway

Key detections on what a technique cannot avoid doing, not on a name, hash or address it can change for free, and prove every rule twice before you trust it: fire it with a safe reproduction, and confirm it stays silent through ordinary admin work.

Related