Detections that survive contact

Behavior over strings, tests over hope.

A wanted poster that says the suspect wears a red baseball cap is useless the moment he takes the cap off. A poster that says he comes in through the roof, cuts the power to the alarm panel, and leaves through the loading dock describes what he does, and he cannot stop doing those things without giving up the whole job. Detection engineering splits the same way. You can alert on what an attacker's tool happens to be right now (its file hash, its filename, the address it calls home to) or on what the attacker has to do to win (spawn a shell from your web server, read a private key, write to a startup path). One set of facts changes in seconds. The other costs real effort to change, and some of it cannot change at all without abandoning the technique.

A detection survives contact when a live intruder walks into it and still trips the wire, even after they recompile their malware, rename their binaries, and rotate their servers. That is the bar. Anything below it is a detection that looked fine in a slide deck and went dark the first time it mattered.

Cheap strings, expensive behavior

Security engineer David Bianco drew this idea as the Pyramid of Pain. At the bottom sit the signals that cost an attacker almost nothing to change: file hashes, IP addresses (the numeric address of a machine on the network), domain names. These are indicators of compromise (IOCs, the specific leftovers a known bad thing drops). Higher up sit the attacker's tools, which take more effort to swap. At the very top sit tactics, techniques, and procedures (TTPs, the how of an attack rather than the what). The name of the pyramid is the whole point. When your rule forces the attacker to change something near the top, you have caused them real pain. When it only forces them to recompile, you have caused them a ten-second wait.

The strongest pull in this job is to write a rule for whatever is in the latest threat report: this hash, this filename, this command-and-control domain. It feels productive. It ships fast. And it is stale before the ink dries, because the next campaign uses a fresh build on fresh infrastructure. Spend your effort one level up. Ask what the technique fundamentally has to do, then write the rule there.

Watch what a process does, not what it is called

Take a real persistence trick: dynamic linker hijacking (catalogued as technique T1574.006 in MITRE ATT&CK, a public library of real-world attacker techniques run by the non-profit MITRE; ATT&CK stands for Adversarial Tactics, Techniques, and Common Knowledge). The dynamic linker is the part of the system that loads shared libraries into a program as it starts. The file /etc/ld.so.preload lists libraries to force-load into every program that runs, including sshd, sudo, and cron. Drop your library there and your code runs inside everything. You will never catch this by hashing the library, because the attacker builds a fresh one per host. You catch it by watching the file that has to be touched.

That watching is the job of the Linux audit system. The matching happens inside the kernel (the core of the operating system that talks to the hardware and controls every privileged action), and a userspace helper called auditd (the audit daemon) writes down what the kernel reports: file access, system calls, and the like. A system call is the request a program makes to the kernel to do privileged work, such as opening a file or launching another process. Put a watch on the preload file and every write shows up with the who and the how attached.

Read the flags plainly. The -w names the path to watch. The -p wa says fire on writes (w) and attribute changes (a) such as permission or owner edits. The -k tags every matching event with the key linker_hijack so you can find it later. Now reproduce the technique the way a defender should, on a lab box, with a harmless library path that does not exist.

The SYSCALL line is where the value lives. comm=tee and exe=/usr/bin/tee tell you what ran. uid=root tells you it ran as root. The one to circle is auid=alice. That is the login UID (auid, the user ID recorded when the human first logged in), and it sticks to every action that person takes even after they switch to root with sudo. uid says root did it; auid says alice became root and did it. That single field turns an anonymous root event into an accountable one. One caveat: auditctl -w is loaded into the running kernel and vanishes on reboot. To make the watch survive a restart, drop it in a rules file that augenrules loads at boot.

-w /etc/ld.so.preload -p wa -k linker_hijack
-w /etc/ld.so.conf -p wa -k linker_hijack
-w /etc/ld.so.conf.d/ -p wa -k linker_hijack

A rule you have never fired is a hope

A detection you have never triggered on purpose is a smoke detector you have never held a match under. You do not know if the battery is in, if the sensor points at the kitchen, or if it screams at burnt toast every single morning. You find out during the fire, which is the worst possible time to learn. Treat detections like code. For every rule, keep a safe way to reproduce the behavior it should catch, fire it, and confirm the alert lands. Then confirm that normal activity stays quiet.

Here is the same idea with Falco (an open-source runtime security tool that reads the stream of system calls through a kernel module or an eBPF probe, which is extended Berkeley Packet Filter, a way to run small sandboxed programs inside the kernel). Rules are written in YAML (a plain-text configuration format). This one alerts when a private SSH key, the secret half of a Secure Shell login pair, is read by a process that has no business reading it.

- rule: SSH Private Key Read By Unexpected Process
  desc: >
    A private SSH key was opened for reading by a process outside normal
    SSH or git activity. Common in credential theft after a foothold.
  condition: >
    open_read
    and (fd.name endswith "/id_rsa" or fd.name endswith "/id_ed25519")
    and not proc.name in (ssh, sshd, scp, ssh-agent, ssh-keygen, git, sftp-server)
  output: >
    SSH private key read (file=%fd.name reader=%proc.name
     cmd=%proc.cmdline user=%user.name login_uid=%user.loginuid
     pid=%proc.pid ppid=%proc.ppid)
  priority: WARNING
  tags: [host, filesystem, credential-access, T1552.004]

Read the condition like a sentence. open_read is a built-in Falco shorthand for a file being opened to read. fd.name endswith checks the path of that file. The not proc.name in (...) line is an allowlist of the tools that legitimately read keys, so ssh and git stay silent. The output line pulls out the file, the reader, the full command, and login_uid (the same accountable login identity you saw as auid in auditd). The tag T1552.004 is the ATT&CK ID for stealing private keys, which matters in a moment. Load it and prove it works.

It fired, and it caught the behavior, not a name. Swap cat for python, curl, or a renamed binary and the alert still lands, because the durable fact is a process outside the allowlist reading a key file. Now test the other side, which people skip: run git and ssh against a key and confirm Falco stays quiet, or you have built an alarm that cries every deploy. Those are the two failure modes you are hunting. The rule that never fires gives you a false sense of coverage. The rule that fires constantly buries the real alert under noise until nobody reads either. You want to find both in a test, not in an incident. Atomic Red Team, a free library from Red Canary, packages hundreds of these small reproductions, each mapped to an ATT&CK technique and each shipping with a cleanup step, so you can run the benign version of a technique with one command and watch what lights up. Run those in a lab or a throwaway host, never on the box you are protecting, because some of them make real changes.

Coverage is a portfolio, not a pile

A dozen good rules are not coverage. Coverage is knowing which techniques you can see and, more honestly, which you cannot. Because every rule carries its ATT&CK tag, you can inventory what you watch straight from the rule files.

Now hold that list against the techniques that actually threaten your environment. Free tools help here: the ATT&CK Navigator colors the matrix so gaps are visible at a glance, and DeTT&CT scores how good each detection really is. This box watches cron jobs, unix shells, systemd service persistence, key theft, and linker hijacking. It watches nothing for T1548 (abusing sudo or setuid to escalate privilege). That is a gap, and because you can name it, it is a decision you can argue about and prioritize. The gaps that hurt are the ones nobody wrote down. A curated portfolio has known holes; a pile of rules has surprises.

Pick one technique you claim to detect. Open a lab shell, run its benign form, and watch. If no alert lands, you did not have a detection. You had a line in a spreadsheet that said you did.

Try this

Work through “Coverage is a portfolio, not a pile” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a process name is a string too. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related