Userland stealth: LD_PRELOAD & more

Hooking without touching the kernel.

Every ordinary Linux command leans on code it did not write. When ls lists a folder, it does not read the raw disk. It calls a function inside a shared library (a bundle of ready-made functions that many programs borrow at run time) to do the work. The big one is libc (the C standard library, the basic toolbox nearly every program on the system links against). Reading a directory, opening a file, making a network connection: all of it flows through libc.

That borrowing is the opening. If you can slip your own code in front of libc, you answer the phone before the real function does. You change what the program sees, then hand the call along as if nothing happened. No kernel (the core part of the operating system that talks directly to the hardware) has to be touched. The program runs its normal binary, calls its normal functions, and quietly gets fed a doctored answer. That is a userland rootkit: a toolkit an intruder installs to stay hidden, living entirely in user space (the normal, unprivileged area where programs run, separate from the kernel), dressing up what your tools report.

The line at the service desk

A service desk handles every request for "list this directory" by sending it to whoever stands first in line. The dynamic linker (the small program, ld.so, that wires a program to its shared libraries the moment it starts) builds that line. Normally libc is near the front, so libc answers. LD_PRELOAD is an environment variable (a setting handed to a program when it launches) that lets you shove a library of your choosing to the very front. When the program calls readdir (the libc function that returns directory entries one at a time), your library answers first. The name for this is symbol interposition: two libraries define a function with the same name, and the first one found wins.

ldd (a tool that lists the shared libraries a program pulls in) shows the whole picture. libc.so.6 is the toolbox ls borrows, and /lib64/ld-linux-x86-64.so.2 is the loader that stands everyone in line. A statically-linked program would print not a dynamic executable here and carry its own private copy of every function, which matters a lot later on.

On a stock Ubuntu or Debian box that file does not exist. Its absence is your baseline. Anything sitting there is worth a hard look.

Watch a library lie

Here is the whole trick in about a dozen lines of C. This library redefines readdir. Every time a program asks for the next directory entry, our version calls the real one, drops any name containing the word secret, and hands the rest back.

#define _GNU_SOURCE
#include <dlfcn.h>
#include <dirent.h>
#include <string.h>
 
// remember where the real readdir lives
static struct dirent *(*real_readdir)(DIR *) = NULL;
 
struct dirent *readdir(DIR *dirp) {
    if (!real_readdir)
        real_readdir = dlsym(RTLD_NEXT, "readdir");  // the next readdir in line: libc's
 
    struct dirent *entry;
    while ((entry = real_readdir(dirp)) != NULL) {
        if (strstr(entry->d_name, "secret") == NULL)  // not hiding it? pass it on
            return entry;
    }                                                  // otherwise skip and loop
    return NULL;
}

RTLD_NEXT (a flag meaning "the next definition of this symbol after me in the search order") is how the hook reaches the genuine libc readdir without calling itself forever. It filters, then forwards. The program never notices.

Same ls, same directory, same kernel. The file secret_notes.txt is still on disk, byte for byte. The only thing that changed is which readdir answered. That single idea powers hiding of files, processes, and network connections. Swap the filter and you change the target: match a magic filename to hide files, match a process name found under /proc to hide a process, match a source port in /proc/net/tcp to hide a connection. A production-grade hook also overrides readdir64, the large-file variant some programs call, but on a 64-bit desktop the single hook is enough to show the effect.

From one shell to the whole box

LD_PRELOAD set in a shell only touches programs that shell launches. That is loud and short-lived. For persistence, attackers reach for the file version. /etc/ld.so.preload holds a list of libraries the loader preloads into every dynamically-linked program that starts, across the whole system, with no environment variable in sight.

/lib/x86_64-linux-gnu/libc_hardening.so

The name is picked to look boring and load-bearing. Now the hook rides inside ls, find, ps, cat, your shell, and every service that restarts. One root-owned file, total coverage. There is a reason the loader trusts this file more than the environment variable: LD_PRELOAD is ignored when a program runs setuid (a binary that runs with its owner's privileges, such as sudo or passwd), so an attacker cannot use the env var to bend sudo. Because only root can write /etc/ld.so.preload, the loader honors it even for those setuid programs. It is the stronger foothold precisely because it is harder to reach.

The tools are now unreliable witnesses

The nasty part is what this does to an investigation. Once readdir is hooked system-wide, the commands you would reach for to find the intruder are running the intruder's code. ls hides the files. ps hides the processes, because ps builds its list by reading /proc (the kernel's live view of running programs, presented as a directory tree) and /proc gets read with the very same hooked readdir. netstat hides the connections, because it reads /proc/net/tcp line by line and the attacker filters those lines too. Ask a hooked tool a question, get the intruder's preferred answer.

This is the core problem of host forensics (examining a possibly-compromised machine for evidence). On a box you suspect, the built-in binaries are themselves suspects. Evidence collected through a subverted userland is evidence the attacker was allowed to edit first.

Getting to ground truth

You beat userland hiding by not asking the hooked tools. Three moves, roughly in order of confidence.

First, direct access by exact name. The readdir hook filters directory listings, but a file or process hider like the one above never touches open, stat, or the read of a path you name outright. A hidden process (with its PID, the number the kernel assigns each running program) vanishes from ps and from ls /proc, yet reading its status file by full path still works, because that read never goes through directory enumeration.

Enumeration failing while a direct read succeeds is the fingerprint of a readdir-style hook. A forensics tool called unhide automates exactly this: it walks the entire process-ID space by direct access and compares that against what ps admits to, then flags the gap.

Second, bring your own binary. LD_PRELOAD and /etc/ld.so.preload only touch dynamically-linked programs, the ones that ask the loader for libc at run time. A statically-linked binary carries its own copy of every function and never invokes the loader, so no hook can reach it. A static busybox (a single small executable that reimplements ls, ps, find, netstat, and dozens more) that you carried in on read-only media reads the real directory.

There it is again, secret_notes.txt, plain as day. Third, cross-check two views that reach the kernel by different roads. netstat reads /proc/net/tcp; ss (the modern socket-listing tool from iproute2) prefers a netlink socket (a kernel messaging channel called sock_diag) and does not parse that text file at all. A hook written to scrub /proc/net/tcp fools netstat and misses ss. A connection that shows in one and not the other is a hook betraying itself. The same logic runs off the host entirely: an audit pipeline (auditd, the Linux kernel's built-in event logger) or an eBPF (extended Berkeley Packet Filter, a way to run small, safe programs inside the kernel to watch events) sensor records the truth in the kernel, before any userland library gets a chance to rewrite it, and ships it somewhere the attacker on this box cannot touch.

So the habit to build is a two-source check. Never trust a single tool's word on a host you doubt. List /etc/ld.so.preload by name, read process environments straight from /proc/<pid>/environ for a stray LD_PRELOAD, and confirm anything a dynamic tool tells you against a static binary you brought or a kernel-level feed you control. When two honest-looking views of the same machine disagree, the disagreement is the finding.

Try this

Work through “Getting to ground truth” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a bad preload line can brick the host. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related