Kernel rootkits & detection
When you cannot trust the kernel.
Every tool you trust to tell you what a Linux box is doing asks the same office for its answers. ps wants the list of running programs. ls wants the files in a directory. ss wants the open network connections. None of them look at the hardware themselves. They all phone the kernel (the core part of the operating system that talks to the hardware and hands out its resources) and repeat back whatever it says. A kernel rootkit is a bribe paid to that office. Once it is in place the kernel keeps answering, everything still sounds normal, and every answer about the attacker is a lie.
You met a weaker version of this trick earlier with LD_PRELOAD (a setting that makes a program load an attacker's library first so its own functions get quietly replaced). That fools one program at a time, and only programs that are dynamically linked. A statically linked binary, or one you compile fresh, walks right past it. A kernel rootkit sits one level deeper. It changes the answer at the source, so every program gets the same lie, whether it is ps, your monitoring agent, or a tool you copied onto the box thirty seconds ago. That is why it is the hardest thing on this list to find from the machine it lives on, and why the hardening course kept hammering on who is allowed to load kernel code.
What a kernel rootkit actually does
Most kernel rootkits arrive as a loadable kernel module, or LKM (code you can plug into a running kernel without rebooting). Legitimate ones are everywhere. Your network card's driver is an LKM. A malicious one loads the same way and then rewrites how the kernel answers a few specific questions. The usual move is intercepting system calls (the requests every program makes to get real work done, like getdents64 to list a directory or openat to open a file). When ls asks the kernel to list /tmp, the rootkit lets the honest listing happen, then removes its own files from the result before handing it back. Ask for the process list, and the attacker's process is filtered out. Ask for open ports, and the backdoor's connection is scrubbed. The kernel does the work truthfully and then edits the receipt.
On an old kernel this was done by overwriting the syscall table, the kernel's phone directory of which function handles each system call. Modern kernels keep that table read-only, so today's rootkits reach it by flipping the processor's write-protect bit for a moment, or they skip the table entirely and hook the functions themselves using ftrace or kprobes, the kernel's own built-in tracing machinery. The result is the same either way. Something now runs between the honest kernel function and the answer that leaves the building.
First signs: a kernel that admits it is dirty
Start with the cheap checks, knowing every one of them can be beaten by a good rootkit, because you are asking the suspect about itself. The most honest signal the kernel gives is the taint flag. The kernel keeps a private note about anything that might make it behave in unsupported ways, and loading out-of-tree or unsigned module code sets specific bits in that note.
That 12288 is not random. It is two bits added together: 4096 means an out-of-tree module was loaded (code that did not ship with this kernel), and 8192 means an unsigned module was loaded. A stock server running only signed, in-tree modules reads 0 here. Anything else is worth explaining before you go home. The taint flag will not name the attacker, but a non-zero value on a box you believed was clean is a thread to pull.
Next, look at what claims to be loaded.
Why lsmod will not save you
Here is the catch that trips people up. lsmod is only a pretty-printer for /proc/modules, and both read the exact same linked list inside the kernel. A competent rootkit's first act is to unlink itself from that list. One line of code, and it vanishes from lsmod, from /proc/modules, and usually from /sys/module/ too. So a module you can see may be worth investigating, but a module you cannot see proves nothing. Comparing lsmod against /proc/modules and hoping for a mismatch is a myth. They drink from the same well. The whole reason detection is hard is that the thing you would reach for to find the rootkit is the thing the rootkit already owns.
Detection is a cross-view problem
So you stop asking the machine about itself and start checking its story against sources it does not control. Three views work, and each one gets stronger the further it sits from the compromised kernel.
The network view. The host swears there is no connection on port 4444. Your firewall logs, a network tap, or a flow sensor on a different machine see packets moving to and from that box on 4444 every night at 2 a.m. The kernel can lie to the programs sitting on top of it. It cannot un-send a packet that a switch already forwarded. A connection that exists on the wire but never appears in ss output is one of the loudest rootkit tells there is.
The disk view. Power the box down, attach its disk to a trusted machine, or boot it from a known-good rescue USB, and list the same directories the live host showed you. Files the running system hid from ls are sitting right there when a clean kernel reads the disk. The rootkit only works while its own kernel is running. Read the filesystem from outside and its edits to reality evaporate.
The memory view, which is the strongest. Capture the machine's RAM to a file and analyze it offline, where the rootkit's code is not running to interfere with you.
Volatility (a memory forensics tool) walks the kernel's structures inside that snapshot the way the kernel itself would, except nothing is alive in a file to filter the results. linux.hidden_modules carves module structures straight out of memory and lists any that are missing from the official module list, which is exactly where a self-hidden rootkit gets caught. Related plugins expose the other lies: linux.check_syscall flags system call handlers that point somewhere other than the real kernel function, and comparing linux.pslist (which walks the process list) against linux.psscan (which scans memory for process structures the list conveniently forgot) surfaces a hidden process. On a live box you can get a weaker version of that same cross-check with unhide, which brute-forces every possible process ID and compares what the kernel admits to against what actually answers.
Signature scanners like chkrootkit and rkhunter still have a place. They know the fingerprints of common kits and will flag a lazy attacker in seconds. They will also miss anything custom, and they run on the possibly-compromised host, so treat a clean result as weak evidence, not a clean bill of health.
Close the door before anyone walks through it
Finding a kernel rootkit is hard. Stopping one from loading is not, and it is the same module control you set up in the hardening course. Three locks, strongest first.
Turn on module signature enforcement so the kernel refuses to load any module that is not signed by a key it trusts. An attacker with full root can write a malicious .ko file to disk all day; if the kernel will not load unsigned code, that file is inert. Check two things: whether enforcement is on, and which kernel lockdown mode you are in (lockdown is a kernel-wide setting that blocks the paths root would otherwise use to reach kernel memory).
Y means the kernel rejects unsigned modules. The bracketed word is the active lockdown mode: integrity refuses unsigned modules and closes the side doors into kernel memory, and turning on Secure Boot sets this for you automatically. If that first line says N and lockdown reads [none], an unsigned rootkit loads with nothing standing in its way.
If a machine has finished booting and never needs another module, take the ability away entirely.
That switch is one-way. Once it is 1, nothing loads another module until the next reboot, not even root, which is the entire point. Flip it only after everything the box needs is already in place, or you will lock out a driver you actually wanted. A small systemd unit ordered late in boot is the usual home for it.
[Unit]Description=Disable kernel module loading after bootAfter=multi-user.target[Service]Type=oneshotExecStart=/sbin/sysctl -w kernel.modules_disabled=1RemainAfterExit=yes[Install]WantedBy=multi-user.target
Whatever else you do, record every attempt to load or unload kernel code, so a load you did not authorize leaves a mark you can alert on. Drop a rule in the audit daemon (auditd, the Linux service that records security-relevant events) covering all three of the module syscalls.
-a always,exit -F arch=b64 -S init_module -S finit_module -S delete_module -k module_ops
lsmod looks normal and its own ss shows nothing on port 4444, yet a flow sensor on a different machine records traffic to that box on 4444 every night. Which reading fits best?ss and lsmod print, but the frames were on the network before it had any chance to edit the story.lsmod prints, so an empty result is the expected outcome either way.cat /proc/sys/kernel/tainted returns 12288. What have you actually learned?When you have set the locks, prove them in one place. cat /sys/module/module/parameters/sig_enforce should say Y, cat /proc/sys/kernel/tainted should say 0, and sudo modprobe anything on a finished box should come back Operation not permitted. Three answers, and the deepest hiding place on the machine is shut before anyone tries to use it.
Try this
Work through “Close the door before anyone walks through it” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: a rootkitted kernel cannot be cleaned in place. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.