Detecting and Understanding rootkits - an Introduction and just a little bit more

Understanding Rootkits on Linux: What They Are and How They Hide

When you first install a fresh Linux system, you may feel like a hardened fortress. Services you don’t need are disabled, netfilter rules are tight, and your kernel is up‑to‑date. Yet a well‑designed attacker can still slip past those layers, hide in plain sight, and gain root access. The tool that makes this possible is called a rootkit - a set of Trojanized binaries, hidden processes, or kernel modules that cloak malicious activity.

A rootkit’s purpose is simple: remain undetected while the attacker maintains control. It can alter standard utilities so that commands like ps or ls no longer reveal the attacker’s processes or files. It can intercept system calls, rewrite kernel data structures, or replace entire sections of the kernel to swallow traffic, drop passwords, or exfiltrate data. Because Linux is open source, attackers study its internals in detail and craft very targeted attacks. Knowing the mechanics of these threats is the first step toward detection and mitigation.

Rootkits come in three main flavors. The traditional type replaces user‑space utilities. The LKM (Loadable Kernel Module) variety hooks into the kernel, substituting system calls with malicious code. The newest wave injects itself into the kernel’s memory without even loading as a module, making it extremely hard to spot with standard tools. Each type shares a common theme: they all hide their footprints while granting persistent root access.

When an attacker lands a foothold, they will typically perform the same series of reconnaissance steps: sniff the network, read configuration files, locate privileged accounts, and finally install the rootkit. The presence of a malicious sniffer or an encrypted back‑door is often the first clue. Once installed, the rootkit may hide network sockets, strip log entries, or even tamper with the file system’s permissions to keep its tracks buried. Because many of these operations are invisible to ordinary monitoring tools, the defender must adopt a multi‑layered approach.

From a practical standpoint, think of a rootkit as a “Trojan horse” in the Linux world. It pretends to be a legitimate utility, but when invoked it alters the underlying kernel behaviour. The attacker then leverages the kernel’s privileges to remain hidden while executing privileged actions - creating new users, opening back‑doors, and exfiltrating data. Rootkits evolve quickly, but their core objectives remain the same: stealth, persistence, and privilege escalation.

Why Detection Is Tricky: The Anatomy of a Rootkit’s Stealth

Detecting a rootkit is difficult because it can modify almost every layer of the operating system. A traditional rootkit might replace the binary for netstat so that all malicious connections vanish from the output. An LKM rootkit can hook the sys_execve system call, allowing it to execute malicious binaries while reporting their existence as normal. The most advanced rootkits can even change the addresses of kernel symbols, making standard memory inspection tools mislead the user.

Because of this, many detection utilities produce both false positives and false negatives. False positives arise when legitimate customisations are mistaken for malicious modifications, while false negatives occur when the rootkit’s changes are perfectly indistinguishable from a genuine system state. Consequently, the detection process is more of an art than a science. Successful defenders rely on a combination of signature‑based tools, behavioral monitoring, and manual investigation.

Behavioral analysis is especially valuable. A rootkit that intercepts execve will not only hide its own processes but also any processes it spawns. Monitoring process creation patterns, comparing the output of ps against the kernel’s internal list, and watching for anomalous system call sequences can reveal hidden activity. Likewise, a sniffer that operates silently will produce traffic anomalies or unusual packet patterns that can be spotted by intrusion detection systems like chkrootkit. The tool is lightweight - under 40 KB - and checks for a broad range of known rootkits, including t0rn, Adore, and T.R.K. Installing chkrootkit is straightforward: download, extract, compile, and run. It offers the -r option to point to a mounted root directory, and the -p option to specify a safe binary directory. This dual‑mode operation lets you run the scanner on a live system or on an offline image with confidence that the tools themselves haven’t been compromised.

When scanning a kernel for hidden modules, the kstat utility is invaluable. It reads kernel memory directly through /dev/kmem and compares the list of system calls against the known good state. By taking an initial snapshot of the system call addresses after a fresh install, you can later detect any deviations that hint at LKM rootkits. kstat also supports a -P switch that lists all processes, including hidden ones, enabling you to cross‑check against ps aef output.

Tripwire remains a solid choice for integrity monitoring. By hashing system files and storing those hashes on a read‑only medium, Tripwire can detect unauthorized changes. If you prefer a simpler approach, build your own database of MD5 checksums, sign it with GnuPG, and compare it against the current system state. While Tripwire is feature‑rich, the manual method gives you complete control and reduces reliance on external dependencies.

Network‑level scans provide another angle. Tools like The Sleuth Kit or Netfilter – The Linux firewall subsystem and iptables user‑space tools.