A long time ago, in the early days of my journey to Linux kernel rootkits, I came across a Phrack article describing a rootkit detection technique implemented for i386. The article wasn’t new and referred to a vintage Linux kernel dated 2003. Something in that paper caught my attention, although much remained unclear… Ultimately, I decided to implement the anti-rootkit concept described in it for modern systems.

A few words about LKM rootkits

Since ancient times, kernel-level Linux rootkits (aka Loadable Kernel Modules (LKMs)) have been using the same mechanism to avoid detection: they simply remove their module descriptors (struct_module) from the linked list of loaded kernel modules expectably called (modules). This trick prevents them from being displayed in procfs (/proc/modules) and lsmod outputs and also protects from unloading with rmmod (because the kernel thinks that no such module was loaded; accordingly, there is nothing to unload).

After removing themselves from the module list, some rootkits can overwrite certain artifacts in memory to complicate the search for their traces. For example, starting with version 2.5.71, when a structure is removed from the linked list, Linux sets the values ​​of the next and prev in this structure to LIST_POISON1 and LIST_POISON2 (0x00100100 and 0x00200200).

This helps to detect errors, and can also be used to detect dangling descriptors of LKM rootkits that were previously removed from the module list. Of course, a smart rootkit will overwrite such ‘screaming’ values ​​in memory with something less noticeable, thus, bypassing the check. For instance, KoviD LKM created in 2022 hides itself this way.

But even after their removal from the list of modules, rootkits can still be detected in sysfs, specifically in /sys/modules. This pseudofile was even mentioned in the Volatility Framework documentation that is used to analyze memory dumps. Analysis of this file also makes it possible to detect negligent rootkits. The documentation claims that the developers had never encountered a rootkit that deletes itself from both places; however, the above-mentioned KoviD LKM successfully does this. What’s even funnier: the first ever committed Diamorphine version also removes itself not only from the list of modules.

This is how Diamorphine was hiding itself in November 2013

KoviD , on the other hand, uses sysfs_remove_file() and sets its status to MODULE_STATE_UNFORMED. This constant indicates a ‘suspended’ state when the kernel is still in the process of initializing and loading a module; accordingly, it cannot be unloaded without unpredictable and irreversible consequences for the kernel. This trick helps to fool anti-rootkit tools (e.g. rkspotter) that use __module_address() during the enumeration of virtual memory contents (to be discussed below).

Searching for LKM rootkit descriptors in RAM

This paper discusses techniques used to search for rootkits in the RAM of a ‘live’ system and in the kernel virtual address space. In theory, such a search can be performed not only by a kernel module, but also by a hypervisor (which is actually better from the protection rings perspective). But I am going to examine only the first variant since it’s easier in terms of PoC implementation and closer to the original. Also, hash-based malware detection in memory is beyond the scope of this article: my research is focused on aspects applicable specifically to LKM rootkits, not to malware in general. Primarily, it’s about research PoCs.

Original module_hunter

Back in 2003, an article by madsys called Finding hidden kernel modules (the extrem way) was published in Phrack #61 (Linenoise section); it described a technique used to search for LKM rootkits in memory. It was the epoch of kernels 2.2-2.4 and 32-bit computers; now Linux 6.14 is on the horizon, and it’s really difficult to find operational x86-32 hardware (which is currently suitable only for experiments).

So, it was a long time ago, and plenty of features have been removed from the kernel or added to it over the last 20 years. In addition, the kernel is well-known for its unstable internal API, and the original source code from Phrack now predictably refuses to compile for many reasons. But if you understand the very essence of the proposed idea, it can still be successfully implemented for modern hardware.

That Phrack article leaves aside many aspects, and without proper background, the author’s logic behind the solution may not be immediately clear. In general, the proposed method is somewhat similar to wandering through the RAM contents in the dark by touch: you walk through the memory region where module descriptors are allocated, and as soon as something resembling a valid struct module is found, you output the contents of the potential fields in accordance with the known descriptor structure.

For example, you know that a pointer to the init function should be at a certain offset; while sizes of various sections of the loaded module, its current status code, etc., are located at other known offsets. This means that the range of memory values ​​of interest at such offsets is limited, and you can estimate whether the current address can be the beginning of struct module. In other words, you can create special checks to avoid extracting obvious garbage from memory and detect what you are looking for to the maximum.

As you understand, not only internal kernel functions have changed since that Phrack publication, but also a bunch of other structures. The original implementation by madsys had only checked whether the module name field contains normal text. On x86-64, you cannot afford this: the virtual address space is much larger due to a larger number of various possible structures; as a result, a huge amount of data in the memory would satisfy such a simple condition.

Another problem solved in module_hunter is that it checks whether the virtual address that is currently under investigation has a is mapped to physical memory. If it is, then the module can access this address without the risk of panic that would drag the entire system down. This check also has to be reworked since it’s bound to architecture.

rkspotter and problem with __module_address()

I had to find a way to traverse memory without crashing the system. And then I came across the above-mentioned rkspotter. It detects several hiding techniques used by LKM rootkits. This enables it to succeed in its tasks even if one of the methods fails. The problem, however, is that this anti-rootkit employs the __module_address() function that was removed from the list of exported functions in 2020, and isn’t available for modules since Linux 5.4.118.

The idea of rkspotter is as follows: the program traverses the memory region called module mapping space (where LKMs get loaded) and uses this function to find out the module the current address belongs to. For a given address, __module_address() immediately returns a pointer to the respective module descriptor, which makes it possible to get information about an LKM based on a single address. Such operations as checking the virtual address translation validity are performed under the hood.

Of course, I could just try to copy-paste __module_address(), but my leisure interest was to reincarnate madsys’s original idea. What other pitfalls exciting challenges could be on my way to a new implementation?

What has to be fixed

To write a new working tool, I have to examine everything that has changed in the kernel over the past 20 years and is related to dangling LKM descriptors. In other words, I have to fix all the compilation errors that would be encountered along the way.

Overall, my objectives are roughly as follows:

• fix calls to altered kernel APIs. The original code is actually very short, and the only kernel API used in it relates to procfs; so, this task won’t take much time;
• identify fields in struct module that are most suitable for detecting a structure removed from the general list of modules;
• examine and take into account changes in memory management in x86-64 compared to i386; and 
• take into account that on the 64-bit architecture, virtual address space is distributed completely differently, and it’s incomparably greater: 128 TB for the kernel part and the same for the user space vs. 1 GB and 3 GB, respectively, on the 32-bit architecture (by default).

Time to move on to the most exciting part!

Reincarnating module_hunter

Fields, fields…

When you are dealing with x64, a simple module name validity check isn’t enough. Experiments showed that additional checks are required; otherwise, the output would contain too much unnecessary information. False positives suck. After examining the typical contents of various fields, the following checks can be selected:

• memory that corresponds to the state field should contain one of the values ​​valid for this field: MODULE_STATE_LIVE, MODULE_STATE_COMING, MODULE_STATE_GOING, or MODULE_STATE_UNFORMED;
• values ​​in the init and exit fields should point to module mapping space (on x86-64) or be NULL;
• at least one of the init, exit, list.next, and list.prev fields must not be NULL;
• list.next and list.prev are canonical: either NULL or LIST_POISON1/LIST_POISON2 (but this isn’t certain and can be omitted later); and 
• module size (core_layout.size for versions older than 6.4) must be non-zero and a multiple of PAGE_SIZE.

Important: the above list isn’t carved in stone and can be adjusted in the future when more sophisticated rootkits appear or for systems where it’s not sufficient for some reason.

When you search for an efficient combination of checks, false positives displayed by an anti-rootkit due to various garbage in memory look something like this:

[ 5944.082676] nitara2: address module

 . . . 

[ 5944.085392] nitara2: 0xffffffffc011c300 “serio_raw”


[ 5944.085435] nitara2: 0xffffffffc0120ff0 “{”


[ 5944.085512] nitara2: 0xffffffffc0129680 “i2c_piix4”

. . . 

[ 5944.087444] nitara2: 0xffffffffc01fd040 “fb_sys_fops”


[ 5944.089131] nitara2: 0xffffffffc02affb0 “`”,


[ 5944.089342] nitara2: 0xffffffffc02c30c0 “cfg80211”


[ 5944.091874] nitara2: 0xffffffffc03cb700 “kvm”

… 

[ 5944.094188] nitara2: 0xffffffffc04be0c0 “nitara2”


[ 5944.670378] nitara2: end check (total gone 66060288 steps)

Virtual memory in x86

Now when fields required to detect an unbound structure have been identified, it’s time to find out what part of the kernel virtual address space should be searched.

The original module_hunter traversed through the vmalloc area: in the past, memory for modules and their descriptors was allocated there. Its size on x86-32 was only 128 MB, which significantly reduced the number of addresses for enumeration (and time required for this) compared to the full address space of 4 GB (even if you only take its kernel part of 1-2 GB).

In the x86-64 address space, the already mentioned separate virtual memory area allocated for kernel modules is 1520 MB in size (same for 48- and 57-bit addresses); it’s called module mapping space. Its virtual addresses are limited by the MODULES_VADDR and MODULES_END macros. Further experiments showed that both module descriptors and their code with data end up in this area. Great! No need to go through the huge virtual address space whose size is measured in terabytes.

Continue Reading 🡒