Detecting Rootkits: User Mode, Kernel Mode, Hypervisor & Firmware

Share and earn Cybytes
Facebook Twitter LinkedIn Email

Rootkit Detection: Techniques and Best Practices

Continuing my discussion of common classes of attacks, this week I’ll be covering rootkits and rootkit detection.

What is a rootkit? You can see it right in the etymology of the word itself; it’s a combination (kit) of software that, once root access is achieved, can carry out stealthy activity of a sort that is usually, but not always, malicious in nature. Rootkit detection therefor can be fairly difficult.

Image for Rootkit Blog

The advantage of having root access is that, as the initial (or root) user you have superuser privileges, giving you full rights to all files and programs on the system. When the host is compromised via this level of access, detection of the rootkit can be thwarted by sophisticated malware, because the tools an analyst might use to detect or resolve the problem might be manipulated by the malware, causing it to yield bogus or incomplete information.

For instance, consider the infamous case of the 2005 Sony BMG rootkit. Initially, Sony authored a package of software for Windows operating systems that had the relatively benign goal of preventing users from copying CD content. To escape detection, the rootkit modified the operating system in such a way as to prevent all files beginning with a particular prefix from being revealed in searches. Its own files then, of course, were given that prefix.

Today, rootkits are typically combined with malware and, as a rule, are much more sophisticated and much less benign than anything Sony imagined. You can classify them by the level of abstraction they occupy on a system:

Rootkit detection: User mode. These operate as user-level tasks, usually by invading existing processes, overwriting application memory with their own contents, or both.

Rootkit detection: Kernel mode. Moving down in abstraction, these rootkits modify the operating system itself (kernel and/or drivers) and are thus substantially harder to detect and eradicate because they can conceal themselves more comprehensively. If you’re using a compromised operating system to look for rootkits, it’s a tool whose results you can’t trust.

Rootkit detection: Hypervisor. Still lower in abstraction are rootkits which modify or replace a hypervisor used to govern virtual machines, each of which is running its own operating system on a shared host. At present, rootkits of this type are not present in the wild, but proof-of-concept examples have been developed.

Rootkit detection: Firmware. These are extraordinarily difficult to address because they are, in a practical sense, embedded in the hardware itself — for instance, a computer BIOS or router firmware — and hence cannot be eliminated even by replacing the operating system completely. In some cases, replacing hardware may be the only plausible solution.

What rootkits all have in common is that they are used to disguise classic malware activity (logging keystrokes, sending credit card numbers to an offsite server, collecting and uploading banking credentials, establishing hidden backdoors for subsequent access by the attacker, etc.) to prevent rootkit detection. And once installed and running, rootkits can lead to disaster as attackers escalate from one application or system to another.

A variety of investigative techniques for rootkit detection

Fortunately, as usual in security, it’s more of an arms race than a one-sided victory. While rootkits have gotten more sophisticated and diverse in nature, so have the tactics and tools available to deal with them. Rootkit detection methods, for instance, include:

A trusted analysis host. To bypass the problem of a compromised OS that may have been modified to conceal a rootkit, simply use a different host, with an OS known to be secure, to do your analysis. A CD, for instance, can’t be overwritten or compromised, and can be booted from.

Signature-based. Much as antiviral solutions look for predictable signatures, byte-level rootkit detection can as well — but this approach is usually only effective against older threats.

Behavior-based. Once installed, rootkits modify system performance in subtle ways that are sometimes detectable. For instance, the timing of API calls sometimes slows, and CPU utilization sometimes climbs. A known clean system with otherwise identical hardware and software can be used to establish baselines for comparison to aid in rootkit detection.

Integrity checking. The idea here is to compare key files or Windows registry entries on a suspect host with clean examples to see if they’ve been changed in any way.

Difference-based. Are installed binaries on a drive identical to their RAM-resident counterparts in a working system? If not, that’s a bad sign (though false positives are also possible).

Memory dumps. Rootkit detection can also be accomplished by analyzing virtual memory dumps because the rootkit hasn’t got a chance to detect and block the analysis — though getting a proper dump may also require separate hardware.

As you can see, it’s apparent that rootkits pose a formidable threat, and not just because they’re sometimes installed by vendors (like Sony) or created by bad actors (like hackers). They are swiftly evolving in incredible ways as researchers break new ground.

For instance, the Jellyfish rootkit was developed last year to show that it’s possible to install a rootkit in a graphic processing unit! This way, it inherits the processing strengths of GPU hardware while also achieving a new and impressive stealth via a logical location in the host that is beyond the purview of most OS-based rootkit detection tools.

And researchers showed last August that it was possible to develop rootkits that exploit a previously undetected flaw in the venerable x86 processor architecture — specifically the System Management Mode, which was added nearly twenty years ago. Staying on top of threats like those, should they be released in the wild, will require security professionals to stay current and may also mandate a new class of security solutions for rootkit detection.

Original Post

Share this post and earn Cybytes
Facebook Twitter LinkedIn Email
About AT&T CyberSecurity
AT&T Cybersecurity’s edge-to-edge technologies provide phenomenal threat intelligence, collaborative defense, security without the seams, and solutions that fit your business. Our unique, collaborative approach integrates best-of-breed technologies with unrivaled network visibility and actionable threat intelligence from AT&T Alien Labs researchers, Security Operations Center analysts, and machine learning – helping to enable our customers around the globe to anticipate and act on threats to protect their business. --
Promoted Content
2018 Threat Intelligence Report
Threat intelligence has become a significant weapon in the fight against cybersecurity threats, and a large majority of organizations have made it a key part of their security programs. This threat intelligence report, produced by Cybersecurity Insiders, explores how organizations are leveraging threat intelligence data, the benefits and most critical features of threat intelligence platforms, and the biggest cyber threats organizations are using their threat intelligence to combat. Download this report now to learn industry findings around threat intelligence.

Our Revolution

We believe Cyber Security training should be free, for everyone, FOREVER. Everyone, everywhere, deserves the OPPORTUNITY to learn, begin and grow a career in this fascinating field. Therefore, Cybrary is a free community where people, companies and training come together to give everyone the ability to collaborate in an open source way that is revolutionizing the cyber security educational experience.

Cybrary On The Go

Get the Cybrary app for Android for online and offline viewing of our lessons.

Get it on Google Play

Support Cybrary

Donate Here to Get This Month's Donor Badge

Skip to toolbar

We recommend always using caution when following any link

Are you sure you want to continue?