The Value of Threat Hunting

This blog was originally posted on Cisco Security Blog by author Ben Nahorney. Reposted with permission. It can happen to the best of us. You can have robust security software deployed in your environment, and yet a threat slips through. Often it happens at a weak point that you hadn’t considered critical or just overlooked entirely. It can be a humbling experience and something that many security professionals, while loath to admit, have faced. What follows is a cautionary tale, but one with a silver lining. It makes the case for threat hunting: A security practice where you look for threats that managed to get past your defenses and have hidden themselves within your environment. It’s a topic that we’re highlighting in our latest report in the Cisco Cybersecurity Series, Hunting for Hidden Threats: Incorporating Threat Hunting Into Your Security Program. Because while no one wants to be caught out by a threat, in today’s threat landscape, it can happen. 

Meet John

It started at home one evening. While watching Netflix, John—whose name has been changed to protect the innocent—noticed an unusual amount of screen tearing. He was using a home theater PC (HTPC) that he had built, and it appeared the device was overtaxed by the stream. It had served him well over the years, but he figured the HTPC was just getting old and began to think about replacing it. What he didn’t consider was that cryptomining was taking place in the background. Unbeknownst to John, a threat had made it into his network without being detected. However, its presence was starting to exhibit side effects. While such behaviors can be explained by other causes, this is a great place for a threat hunt. A hunt is best begun by testing a suspicion or theory. For example, in looking at systems that exhibit screen tearing—could it be cryptomining? In a larger network, you may have users reporting strange issues like this, which can serve as the basis for a hunt. Computers turning on in the middle of the night—could it be a threat phoning home? Upload speeds spiking for short periods—is it data exfiltration? A periodically unreachable web server—DDoS activity? All of these are good starting points.Each of these activities could be explained away by other, non-malicious factors. However, threat hunting requires a more balanced approach: It’s best not to think that every oddity is caused by malware, but it’s also important not to dismiss it too quickly.John’s thinking fell on the side of the latter. His security implementations seemed adequate for a small, home network. He had a router with a firewall that included deep packet inspection (DPI), the HTPC was on a different subnet from devices it had no business talking to, and endpoint protections were in place and up-to-date. Well, in place on all but this HTPC.This was a critical error. The HTPC was running Linux and John had fallen prey to security through obscurity thinking. It’s a situation where he needed to implement a new security policy within his network to cover Linux PCs.  

The goal of threat hunting

This is in line with the overarching goal of threat hunting. It’s not just about uncovering threats, but also implementing policies and playbooks to shore up your security posture. In fact, some of the most successful hunts may not uncover a threat at all. Rather, they identify a weakness in the environment that needs to be addressed. John wishes that he could say he became suspicious and started a threat hunting investigation for cryptomining. However, since nothing was flagging this as cryptomining, he wasn’t, and he didn’t. This is why having adequate logging enabled is critical. You can’t detect what you can’t see, and without logging or other monitoring tools turned on and reporting on the systems within your environment, it’s difficult to accurately assess your exposure.The truth is that fortune played a part in identifying the threat. Being a new Cisco employee, John had the opportunity to roll out Cisco Umbrella on his home network. After switching his DNS settings over to the Umbrella servers, and checking the logs after about a day, the presence of a threat was clear. Umbrella detected activity from within his network that was attempting to connect to known cryptomining sites.Cryptomining events (data taken from Cisco Umbrella) 

After the hunt

Since a threat had been identified, this is the point where a hunt began to transition into a cleanup. John quickly grabbed a Linux antivirus scanner, installed it, and ran a scan. The results came back with six separate cryptomining installations, sprinkled around the home folder and the browser’s temp folder. John zeroed out each file’s permissions and the cryptomining events disappeared. Even better, the screen tearing was gone. After a threat hunt, it’s important to get policies in place to prevent the threat from returning, as well as create a playbook or automation to check in the future. In John’s case, Umbrella took care of the latter. To shore up the HTPC, John formatted the entire system (to be safe), installed a more security-focused Linux distribution, and installed AMP for Endpoints. When discovering a threat during a hunt, it’s also important to cross-check other systems for signs of similar activity. Gather indicators of compromise (IoCs), such as the hash values of the cryptomining files, and check for their presence on other systems.One interesting side note: While John was confident he was cryptomining-free at this stage, he took the six cryptomining files and ran them through VirusTotal. Each file came back with slightly different results, but was detected by the generic signatures of 5-6 antivirus engines, further solidifying their malicious classification.What surprised John was that the next day, when he logged into his AMP dashboard, he discovered that AMP had quarantined six files on his Windows PC. John had pulled the cryptomining files off of the HTPC, zipped them up, and had planned to archive them. However, because he scanned these files through VirusTotal, AMP was automatically updated. The files, having previously been flagged as “unknown,” were now known to be malicious. AMP pulled them out of his archive and quarantined them—an interesting turn of events that highlights the power of an integrated security solution.  

Lessons learned

As a result of this experience, John’s security posture has improved. However, he’s not naive enough to think that he’s 100 percent secure. Since the incident, John has checked his environment for published IoCs using tools like Cisco Threat Response, and has enabled further logging to be able to check for unusual activity. Whether we’ll admit it or not, the fact is John could be either you or me. Things can get through our defenses. And the consequences of having a hidden threat on a large network can reach much further than cryptomining software on a single PC. This is why threat hunting is such an important tool in any security arsenal today. Want to learn more about threat hunting? Check out our latest paper on the topic, Hunting for Hidden Threats: Incorporating Threat Hunting Into Your Security Program. In it, we go further into threat hunting, explaining in more detail what it is, how it compares to other security disciplines, and how to kick it off within your organization. Download your copy today!