Hedging Your Biases

00:00

says Module one less than six. Hedging your biases

00:06

this lesson. We have two main objectives.

00:09

The first is to go over the importance of collaboratively assessing attack Wrappings

00:15

is going to cover a bit of why some of that Step five is so important from the attack mapping process.

00:21

Then I'm going to get into some about analyst and source biases in some ways to hedge against them.

00:32

So Step five of the process we gave, which is to compare with other analysts, can be really important in doing attack, mapping or, frankly, any other intelligence analysis.

00:43

This can help hedge against analyst biases,

00:47

so I'll be getting into a number of different reasons why different analysts might come up with different answers.

00:55

But everyone has a different set of experiences in background that they're drawing upon as they do intelligence analysis.

01:03

So where one analyst may see non application layer protocol and another may see custom command and control protocol,

01:11

it's important to figure out why these differences exist.

01:15

Be consistent in how you map and apply techniques.

01:19

If other analysts can't review, your mapping is try to make sure that you're at least doing it the same way as you go across different reports and how you're using a given technique,

01:33

so it can also be tempting. And once you're experienced with attack, you'll probably occasionally skip steps in the mapping process.

01:41

You may be going straight to identifying an applicable technique or sub technique.

01:47

You know, maybe you don't need to go through a long process to identify

01:51

fishing spearfishing attachment in a report.

01:55

But it is important to remember that this does increase your bias.

02:00

It's drawing from availability, bias, the techniques that you have in your head that you're already familiar with

02:07

versus the full range of techniques.

02:09

So it's probably something you're eventually gonna work up to doing, but it wants to always be done with a little bit of caution.

02:21

So I'm going to get into one of my favorite areas of cyber threat intelligence,

02:25

which is biases and intelligence reporting.

02:30

I'm going to talk about the specifically in terms of attacked map data, but these exist all throughout Cyber threat intelligence.

02:38

The first thing about biases is it's important for us to recognize that they exist and to understand some of what our biases are

02:47

in cyber threat intelligence.

02:50

I'm going to get into two key types of bias

02:53

in areas like the technique examples that are in attack. And so these these biases exist in the data that the attack team puts out an attack groups and software wrappings as well as work. You do yourself in mapping attack.

03:07

And so the two types I'm going to get into our

03:10

bias introduced by us as consumers

03:14

and bias that's inherent in the types of sources we use.

03:19

Understanding these biases is the critical first step

03:23

and effectively leveraging the data.

03:28

So the first bias is in the set of sources that you use.

03:31

So these are actually the percentages for

03:35

reports that we have in a tax groups and software packages and attacked at mitre dot org.

03:43

The vast majority of the material that we are able to leverage

03:46

is coming from security vendors is mostly coming from incident response.

03:51

In some cases, governments have put out reports things like indictments. Uh, and in some cases there's high quality press reporting that gets into activities that adversaries have done.

04:02

And it's not that there's anything bad with this, but it's important to understand the biases in the specific sources that are making up our intelligence

04:13

and you're set of sources is probably going to look a little bit different than this. Attack has some specific constraints in using free and open source threat intelligence reporting.

04:26

Another set of biases that we're going to have as we go through data

04:31

is novelty and availability, bias

04:35

and as the attack team were absolutely

04:40

hit with novelty bias.

04:43

So somebody is putting out a report. It's talking about, uh, technique. We've seen a bunch before a group we've seen a bunch before, so

04:51

Fuzzy Duck is using power shell again.

04:55

It might not be as interesting for us, and it might not come up to the top of the queue is fast for us adding it to attack.

05:01

Whereas, you know, some brand new technique, we have out transmitted data manipulation. Very few actors do that,

05:10

Uh, we don't have report in on a P T elite, so a P T. Lee, using transmitted data manipulation, is going to be a lot more novel to us, and we just need to be aware of these biases And are they blinding us to important activity?

05:26

As we remember ourselves, we again have that availability bias

05:30

attack is big as hundreds of techniques,

05:32

and they're going to be a subset of techniques that you remember in your head that you know that you're familiar with. You can immediately go and map to,

05:42

and you're more likely to find those techniques and reporting. So it's important to recognize

05:46

if you found a certain frequency of a technique to understand what that may mean in terms of how many times it occurs.

05:58

They're also biases in the sources we use,

06:01

so to get into two availability and visibility,

06:05

the people that are creating the reports for using have their own availability bias.

06:12

They have behaviors they've seen adversaries do before. They understand, as they're looking at their data

06:17

and are able to go in and analyze those behaviors a lot more effectively.

06:23

Whereas there is a wider range of things that adversaries might actually be doing,

06:28

they also have visibility bias. So a lot of the data that we're talking about is coming from incident response, so it's likely that it's only types of data that can be gathered afternoon intrusion that are being included in the reports.

06:42

So there are certain types of behaviors that appear more, depending on the type of sensing and the type of data that you're actually able to use in your reporting.

06:57

It's also victim novelty biases in the reporting we use.

07:00

Victim bias is that some victims are going to be potentially more interesting to be reported upon and so more likely to generate a report. So it could be some particular industry, a really big name, company and incident happening to

07:17

the other way that who the victim is impacts. The reporting is that in a lot of cases, companies are getting permission from victims before reporting on them, even if it is anonymously

07:29

so it can have a big impact which type of industry what kind of reporting requirements they have. If the report even comes out at all based on who the victim is,

07:40

and given that we're working off of, in a lot of cases, free reports,

07:45

there is some novelty bias that we need to watch out for two

07:48

and a lot of cases threat Intelligence reports are coming out of a marketing budget,

07:55

so there is some pressure where if there's a group that's had frequent past reporting,

08:00

maybe it's a little less likely We're going to see a report on this, and we've seen this out in the wild, where

08:07

groups like a P T. 10 there was no reporting on them for a number of years, even though people were seeing them out in the wild. But they just weren't doing anything particularly new and industry

08:18

interesting. Lo and behold, they break into a bunch of service providers, and suddenly there's reporting out there again.

08:24

Whereas it might be more interesting to get a report out on the new group on the block

08:30

a p t. 1338 instead of a PT lead.

08:35

So these aren't bad. They exist. There are things that we need to recognize,

08:41

and there's some strategies we can take for hedging these biases.

08:45

And the first is that Step five that we gave

08:48

collaborate.

08:50

If you're collaborating with others, it can help mitigate, especially your own biases.

08:56

Diversity of thought and diversity, period on your team's makes for stronger teams.

09:03

Justin, calibrate your data sources. Understand how your data is potentially skewed and adjust for that as you work with it.

09:11

Try to work with as diverse a set of sources as you can.

09:15

Uh, and oftentimes the absolute best data is going to be that you gather yourself where you've got full access to all the information around it and have a much better idea of how that data is shaded.

09:28

And finally, in working with all of this,

09:31

we were talking about gaps. We're talking about places you might not see.

09:33

What we do have is an opportunity to prioritize the known.

09:39

Hopefully everything we're talking about are things that we do know

09:43

as opposed to worrying about the unknown.

09:48

It does mean that we may not be able to say that

09:50

spearfishing attachment is more popular than

09:56

a supply chain compromise,

09:58

but it does mean that we can say Okay, we've seen both of these existing out in the wild.

10:07

So I've gone over a couple of things in here, uh, talked about why it's so important to work with other analysts collaboratively assess attack mapping as well as other threat intelligence. And I've gotten into some key types of bias that you're trying to hedge as you do that collaboration and other work with threat intelligence.

10:28

So I've just gotten into the second part

10:31

of this attack for CT journey,

10:35

getting into a number of steps to help you map narrative data to attack.