Enterprise Security Leadership: Measurement & Metrics for Cybersecurity

00:00

This course is powered by Sai Buri for teams. Security leaders encounter new workforce challenges daily Cyber A for teams helps organizations build a cybersecurity enabled workforce to tackle new challenges, handles security incidents and prevent data breaches. If you'd like to learn more and see how other security leaders like yourself

00:19

are utilizing Sai Buri for teams,

00:21

you can schedule of free demo in the link below or search business in the navigation bar.

00:27

I'm gonna be asking a lot of questions, imposing a lot of issues. I hope today there's a little bit more active for you, not necessarily being active. You know, engaging a bunch of conversations with too many of you toe have a just a general conversation here. But,

00:46

um, if you had pencil paper, you know, for your own use as I pose some questions, it be good for you to challenge yourself

00:53

and asked whether some of the things that we're talking about would be,

00:58

um, something You're comfortable and you know, the answer to Andi or something you don't. And then you know we can You can try and rectify that.

01:07

I wanna re emphasize that this is a course in executive leadership. So this is not a tech course and notice, measurement and metrics

01:17

you're going to see our emphasis here is on two things. One is

01:22

how to communicate

01:23

with executive teams when they're asking quantitative type questions like, What should we be doing?

01:30

So that's the emphasis here versus this being, you know, me, putting on my computer scientists hat and talking about the latest in quantitative metrics. It's not going to be The latter is a little bit. We'll show you a couple of very interesting things that I suspect you haven't seen before.

01:49

But But that's the idea here, right? This is not there's not a course in America methods.

01:55

If it was, I'd be happy doing it. But But this is different. This is communication. So make sure you're steering your

02:04

your attention toward how you would do things. And some of this stuff like you talk a little bit about cyber risk,

02:10

is actually quite trivial. But communicating it to an executive is not trivial. I've tried everything.

02:17

I'm gonna show you a couple things that have worked for May maybe work for you because s So this is, um maybe the more accurate description of this session. It was our Brad Pitt here. Looking at our session

02:30

would be communicating metrics

02:34

two executives about cyber security. Okay, so the word communication,

02:38

I think is pretty important here, perhaps as important

02:43

as the material that we will go through.

02:45

Okay, so those were just a couple of general announcements. Other than that, I may have to quit a few minutes early here, Rob. So we'll do. We'll go about 55 0 minutes

02:55

on this session.

02:58

You have a couple minutes to get to your next meeting as I need to get to get to my next so s. So let's go ahead and get started. So, really,

03:06

where a lot of this comes is that there are questions

03:09

that come from board members, executives, practitioners, whomever.

03:15

And when these take on a somewhat quantitative nature,

03:20

then it's up to you to make a decision how to deal with it, right? Like the canonical question is, what does it mean to get attacked? Like, how many times do we get attacked today? And one CSO might say we get attacked a million times per minute

03:36

and another might say we got attacked three times last year

03:39

and they might both be right.

03:42

But that can't be right If there was a common discourse right to accountants.

03:47

Can't report,

03:50

you know, numbers. Like in a public company. You can't report things in a way that just make something up

03:57

where you say we did, you know, 10 million quality units of sale last year. And, you know, what's the quality unit? It was something we made up. You know, we may not be selling

04:08

getting money, but we're getting other benefits. So we think that's a better way to build our balance sheet

04:14

You couldn't do that is the CFL. But you can, and many do, in fact, follow that approach in security where we make up a framework, and then we follow it. So keep that in mind that this is the Wild West here.

04:29

Now, again, with your pencil paper, I'm going to ask the question here, and I want you to see if you have some ideas here. So first is

04:35

let's say you're asked by your CEO, as I've been asked a million times.

04:41

How secure are we? And I made the question well formed. How secure are we against meaningful cyberattacks? Nobody's going to say it that way, but that's what they mean.

04:49

So it might be Hey, you know how we doing? How secure are we here?

04:55

Hey, we stopping all the attacks and it might be over a drink. It might be during a meeting. It might be in the elevator.

05:00

It could be any number of circumstances where you're being asked this question,

05:05

and you can either provide a quantitative or qualitative answer, depending on the circumstance.

05:14

So let's say you're at a board meeting

05:16

and the question comes,

05:18

How secure are way? And there's the expectation that there would be something quantitative provided back.

05:26

Well, there's a few things you can do here. You can provide some sort of score. Maybe there's a one through 10 score you've prearranged.

05:32

Maybe you're working with companies like security scorecard or, you know, a company like that, and you have a number you say. Well,

05:40

according to the third party, were on 8.2. Here's what that means. Um, that's one approach. You take something another, you give a very subjective answer. That was always my approach is to see. So hey,

05:53

you know, remember Randall Stephenson What was called me Doc PhD. Hey, Doc, how we doing? Then I'd say, Well, we're doing

06:00

There's good as could be expected or something like that.

06:04

You know, that's a qualitative answer where you're avoiding the quantitative and that sends a message. Meaning,

06:12

you know, if there was something more specific that I trusted that I could give you that was a number I would.

06:17

But by saying as good as could be expected or gosh, we're doing our best or some qualitative thing. You're sending a message

06:25

that quantitative responses to that answer may not be very useful or even appropriate.

06:31

Um, you could do something like, you know, ah, positional thing where it's not numeric, but it's We're definitely better than this, but not as good as that. You know, some people would say, Well, we're certainly better than, uh, you know, government agencies. Ha. But where we may not be as good as some of the bigger banks, you know, it's something like that

06:51

puts things in

06:51

perspective,

06:54

but you,

06:55

a za leader, need to be able to answer this question. This is the first question that you should have an answer to

07:03

every day of the week You should always have an answer to this because you're going to be asked this question

07:09

very opportune times. Um, so it should be on top of mind. So that's number one second question that comes pretty frequently is how we doing now Compared to some other time and again, I wrote this in a

07:23

the logical fashions and how you might hear it from a board member. How does our security this year compared to security last year?

07:30

Perfectly reasonable question.

07:31

So again, you have some options. This is where

07:36

numeric scales make this so much easier question to answer. So if you're

07:43

score this year is 10.3 and your score last year was 16.4. Well, if higher is better than what's going on, why are we getting worse?

07:54

Um, if you answer Oh, we're doing better.

07:58

And yet everyone on the team knows that there was a big ransomware attack or phishing attack on a PT.

08:05

That was just a nightmare. Then there. Then there's this cognitive dissonance here where you're saying one thing. But the sensibility of the executive team is different.

08:16

So how do we do that? This is sort of trending right. How we doing this year versus last year is probably the second thing

08:24

that a security executive have better have an idea of. What direction are we heading? The first one is where are we? Point in time. How we doing today?

08:33

For good. Okay, The second one is

08:39

how where we headed.

08:41

Are things getting better? They're getting worse. And how do you measure? What do you what do your tools for Measuring. How do you come up with,

08:48

you know, like our protections better Has the threat gone down? Have the protections got better and the threats gotten worse? Have the protection got better? The threats gone down the reverse, the threats gone up, protections gone down. What? What is it?

09:03

And can you back it up? You know, do you have some numbers to back this up? So this is a really important second question. As you can imagine, this one chart is important for an executive as we build it. This would be the one to print out and having your back pocket and make sure you have answers. Here's the third one.

09:20

Are we spending the right amount?

09:24

We're saying, Hey, we

09:24

it's our budget, right? You need more. Are we giving you too much?

09:31

Well, what do you guys? Not too many people would say. Yes, I'm getting too much.

09:35

But how do you figure out the correct amount?

09:39

Like, let's say you're you do it based on revenue. That's what most people would start with.

09:45

They would say, Well,

09:46

um,

09:48

we're a

09:50

$1.5 billion company.

09:54

Okay, so 10% of that, I know he's gonna send 10% on security. But if they did the $150 million then they're gonna happen. Even I t Yeah, I could spend

10:05

5% would be 75 million. Now, sometimes you read in the trade press

10:11

that companies would be wise to spend 5% of the revenue on I t.

10:16

I guess if you're

10:18

filling through digital transformation and you're trying to automate

10:22

than spending $75 million if you are $1.5 billion company,

10:28

um,

10:30

maybe that doesn't Doesn't

10:33

That wouldn't face the finance team. I mean, 1.5 billion is 1500 million.

10:39

And if you're only going to spend, you're gonna spend 75 of that million on I t that makes sense. And then a lot of people would say Spend 5% on, um, I t that Maybe you should be spending 1% on cybersecurity, and 1% would be,

10:54

I don't know, 15 million or something. So,

10:58

um

10:58

so these are the things you need to have in your head? You know? What? What are we? What should we be spending? What, like Let's say you decided it was 15 million and 1%

11:09

but your budget is

11:11

$500,000.

11:13

Um,

11:15

that's not gonna be right. You know, that seems too low. But I don't know, maybe you have it scattered. Perhaps cybersecurity is distributed across the organization. And if you went in and did an aggregate

11:28

collection of what's actually being spent as it's distributed across the business,

11:33

maybe that's good. For example, if I said what do you spend in quality?

11:37

What does the company spending quality? Would you be better off having a centralized quality department the way we used to do in the 19 eighties? When you

11:46

patrol trying to become more Japanese, like this quality group,

11:52

build a car and then go through the quality thing where you fix it

11:54

where we be better embedding quality into what everybody does.

11:58

So maybe you believe in betting security into what everybody does is better. And you shouldn't have a big security team, right? Maybe that's how you what you believe

12:07

or you believe. No, that's nonsense. You've been very strong, powerful security team. That's these air, all really important questions that an executive doing cybersecurity should have an answer to

12:20

so again for me. I always believed in both, I think, strong, centralized management and capable, decentralized operations. And and that's how I would answer a question like this. I would know and always know what our budget is. Where I believe it is,

12:37

what the trending is. We're like these air three things so far

12:41

that you should have answers to and they are metrics.

12:45

This is not something you know. They're not asking your opinion about something here. This has to be based on data. Measurement is the act of collecting data. Metrics is the process of interpreting that data. Do you follow? Let me say it again.

13:01

Measurement is when you collect data

13:03

and any dummy can do that. Automation could do that. Your sim does that

13:07

metrics involves interpretation. Okay, that's where you set a goal or you make some determination or you can say something about the data. That's what we mean.

13:20

We're performing metrics where it's a verb or were interpreting metrics where it's then and now. Okay, let's look at the next one.

13:28

This famous board question

13:31

how do we do against our competitors? Are peers better? Are we better or worse?

13:35

You give me all these numbers,

13:37

Um,

13:39

and I'm gonna get to the chat questions. Here's a lot of really good questions, so I'll pause it. I'm gonna do one more than I'll pause at the end of this chart. So keep the questions coming. Um, how do we compare it against peer groups? This is really important,

13:54

and this is goes to the philosophy

13:56

that a security executive needs toe have

14:00

when building a program. So here your three options

14:03

when you build a security program and this should be something that's discussed with senior executive team, the board, if you're interviewing

14:11

for a security position here, is that this is important, you'd want to know the answer to the following question. You'd say, Are we where we trying to take the program, Mr Answers, one would be

14:22

bare minimum. It's like, Look,

14:26

remember that we sell hammers at Home Depot comment that a lot of the former employees said He said a lot.

14:33

What? Nobody's coming here because of our cybersecurity with not a differentiator for us.

14:39

We just want to do the minimum so that we don't get hacked, that we're doing acceptable cybersecurity that we're doing, you know, we're being responsible. We're not going to spend an extra penny

14:50

where unnecessary we sell hammers. I think that, frankly, the airline industry is a little bit like that. Like the airline industry.

14:58

I know they don't wanna be safe, but you know you're not gonna fly Delta because you think they're less likely to crash than American. So what happens is they all arrive at a reasonable safety approach and everybody follows that,

15:13

and you spend more on it. Well, I mean, I hope they would, but I don't think they dio. I think they just all agree on what's best practice, and nobody has an advantage that way. But also nobody falls below some acceptable minimum,

15:26

so that so meeting the minimum would be one answer. Another would be. I wanna be right in the middle of the pack.

15:31

I don't want to be the worst. I don't wanna be the best. If there's 10 companies, let's say there's 11 companies in our pack Then I want five on top of me and five below. I wanna be in the middle somewhere. I wanna be both average and median.

15:46

That's fine. That means you. That's good. You don't want it to be a problem where you don't want to overspend there. The third answer would be I must be the best.

15:56

We want this to be a differentiator. We want cyber security to be

16:02

the thing that drives people to what we dio. You think like some cloud companies, um,

16:07

you know, would want to move in that direction where you say are differentiators. Our ability to protect resource is

16:15

so it must be the best program. If you're a military organization, you better have an amazing cybersecurity program. It can't be middle of the pack. You can't say we're going to be the National Security Agency. And they were like our security to be somewhere in the middle or bare minimums. Ridiculous. So one of those three things.

16:34

So this question would be measuring the distance from where you are

16:41

against what you had intended.

16:44

So if you say we're at the bottom of our peer group but we're

16:48

within

16:49

the where we targeted were meet all our certifications.

16:55

But we decided that because we sell coconuts for a living,

16:59

we don't need to have great cybersecurity. So we just do the minimum. We spend our extra money getting nicer coconuts, Z that'd be fine. Then how do you compare against peer groups? Who cares were lower. Everybody might clap that you're not wasting money. You get the point.

17:14

So number four is one that you want to answer contextually to. The decision that you've made is an executive team. And then number five is always about transferring. Right now this most of you look at this and if we do word association

17:27

or phrase association, you'd say risk transfer and you run immediately to cyber insurance. Right? That's what most people would equate with the process of transferring risk,

17:37

um,

17:38

cyber risk.

17:40

And yes, that is an important consideration. But ultimately, everything you dio

17:45

is a risk transfer. But usually when you evolve third parties that's important. For example, if you hire a manage security firm,

17:53

you can think of that as a risk transfer. All. I understand that it's operational transfer because you're taking advantage of their owners. Economics

18:03

in building common reusable, unified infrastructure multi tenant

18:10

on. Do you benefit from that, right? If

18:14

you know a large like Trustwave or if you know Broadcom or affects censure or 18 t they, they say, Hey, you could build your own sock or you use ours.

18:23

Well, I mean, just a simple kind of cartoonist level the facility, the air conditioning, the building that the the chairs, the people you're sharing that you're paying for that. So there could be some significant reductions just because you're sharing. But transferring risk is all about. By involving others,

18:44

I somehow dilute my ability. Now for insurance.

18:48

There's really just four questions. You can write this down, and I've spent a lot of time with some experts in this area. My friend Anthony Belfour, from a on in other places, have helped me learn on. I've also worked on and signed and been involved in several policies. There's all this complication but comes at the four things that I ever cared about

19:07

in looking at this, and I looked at a bunch of number one.

19:12

What's the coverage? What's how much to use the amount, And that's always hard to figure out, right? It's sometimes it's per case per unit for this, and you're like what

19:22

you need to understand, like what is covered? You know, what's the amount? How high's ago? And for what? So that's number one. Number two is what's the deductible? There's all kinds of weird ways they write that

19:33

self insurance option or what it's like. What? Just tell me what my deductible is like. What's the coverage of deductible when you can write, write on a piece of paper?

19:44

$50 million coverage,

19:45

$10 million the death or whatever it ISS. But you should write that. Then that's to the number three is

19:52

what are the premiums? And it's amazing how maney CSOs don't know because the insurance just paid by the finance team.

19:59

So we've been asked, um,

20:00

as if it didn't matter. And I think that's one of the reasons cyber insurance has been popular

20:07

because, he says, don't pay the premiums. If you're paying those premiums every year, you're gonna know how much that is. Let's say it's $5 million for the $50 million policy,

20:15

which would rather have a full out Splunk um, implementation for five million or insurance policy and say men and again, I know that's a high premium. It's probably not gonna be that high,

20:26

but

20:26

then you have to question, let's say you getting $50 million of coverage and the premiums air million dollars a year.

20:33

Wouldn't you want to ask yourself?

20:37

I would have to do

20:40

50 years.

20:42

You know of paying them a million

20:45

to cover one payout

20:48

in 50 years and chances of me filing a claim in the next

20:52

50. Using pretty high,

20:55

you have to ask yourself, what in the world is going on? And that leads to the fourth point? And that's what are the terms.

21:00

You know what's included. What if government finds included? Do I You know what's been your experience? What payouts have you may give me an idea.

21:10

Have you never paid out to falls the amount again? I'm just asking

21:15

these air like, see so type questions

21:18

where an insurance expert might ask something else. These are the things you should be asking. But the point is that how are we transferring? You should have some concept of both insurance and also other things. You're doing a transfer. So let me take a minute. Is a bunch of questions here in the chat. Let me see if I could go through them.

21:37

Js is saying Organizational cyber risk rating comes in two flavors with access to internal pure Estelle.

21:44

That's right. But when you say pure external nowadays with zero trust

21:48

internal and external could be kind of confusing, you're exactly right. This is the domain, but I think increasingly we'll see a blurred distinction between internal and external. I think your your your point is right, but I think over time it's just gonna be

22:03

whatever date is relevant. So that's a very good point, Jazz Daniel saying. Could we share the president? You have the Sai Buri. People will make sure we get that up. Thio Js. I'll say what they did propose to collect and process to answer these questions again. That's kind of up to you. That's part of your program.

22:22

Most people would have a GS GRC program

22:26

where you're collecting information collecting stats collecting compliance. And they also have some sort of data analytic program like their sin would be at the root

22:37

of something where the hunt team in the sock analysts and the manage security teams who do so then you have metrics from that. So there's kind of these two types of data compliance data and cybersecurity live operational metrics, and you probably wanna put those two things together.

22:52

Christopher asking. So how can you work? Uh, so how do you work into these types of questions of functional level of protection? Security program delivers Your organization is a large amount of cognitive dissonance already because most people would, um, as an expert, How would you handle on organization, moving from primarily us to clad

23:12

thes air? Very common questions. What Christopher's describing

23:17

is that work is complicated thing. There's a lot going on. You're moving around. It's global. It's not domestic.

23:25

These air, not simple kinds of things. But I'm just telling you, is an executive. Trust me,

23:32

the conversation that you will have with other executives and with boards does not take into account or even allow you to go there. In terms of complexity. You can't you can't have the CEO or the head of your audit committee come to you and say, How secure are we? And you say

23:52

That's a complicated question. You know, we're a complex business. We have both us and non us. We're moving this to that. We've got that. That's not the way it works. I wish I could tell you that it did.

24:04

It doesn't work that way.

24:07

It's something that comes with the job of becoming an executive. I learned the hard way. I'm a I'm a scientist. That's what I dio my

24:17

as a scientist.

24:19

We're trained to couch our answers when you're at a conference and somebody asks you a question.

24:26

As a trained computer scientists, your instinct is to say the following. You've heard this many times. Well,

24:33

I'm sure there's 100 people in the room here that know more about this than May, and I would defer to them to give you more detail, answer. But in my area of expertise, I can tell you in this narrow area that I believe that the following, but if you ask something broader, you know there's a lot. That's what we learn. A scientists to make sure that we don't go

24:52

in areas that we can't back up with. 10 levels of references and study and detail and so on.

25:00

Executive don't operate that way.

25:03

And it's why scientists sometimes make very poor executives because

25:07

you really do it some point. Have to just gulp a couple of times, Look around the room. A decision has to be made, you know better than the others.

25:18

And we're going to get the sentiment in a minute.

25:21

So make the call.

25:22

How secure are we?

25:25

You can profits by saying, Well, listen, recognize we're a complex business, but my answer would be

25:32

I think we're okay right now.

25:33

Like, if you're not comfortable saying that,

25:37

then you gotta think through your career planning, because you at some point you wanna be a C, so you're gonna be required to make those calls. It's not easy. So, Theis essence of those questions. Or what about this? What about that? What about this? What about that? That's your day job.

25:52

But when you're having discussions, metrics discussions with other executives,

25:56

you gotta figure you got to make the call. Buck stops with you at some point, so see if there. Any other

26:03

questions have popped up. I think we got through most of those.

26:07

Good. So let's go to something that I think is kind of interesting.

26:11

And this is decision making.

26:14

There's a picture of the Federal Reserve from like whatever, like 18, something

26:19

thes look like some people who mean business even though it's a bunch of, you know, middle aged white men. So I guess we hadn't learned diversity. And so they write up the bad. I know they're making a bad decision because they don't have a collage of different input, particularly from women. But be that as it may,

26:37

this is a group

26:40

that I'm going to gas

26:41

would be following something and called a sentiment driven process. And here's what that means. Sentiment means your belief, your gut.

26:51

So when you get groups of people who have a lot of experience,

26:53

they're going to listen to a pitch, listen to some data here from the expert, kicked the expert out that door over years to the door in the back there,

27:03

and then they're going to sit back and say, Hey, Bill, what do you think? And Bill's gonna lean back and go I've been looking at these things for many years.

27:14

My gut tells me that we should do this. What do you think? Yeah, I was thinking the same thing. What do you think?

27:19

My God tell blah, blah, blah. We have a president of the United States right now who manages by God. So this is not an unusual thing

27:27

to be sentiment driven.

27:30

So that that really, is that the opposite is the polar opposite of data driven

27:37

data. Tells you

27:38

that if we

27:40

do the following

27:41

that, here's the Here's what the outcome will be.

27:45

It was salespeople. Many cases air, very data driven, where they'll say

27:49

if I make 100 sales calls,

27:52

that on average results in 20 responses, which results in 10 interviews, which results in one sale. So I know that every 200 calls

28:03

is one to hundreds of its sale

28:04

sales for me are $200,000.

28:08

So I know every call that I make is worth 1000 bucks to May.

28:12

That's a data driven sentence. And when they're on the phone

28:15

Hi. You know, like to sell your stocks and bonds. Oh, not interested. Click OK, no problem. Just made 1000 bucks.

28:23

Hi, I'm a and you called it. You get the point. Like that's data. That's how data driven

28:29

executives

28:30

manage a business sentiment driven go by their god.

28:34

Now, what does this mean in the context of cybersecurity? It's actually something I spent a lot of time on. So here's ah, friend of mine, This is Dan here. Dan is the, um

28:47

I was one of the residents security experts at in Q Tel, which is the venture arm of the Central Intelligence Agency CIA.

28:56

Um, on here he wrote the forward to this really excellent book. Um, if you haven't bought Hubbard and Cyrus UN's book, they write these how to measure anything. Siri's books, they're they're experts and measurement. And they did this book on cybersecurity risk. You could see Stew McClure from formally silence. Uh,

29:17

wrote that on Dan Wrote the forwards a really good book. I recommend you buy it.

29:21

Um,

29:22

I don't know much. It is. I think it's a lot. It's a wildly book. I have the paper copy. I make my graduate students at N Y U by this, um, so I think it's worth reading, and usually these books are just terrible. This one's good.

29:36

But Dan, years about I'd say about a little over a decade ago

29:41

called up a bunch of people that he knew, including May,

29:44

and said that he had this idea and he was working with another friend of ours. Same local Parikh. Many of you may know what exit Wells Fargo does Risk What wonderful guy.

29:53

Um, s O they had this idea.

29:57

So why don't we collectively

30:00

try to aggregate

30:03

the sentiment around cybersecurity of

30:07

on appropriate number of experts who've been doing this a while?

30:12

So it was like a 2030 people, Maybe a little more. But something like that who agreed that way do that. I agreed.

30:19

A lot of people doing it on shadow I t.

30:22

To be honest with you. So getting this approval from your legal to respond every month to a bunch of questions about how you feel about cybersecurity because this was through the idea here was to measure sentiment. What your gut feeling?

30:37

Ah, lot of people didn't think that was gonna pass muster with legal. So if you were the CSO for a large bank,

30:47

uh, you do it from your whatever account Yahoo,

30:53

Gmail or something.

30:56

So we started doing it and and I would send mine in. Others would send you publish it back. And I remember the index started at 1000. It was using a compound exponential the way you would calculate interest. But But it was intended to be sort of like the New York Stock Exchange index,

31:15

where, based on the results that come in,

31:18

the index would move up or down, and it would be context sensitive. So the mathematics is designed so that it's not this choppy thing where you go up and down and there's no memory

31:30

like three bad responses followed by one Good,

31:33

um,

31:34

should be different than three. Good responses followed by another good. They should have different effects and that. So that was the idea.

31:45

Nevertheless,

31:45

um,

31:47

so we started doing it,

31:48

and after a few years, I know McCool was doing much of the work is very manual. Um, I had left 18 18 2016. I retired and went to N Y u I'm research professor in you.

32:00

In addition to doing tag cyber, obviously,

32:05

and Dan called us and said, Hey, would you guys like to take this over. This is kind of a lot of work on. And, you know, we'd like to read. Reinvigorated gets, um, grad students and I raised my hand and said, a man, Totally. So I started a group over at the Center for Cyber Security at N. Y. U

32:22

Recruited Some students recruited some colleagues that we're all pitching in time. It's like more a labor of

32:29

of interest in love than anything.

32:30

Um, and we took over the index and got the thing hosted onto the on the N Y use. So So this is what was some of the factoids about the index.

32:42

So every month we send this thing out to about a couple 100 experts.

32:50

We tend to get

32:51

30 to 40 responses each month.

32:55

It's not always the same one, so the data model is an unusual one. It's like

33:02

each of the experts is a

33:06

an unreliable data source, you know, I'll tell you what I mean by unreliable. I don't mean the person is unreliable. I'm just saying, as a data generator, that source is not reliable to provide an answer. Every month

33:21

we'll do it once in a while, like you get busy, forget you do it. Sometimes you do other times you don't.

33:27

So you get this collage of experts are all reporting in. And by the way, if you're wondering in a minute I'm gonna introduce the idea You're all welcome to contact us.

33:36

If you'd like to be part of this love toe, have you

33:38

what we do like to sort of know just a little bit about what you dio. We don't collect metadata about you. You can send it in anonymously. We don't

33:49

aggregate based on much other than the event, the time and your email address. So if you're Mickey Mouse 123 at gmail dot com And that's all we know about you. But in the beginning, we like to connect with you to make sure that you you have some sort of chops. You know, a reasonably low bar

34:08

acts. I'm lying

34:09

the bars medium. You have to have at least some

34:15

some means for having an opinion here. We don't want to just ask anybody here. These were people who work in the industry,

34:21

but we send it out and we're asking six questions and I didn't want to change the questions because Dan had set these things up

34:30

11 years ago. And if you change the questions, you mess up the science, right? I did eight grad students last semester. Um, who rewrote all the questions beautifully, Wrote a paper. I think it's gonna come out in computers and security. I will make sure you will get the reference if we get

34:49

reference before we finish our class. Um,

34:52

but we asked these questions

34:54

and it's things like, you know,

34:58

Hey, do you think hackers, What you got there? The risk of hackers going up saying same. How about the weapons? You see the malware, the bad stuff coming

35:10

better. Worse. Same.

35:13

Um,

35:14

how about the effects? Do you see this? And you get the point like in these six different areas.

35:20

And then we basically graph the stuff.

35:23

Hey, here's what it looks like. It's this Montagne ycl e. It's really the bottom one here that matters. I don't know why I didn't put something more recent. 8. 19 is just the chart I had in here. If you go to N Y u C c s and you look up Index, you can get to this page

35:42

we're gonna redo the website. It's not the greatest website, but the data is there. We provide the Jason to anybody, wants this open source data. Happy to give you all the

35:51

the numbers that we've collected. There's nothing secret here, Um,

35:55

except the identity of the people reporting.

35:59

But it's if I showed you the the index value, As of you know, October 1st 2020 20.

36:07

It just continues to go up.

36:08

So what does that mean? That means that when you ask a bunch of experts,

36:13

they're just getting monotone Mickley more and more depressed.

36:17

Eso

36:19

You know what that means? We've tried really hard Thio

36:22

line this up against shape like we took CV indexes and lined it up. We

36:30

taken big incidents and lined it up. I had all these students do this work to look for anything

36:37

that might suggest that this index would be useful for making predictions.

36:42

We did find some interesting correlations like, for example,

36:47

right after a big incident, A big public hack. You know, the ones I'm talking about, the ones that

36:53

Brian Krebs would be writing about and that might make the evening news stuff like that. The number of people reporting goes way down the in that right afterwards, like everybody's afraid to report s. That's interesting because it does tend to drag the numbers down because you would think

37:14

that right after a big incident, everybody would want to report higher fear,

37:20

greater risk

37:22

sentiment That's more negative. But the fact that we get fewer results than which is perfectly explainable, I said, a lot of this is shadow. So people doing its shadow, they don't want lawyer the lawyer to know.

37:34

And, man, it was just big hack. You really don't want the lawyer to say they find out that you're telling Dan Gear in Edinburgh so you guys don't tell anybody I'm really depressed

37:45

that zvehr e imperfecta in that sense, I think.

37:51

But that that makes think of worse right? This number would actually be worse, because we're predicting that their answers would be more pessimistic, not more optimistic, because how many times is a bit of big win in cybersecurity? I don't know. Maybe you goto our PSA and there's somebody vendors, they come back all inspired,

38:08

believe that things were better on. Yeah, I think

38:10

nobody ever said that. But if you did, You could imagine maybe somebody would, But you wouldn't be. There wouldn't be this propensity to not report

38:21

because you're inspired. But there would be a propensity to not report if you're fearful.

38:25

So we found some things like that in the data. That's really, really interesting,

38:31

Very indicative of,

38:35

you know, more work that needs to be done in this area. So there's a couple things that we're doing. We're probably gonna launch Cem Parallel in disease

38:43

that that do change some of the questions. Um,

38:46

I need a little bit of money to do that. And there's a couple of companies that have been just wonderful and providing some support to N Y U

38:54

for us. A of Cisco is giving us a little grant to help us do some work.

39:00

My good friend Ray Rothrock, who runs Red Seal, has been a

39:04

wonderful proponent of things. Has helped is a couple of venture capital firms Evolution Equity and Forge Point that they're making some grand. So if you're part of an organization where you think you think when I say big grand, I'm talking, you know, $10,000 but to a bunch of students Thio University level. That's

39:23

like that's really, really leverage. We use the money to host website to run seminars to pay grad students. It's that kind of thing that costs here are not massive. But what I would like to do is a little bit more marketing.

39:37

So maybe some of you have some suggestions on how we can market the thing back there, give you this is, uh,

39:45

a little bit of the kinds of things were like, we're gonna think we're correlating results is of paper that we're writing. We've issued Cem press releases on this, but the question really is, does the collective gut feel the sentiment

39:59

of experts correlate with past, present or future? And we've come to the conclusion that right now

40:06

we really can't draw that conclusion. So look, if you have some interest in participating,

40:12

drop me an email, um,

40:14

and we can talk. This is kind of cool stuff, and this is right down the This is right in the wheelhouse of what we've been talking about earlier. I mean, let's go back.

40:24

This is our really important chart here.

40:28

This is your index card. So you as a cyber executive when somebody asks you,

40:34

What are you thinking about? In terms of metrics,

40:37

these air basically quantitative metrics, that air probably data driven and that are things that you've synthesized into answers that makes sense to you and that drive the discussion to something that is useful. But this is evidence that gut feeling sentiment

40:57

also need to be taken into account. Let me give you a little example of start here,

41:00

but I think you all know that I spent a little time on the board of a bank. I can't tell you how many times,

41:07

you know, big report would come in financial discussion. Detailed, detailed data

41:14

about some decision. It could be an investment decision or whatever. And I'm, you know, on the computer science, I'm trying to keep up. But listen on, look at the data. And to them, it seems so obvious to me. I'm reading it, trying to make sense to it. But I would notice that the after the presenter would finish the presenter beach pushed out the doors would be closed, just the board.

41:35

All the data would be pushed away, things shot, everybody leans back, and then it would be a sentiment discussion so the data would inform the sentiment, but it would be integrated with the combined experience of the executives in there. Let me say that again. Because that's important

41:52

the way decisions are made. My observation around, cybersecurity around, risk information risk around. Just business

42:00

is that data is presented,

42:02

quantified its prison graft. And all this stuff is given in a lot of you on this call. Have put together those power points for the executive team in the boards. And you're probably wondering, how do you do it when you're sitting there, How will you do it? What happens is you synthesize the data.

42:19

You watch it, you absorb it, you read it, you listen

42:22

and then when you're done,

42:24

kick the expert out, close the door, pushed the data pile away, and then integrate sentiment, experience,

42:32

judgment and the data that's been presented into a good decision.

42:37

So when you're presenting toe aboard, keep that in mind. They're going to take your guidance into account, but it's not gonna be, I tell you acts they do. Why? That's not the way most boards work. They're really going to take what you dio into some sort of

42:54

a collage of different consideration.

42:58

So let's see

43:00

why I said we were gonna finish early, but I want to get to these questions here. Eddie. You referring to Max your M score Cyber risk rating scale? Because no, I don't even know that.

43:10

Uh, James, you want to mail me that? I don't I don't even know what that's what that is. I'm ignorant to that. But the rate I've seen, like security scorecard, right? You've seen that are a bit site stuff like that. So, um, that that's the stuff I'm familiar with. Please don't know any service.

43:30

We're rate that. That rate internal accident. Yeah, that's it. There's services that

43:34

that do that.

43:37

Um, Ed, can you send us your email? I'm interested in, But yeah, again, my email is this.

43:44

That's a M. Rosa tag dish cyber dot com. It would be very very. I could send you the paper and you can take a look and then I'd be funded. This is all non profit. So if you dio make some sort of ah grand, I want to get involved. It's all it goes Thio University. So I think that

44:02

that sort of thing is uh,

44:05

is tax deductible? Um,

44:07

don't know exactly how that would work, but it is the donation to a non profit.

44:13

So that's really to the message for today. Really? Is that we think

44:17

metrics measurement

44:20

as practitioners already know what? That you guys know how that works. There's probably people who

44:27

you know quite a bit about that. Looks like somebody here knows about a cyber risk rating scale I never even heard of. So there's a lot of experts here who know the detail. But I wanted to communicate to you today.

44:37

Is that when you're discussing this is an executive This, of course, in leadership, right? It's not, of course, metrics. It's about communicating.

44:45

And that one chart I showed you is really important. That would be the kind of thing that you want to make sure

44:52

you have in your mind answers to those things and start practicing now so that when you do become the sea so you have that a second nature. Ask any see? So you talk thio.

45:01

Um,

45:02

they say, What are the key challenges you have? And they're all going to say it's communicating with the executive team communicating, communicating? That's always the issue.

45:12

So, listen, what I want to do is we're gonna sign off a few minutes early today, but for next week when we do our a m A. I hope you guys will come with lots of really good questions. Uh,

45:23

if you don't, I guess i'll do finger puppets. Um, figure something out.