Time
1 hour 24 minutes
Difficulty
Beginner
CEU/CPE
1

Video Description

In this segment we look at some of the mechanisms Zeek makes available for managing log streams and organizing and filtering events. We'll use this as an initial introduction to the programming language and review a few live script examples to illustrate how various log management tasks are carried out.

Video Transcription

00:00
In this next section, I'll discuss how to use Bro's logging framework, and it's filters to accomplish several log management task.
00:10
When you first get underway with your bro deployment, you might find yourself overwhelmed with sheer volume of data.
00:15
And that way, the verbosity of those laws can be both a positive and a negative for many people.
00:21
As network traffic analyst, our job is to sift through mounds of network traffic data and look for ways to eliminate data we don't need
00:28
and to try to make more important events more apparent and actionable. Fortunately, Bro's logging Framework has built in functionality to help you deal with what might seem like a log data deluge. At first,
00:40
Bro's filters are applied to Bro's long streams, using handlers for the bro in that event.
00:45
When the next slide, I describe the broken IT event in more detail
00:49
and the remainder of this segment I'll describe several log management tasks, including controlling the fields included in a log stream
00:56
filtering, which records get written to a long stream
00:59
in dynamically routing records to a law extreme based on their content.
01:03
The logging framework is rose mechanism for producing writing to and managing its log out puts this framework supplies the types and functions that broke protocol analyzers used to create their log streams,
01:15
and bro users have access to all of this same functionality.
01:19
Log management actions are performed in Bro's Bro Underscore Knit Event.
01:25
We're going to discuss this and many other events in much more detail later. But for now, just know that Bro executes the bro Underscore Knit event
01:34
once every time it starts.
01:37
Event handlers for this bro Underscore knit event can use functions provided by the logging framework to create,
01:42
disabled or modify bro long streams.
01:46
As I mentioned in the previous segment, not only can bro logging filters be used to duplicate log streams, they provide extensive capabilities for pruning, categorizing and routing log events.
01:57
Each of Bro's log streams has a default filter applied, which is conveniently named default.
02:02
The default filter sets up the log stream as defined by the module that created it, which usually means it would use the tab delimited format and the path name that was specified by the stream that created it.
02:14
So, in the case of the http module,
02:16
the low path or file name is http.
02:21
You can accomplish many log management task just by modifying the default filter. In fact, there's nothing stopping you from using the default filter exclusively.
02:30
But Bro locks can have multiple filters applied
02:32
in the previous segment. I used a custom filter to create a duplicate Jason version of the Con de NS and http looks
02:40
in these situations, the custom filter is applied to the log stream. In addition to the default filter,
02:46
bro filters also provide an easy mechanism for managing what fields appear in the log file.
02:52
This is done by providing a list of field names to either the include or exclude fields of the filter. Well, Seymour about that in the next few slides.
03:00
In addition to those I mentioned, Bro Log filters provide more advanced features through the pred and path funk fields.
03:07
The Pred field allows you to specify bro function that determines whether or not an event will be written to the log stream.
03:13
Alternatively, the path long field allows you to specify a function that determines the name of the log file. Bro will write the event, too.
03:21
Both of these very powerful features that we're going to take a close. Look at
03:24
the first example. I'm going to show us how to disable a log stream
03:30
early on in your bro deployment, you might feel overwhelmed by the amount of log files. Or maybe you don't want to use the storage or index space
03:37
to keep logs we don't need.
03:38
In any case, you can use the disabled stream function from the logging framework to turn off a specific file.
03:46
As you can see here, I have to find a handler for the bro. Underscore Knit event.
03:51
This is the first of many times will see this
03:54
and the body of the event handler. I have a single statement that caused the disabled string function fromthe log module that functions argument communication, Coghlan, Coghlan Log
04:04
is an N. Um tight.
04:06
We'll cover this in more detail later, but for now, just know that this is the identifier for the communication log stream.
04:14
This identifiers, the only argument required for the disabled stream function
04:18
calling this function in this manner has no effect on the communication framework itself.
04:24
In fact, it continues to run and operate normally.
04:27
The only change is that bro doesn't create the specified log stream
04:31
for this example in those two follow, let's say I'm mainly interested in using bro to closely monitor inbound Web connections to my environment.
04:40
I'm working with pros http log, which contains a lot of verbose information about Web transactions.
04:46
I'm just getting started with Rose Data's, so I'd like to reduce the amount of information I'm working with initially and focus on things I'm most interested in.
04:55
To help with this bro allows you to disable specific fields and log streams.
04:59
Specifically, bro filters have to built in mechanisms for controlling which field show up in a long stream. The include and exclude fields
05:09
that you can see in the example of defined. A handler for the bro Underscore innit? Event
05:14
Note that this code could actually go in the same handler as the disabled stream example I showed previously.
05:19
The first statement in the handler body uses the law get filter function to create a local copy of the default filter in use by the http law.
05:30
This copy is stored in the variable filter.
05:32
At this point, I could make some custom is ations to the default filter
05:36
I assign a value to the include field, which is a set of strings enclosed in double quotes
05:42
containing the field names I want to keep in the http law.
05:46
A set is a built in container type that stores unique values and is what both the including exclude filter field expect to receive.
05:54
We'll talk more about container types in an upcoming section.
05:58
Finally, my last statement will add my copy of the default filter back to the http log
06:02
using the ad filter function.
06:04
Since my copy of the default filter also uses the name default theory, Jinnah ll default filter will be overwritten.
06:12
Illustrate. Another way you can work with log filters are now show you how to remove certain fields from Love Street.
06:17
Instead of creating a copy of the default filter, I removed the default filter and apply a new custom filter. My custom filter would tell bro what fields to exclude from the http stream using the exclude field of the filter
06:31
prologue streams can have more than one filter in the default filter isn't necessarily required,
06:36
so I can safely remove it and use custom filters exclusively.
06:41
As before. This is done inside of a bro, innit. Event handler
06:45
using remove filter built in function. I first delete the default filter applied to the http law stream
06:51
next time to find a statement that calls the ad filter function.
06:55
This function received two arguments. The log I D. In this case, the http colon colon log
07:01
and the filter record
07:03
the square brackets you see here declare an in line or nameless instance of a filter. The fields inside of these brackets correspond to the fields in the filter record definition,
07:15
specifically the name and exclude fields.
07:18
I'm just using this example to highlight this difference in syntax.
07:23
Remember that when you see these square brackets used in this way, it's actually a way of creating an unnamed instance of a record.
07:30
This is typically done inside of function arguments. In such cases, brogel attempt to process the fields and values inside the brackets. As a record of the expected type, Bro will accept similar declarations shorthand for many other types.
07:44
This is a condensed syntax that is a mouthful to describe and can be confusing at first. But it is an efficient way of creating code and something you'll see a lot of in bro script.
07:54
Another very common log management task is the filter. Which events get written to a Log Street continuing on with my Sarah scenario from before I modified the fields. I'm including in the http log output, But let's say I'm seeing a lot of Web activity to host. I'm not really that concerned about.
08:11
I want to restrict what's getting written to the Log Stream,
08:13
and Bro filters provide the pred field for exactly that.
08:18
Pred is an abbreviation for the word predicated.
08:20
Deep Redfield allows you to specify a function that does some evaluation of each event before it's written to the low.
08:26
This functionality serves as a good way to organize and categorize your network data.
08:31
The first thing I need is a list of Web host that I want him honor
08:35
to make this easier to read and adjust. Later, I'll create a redefine herbal constant that is a set of strings.
08:43
The strings in this set will be the names of the web host I wanna monitor. I'm applying a filter to the http log stream. So I have the field available in the http info record toe work with here.
08:54
The low filters Pred field expects a function that returns a true or false one of Bro's bull types
09:01
when past two given event.
09:03
If the result is true, bro will love the event. If it's false, bro will drop it.
09:07
Typically, the true false result is based on some tests. The function performs on the event being law.
09:13
In this case, I'm only interested in logging traffic to specific Web host.
09:18
I'll name my function is underscore monitored,
09:22
and inside. I'll add some logic to see if the host field of the http info record is in my list of monitor Web host drinks. I'll return true. If it's not, I'll return false
09:33
Now, Like in previous examples, I need to configure my new filter, which is done inside of a bro. Underscored, innit? Event handler.
09:39
I used the remove filter function here and delete the default filter for the http log
09:46
Next, I'll use the ad filter function to create a new custom filter.
09:50
My inline filter instance has a name field of my underscore Web underscore Host
09:56
and Pred Field has a value of is underscore, monitored as defined above.
10:03
When we run, this bro will only output events in the http love that air for my specified Web posts.
10:09
In this example, we saw how to find a global constant, ah, function in how to assign a function to a record field.
10:18
Now that I have my http logs refined to the fields I want and I'm looking for the host I'm interested in,
10:24
let's say that I want to start understanding how clients air getting to my Web applications.
10:28
Specifically, are they being redirected by an external sight? Another part of my Web application? Another local website?
10:37
Or is the client requesting the girl directly with no redirection?
10:41
Some information can be inferred from each of these situations, but for now I just want to understand what's going on.
10:46
Like with many programming language. There are a lot of ways to tackle this problem, but here I'm working with log filters.
10:52
The path underscore funk filter option provides a way to do this kind of organization using log files.
10:58
Specifically, I can use path underscore funk to assign a function that returns the name of the log file.
11:05
An event should be written out to
11:07
to help figure out how Web clients air getting to my Web host. I'm going to organize incoming Web request into three different log files.
11:15
No underscore reader
11:18
offsite underscore reader and local underscore reader.
11:22
To start, I need to add a new function.
11:24
When my function is executed, it will be passed. Three arguments automatically
11:30
the idea of the log
11:31
the current path of the log
11:35
in the http info record instance that represents the entry being written written to the lock.
11:41
Again, I'm demonstrating a few different uses of bro language types, operators and common programming patterns.
11:48
The important takeaway is that this function performs three tests and returned to New Path name based on the results of those tests.
11:56
If there is no refer, that path will be no reader.
11:58
If the refer is a U. R L. On a remote host, the path will be off site reader.
12:05
And if the refer is a local your eye, that path will be local reader
12:11
now that I have my function to find, my final step is to make the changes to my filter.
12:16
Since I've already got a custom filter applied to the HDP log stream, I'll simply assigned my function name to the path Funk field of my filter.
12:26
Broke will then use this function to to determine the destination file name of every event that it would have normally have written to the http look
12:37
in this segment, I discussed how Bro's log streams can be configured and managed using filters. Log filters are a powerful and extensible way for bro users to categorize and organize their log data
12:48
filters provided lots of flexibility and could be used to control what fields are included in the log file.
12:54
What events get written to the log file
12:56
in what log files certain events should be written out to.
13:01
I also use this section to introduce you to some common bro scripting constructs and provided descriptions of some of the syntax you'll see throughout the bro. Source.
13:09
We also briefly discussed the Bro innit Event and its handlers
13:15
in the next segment will revisit the concept of the event and discuss many other built in events that can be incredibly useful for implementing custom. Bro use cases

Up Next

Intro to Zeek Scripting with Bricata

The goal of this course is to provide you with an introduction to Zeek (formerly Bro) the application and the programming language. While the logs Zeek produces natively can be extremely useful, its full value is realized through its scripting interface.

Instructed By

Instructor Profile Image
Bricata
Instructor
Instructor Profile Image
Adam Pumphrey
CEO and Principal Consultant at Nimbus LLC
Instructor