Time
6 hours 3 minutes
Difficulty
Intermediate
CEU/CPE
6

Video Transcription

00:00
Hello and welcome back to the Splunk Enterprise Certified Administrator course on Cyber. In this video, we're gonna be doing a lab. Where will be reviewing the http event collector also called heck or HTC one of the Splunk in put options you have for sending data into Splunk.
00:18
So before we get into actually setting it up, let's talk about what this input is, what purpose it serves, like why we would actually use it.
00:28
So basically, there's
00:31
a couple core features to this input method. So first is that it's agent lists. And so what that means is you don't need
00:42
a Splunk universal border
00:44
and then second, uh,
00:47
it authenticates off of a token and then also, I guess, important to note that it's it. It sends logs in a streaming mechanism. That's not necessarily
00:57
ah, super unique, but still worth noting. So the use case for this is basically when either of those 1st 2 criteria is valuable. Either you have a system that you don't want to send, or you don't or can't put an index on agent on or,
01:17
um,
01:19
if you want the authentication security provided to you by a token,
01:26
uh, it's also very good for, like, super high data volume problems because it is very scalable. So, like one of the most prevalent examples of when you would use this input as if you have a large AWS environment
01:42
and you were going to send the data down the, uh like fire hose or Lambda functions.
01:49
Then you can send it directly to http event collector. And that's like a super scalable and, uh,
01:57
better better option than, like pulling from an A p I. Because, like a P I pools, they're gonna be limited in volume base and its pool. So you get time lapses and stuff, whereas if it's being streamed directly and it's a very scalable solution, then you have something that can actually handle that high volume of data. So that's like
02:16
the biggest use case for HCP event collector.
02:20
So I have the documentation for how to set this up
02:23
here just so mostly just to reference this at the bottom, which is the Carl Command we're going to use to test that are setting it up. Worked, uh, actually setting it ups pretty straightforward, so I don't know how unnecessary these documents are, but it is interesting to note and good to know, for those of you,
02:43
um, watching
02:45
that depending on how your environment is deployed, whether it be spoiling cloud if it's self service or managed or if you have a known problem deployment, that is going to have some significant impacts on how you go about setting this up. So this review is really directly relevant to on Prem deployments.
03:05
So
03:07
the way you set this up as you go to settings and I'm going to do this on my search head,
03:10
technically, this isn't how you would normally do this, but for the purpose of the lab, it will spice. Um, way I would normally do. This is probably
03:21
have a multiple, every four order solution, where with a load balancer in front of them. And if I was in AWS, for example, I'd have, like my firehose sending to my load balancer
03:36
the A G C. And then I would let it be load balance between the two. So that's how I would normally do this. But just for the sake of the lab and demonstrating, I'm gonna set up on my search head here, and then I'm gonna send the data from my indexer to the search head, which yeah, doesn't make any sense. But it'll
03:55
do for the purpose of demonstrating how to do this.
03:59
So you go to settings. Data inputs, http event collector, understand? Select add new because that's what we're doing. And we'll call this a C C. Test
04:10
input.
04:11
We're not gonna bother with either of these options. Output group would be if I was forwarding this to some split devices which in this case I'm not. And then indexer acknowledgement would make it so that basically the data being streamed this ways like ST Paul. So basically, when the data sent
04:30
the indexer needs to acknowledge that the data was received.
04:33
If the data was not received, then the source can resend it, so that gives you, like, another layer of data resiliency. So that's a good feature. But we're not going to use it right now,
04:46
so source type will call it
04:48
head test,
04:51
and we won't put a description. That's fine. For now, we're going to send this to Maine
04:57
and review
04:59
Oh,
05:01
it seems like I already made that.
05:04
Let's do this custom main review
05:10
and submit. You just get a chance to look this over, Make sure everything's right before you create your token. And now we have a token. So that's gonna be valuable. I'm gonna leave this up for now. But just so you're aware there is one mawr Little trick to this and that is that by default,
05:29
your http tokens are going to be disabled. So all tokens are currently disabled. They could be enabled in the global settings. So you can see this says here disabled and you don't have an option to toggle it here. So you have to actually explicitly enable all tokens first,
05:47
and you specify the port. If you want to change it, we're not going to
05:53
so and also
05:55
you see that's where you get to enable SSL. And if you use the deployments ever, that's a valuable option. If you want to make this configuration and push it out, you'll need to check this. I think that has to do with some encryption stuff because basically each of your devices will have its own splitting secret key. So
06:13
I guess maybe this pushes it out unencrypted. And then once it gets the device and encrypt that,
06:16
not 100% sure on that. But I would think that that makes sense that that's how that works. So
06:24
now our token is good and ready to be used. So let's go back to this documentation and just see how we contest this with this. Send data so
06:35
we can use a curl command. I'm just gonna copy this
06:40
as
06:42
a
06:44
template and we are just gonna send Ah, hello, world event.
06:47
So and this is the format you have to send it data through with the HDP event collector. Basically, this is the field name on This is the value, and it needs to be event. And then whatever event data you want in there, you also set some metadata fields. This way, if you want to give you a little bit more granular control over it than just setting it for that token as a whole,
07:12
so just some good information to know. So I'm on my index there. Another thing you can see here. I had already done it, but make sure that whatever device you're forwarding to is listening on the appropriate port. So it'll either be 80 88 or if you're using SSL for 43
07:30
So up since I highlighted that I copied it. So let's go back here,
07:36
get our sample
07:39
pace. That. And so now there are a few things we need to change. Will need to sub out there token for or token. And also change this destination to three i p. Address of my device.
07:53
So let me just clear this out.
07:56
I should have it memorized by now, but I don't. So
08:00
there we go.
08:01
Paste that in.
08:03
This is gonna run it. It's not gonna work. Whatever.
08:07
Okay, that didn't even paste it in there.
08:09
So never mind.
08:11
I'll just
08:13
type it.
08:13
Do it the old fashioned way. 1 92 1681108
08:20
And now we need to go grab or token
08:22
and replace this value here. With that, I could have probably done this in no pad to be a little bit
08:30
cleaner versus
08:31
trying to do it this way is a little bit convoluted,
08:35
but
08:37
that'll work.
08:39
Okay,
08:41
so now we can run this. Everything looks good. It's doing a once over. And you see the Jason response. It was a success code zero. So we should be able to go back to our search head
08:54
search and reporting.
08:56
And if we just look at in Mexico's man, I don't think I have any other data in there.
09:01
So you can see a hello world source
09:05
is the source we specified
09:07
source type. Is this source type of specified. I'm sorry, this is automatically generated. We didn't specify this. So it's just gonna be http colon and then the name of that token
09:18
and then
09:20
the source type of specified and that is the host where the data came in through. So that's it? That's how you do they see to be event collector, you just need to set up a token and then leverage that token and Ford events in the correct format
09:37
and Splunk will receive them. Make one
09:39
trick. Make sure that your firewall has an opening so that that connection could go through and also make sure that you enable your tokens globally
09:52
so that you can actually use the token. If we had not done that, I could do a quick example
09:58
of just disabling this and trying to send that event again.
10:03
You'll get
10:05
this notification if the individual token is disabled. I think you'll get a different
10:13
notification.
10:13
Yes, so you'll get failed to connect. Connection refused if all tokens are disabled. So essentially, this device basically still isn't listening for this information. Um,
10:26
if you didn't have ah, row double. If you couldn't route to this, you would get like
10:33
like no no available route error log or something.
10:37
Ah,
10:37
And then, as you could see, if the device is listening for tokens But this individual tokens disabled, you'll get this message. So that's everything you need to know about the A. C. C. Input to get started working with it. Some of this might take some play around in a deaf environment toe to really get down hat.
10:56
But this should give you a very good starting point and help you understand.
11:01
You know what this input is and when you should use it and how to set it up. So that's it for this video and we'll see you in the next one

Up Next

Splunk Enterprise Certified Administrator

The course is designed around the guidelines provided in Splunk’s Test Blueprint for the Certified Administrator certification, Splunk Docs, the Splunk Data and System Admin courses, and the experience of a Splunk Professional Services Consultant.

Instructed By

Instructor Profile Image
Anthony Fecondo
Splunk Professional Service Consultant
Instructor