Part 2 - Spidering

Video Activity

This lesson covers spidering. Participants learn about: 1. What is spidering? 2. How to spider with BurpSuite 3. How to spider with ZAP 4. Spidering in other programs Spidering is a technique used to map a web site and identify pages that all users have access to and is done either actively or passively.

Join over 3 million cybersecurity professionals advancing their career
Sign up with
Required fields are marked with an *
or

Already have an account? Sign In »

Time
4 hours 20 minutes
Difficulty
Intermediate
CEU/CPE
5
Video Description

This lesson covers spidering. Participants learn about: 1. What is spidering? 2. How to spider with BurpSuite 3. How to spider with ZAP 4. Spidering in other programs Spidering is a technique used to map a web site and identify pages that all users have access to and is done either actively or passively.

Video Transcription
00:03
>> Welcome to Cybrary.
00:03
I am Raymond Evans and I will
00:03
be your subject matter expert for
00:03
Cybrary's web app penetration testing course.
00:03
In this video, we will be discussing spidering.
00:03
What will be covered in this video,
00:03
we're going to discuss what is spidering.
00:03
We're going to talk about
00:03
>> how to spider with Burp Suite.
00:03
>> We're going to discuss how to spider with ZAP,
00:03
the Zed Attack Proxy.
00:03
Then we're gonna talk about spidering that exists in
00:03
other programs that you may want to know about.
00:03
What is spidering? Well, spidering is
00:03
a technique of mapping a website,
00:03
and identifying all the pages
00:03
that are accessible to any user.
00:03
Basically just crawls the page and finds everything
00:03
that a user can click on and
00:03
>> interact with on a website.
00:03
>> This can be anything from pages
00:03
that users are supposed to access,
00:03
to some documents that may be stored on the server
00:03
and may not necessarily be for a normal user's eyes.
00:03
Something like a person's resume or
00:03
an internal address list or something like that.
00:03
Sometimes these are put onto servers and
00:03
people don't realize that
00:03
those are public-facing servers.
00:03
They put it on there thinking, "Hey,
00:03
I can access this later,
00:03
or I can just distribute it out to
00:03
the internal network here."
00:03
But in reality, people
00:03
from outside of the organization can view it as well,
00:03
and sometimes that can lead
00:03
to sensitive data being leaked.
00:03
That's why you may want to do
00:03
spidering to find any of those sensitive files that
00:03
may be out there that you necessarily don't want
00:03
people to see. How is it done?
00:03
Now, there's two ways,
00:03
there's actively and passively.
00:03
When active spidering occurs,
00:03
the tool being used to spider clicks on
00:03
every single link in every button and
00:03
fills every single format.
00:03
The tool will continue to follow
00:03
each page and not stop until told to do so.
00:03
This can be dangerous because it can be seen as
00:03
an attack if the tool finds
00:03
something like an admin page,
00:03
and click "Submit" on
00:03
a button that deletes users or deletes pages.
00:03
This can be really really dangerous if
00:03
some administrative page is left public-facing,
00:03
and not locked down properly.
00:03
You want to be careful while spidering so you don't
00:03
accidentally cause a loss of data.
00:03
It can also be done passively.
00:03
Additionally, spidering can be done passively.
00:03
When a passive spider occurs,
00:03
it acts just like an active,
00:03
however, it will stop at the next page.
00:03
Passive tends to be safer than active because it
00:03
really doesn't click on every single link.
00:03
It gives you information like info
00:03
from HTML code headers and things like that,
00:03
and it's generally safer.
00:03
Spidering should be done prior to testing for
00:03
vulnerabilities on a web page
00:03
for a few important reasons.
00:03
One, creating a website map gives automated tools
00:03
the ability to identify every possible vulnerable page.
00:03
It also gives a tester a better picture of the website.
00:03
Spidering can also identify pages that
00:03
shouldn't be available to every user.
00:03
Things like we said before,
00:03
admin controls unfinished pages
00:03
or pages that contain sensitive data,
00:03
or where you can download
00:03
>> files from that are sensitive.
00:03
>> Spidering is a very important tool
00:03
in performing web app pen-testing.
00:03
You will want to spider every time you
00:03
perform a web penetration test,
00:03
to get an idea of your terrain.
Up Next