Disk Latency Troubleshooting

Video Activity
Join over 3 million cybersecurity professionals advancing their career
Sign up with
Required fields are marked with an *
or

Already have an account? Sign In »

Time
21 hours 25 minutes
Difficulty
Intermediate
CEU/CPE
21
Video Transcription
00:00
>> Hey there Cybraryans and welcome back to
00:00
the Linux plus course here at Cybrary.
00:00
I'm your instructor Rob Gills,
00:00
and in today's lesson,
00:00
we're going to be
00:00
discussing disk latency troubleshooting.
00:00
Upon completion of this lesson,
00:00
you're going to be able to use
00:00
disk latency troubleshooting tools
00:00
like ioping and iostat,
00:00
and we'll see how we can use
00:00
those in our demo at the end of the lesson.
00:00
As we said in the last lesson,
00:00
there are just a handful of issues related to storage,
00:00
and these generally end up being
00:00
either running out of disk space,
00:00
having disk latency, or having
00:00
>> failing storage hardware.
00:00
>> In this lesson, we're going to actually focus
00:00
on disk latency troubleshooting.
00:00
Disk latency is a lot like network latency,
00:00
and that it measures the delay in data transfer.
00:00
The higher the latency is,
00:00
the slower the data transfer we're going to see,
00:00
and in general, we always want low latency.
00:00
Now, in Linux,
00:00
there are two tools that we can
00:00
use to troubleshoot disk latency,
00:00
and those are ioping and iostat.
00:00
Let's take a look at these two tools with some demo time.
00:00
Here we are in our demo environment on CentOS,
00:00
and the first tool we're going to look at
00:00
today is the ioping command.
00:00
Now, the ioping utility is used to test disk latency,
00:00
disk [inaudible] grades,
00:00
and what are called sequential read speeds,
00:00
but it can also destroy data on disk.
00:00
Definitely proceed with caution.
00:00
I explicitly cloned my VM for this lab to be safe,
00:00
so that if I blow away this VM,
00:00
I don't care, it's fine.
00:00
But to see this latency with
00:00
the default values and from the current directory,
00:00
we can run ioping this way.
00:00
We just run ioping and then dot,
00:00
and that just indicates we're
00:00
running this from the current directory.
00:00
Now, the directory that I'm actually running this from
00:00
is home, rob desktop,
00:00
and that's on a separate partition
00:00
from the root partition,
00:00
so that's even a little bit more safe.
00:00
Worst-case scenario, it might
00:00
mess up this home file system,
00:00
but I don't really care, I'm the only one using this.
00:00
Let's go ahead and hit "Enter",
00:00
and now we can see that it
00:00
is providing us with information here.
00:00
This is going to actually return time in microseconds,
00:00
and in some cases, you see,
00:00
US, which is one-millionth of a second.
00:00
Let's go ahead and just hit Control C here.
00:00
I'll show you some other nice things about ioping.
00:00
One thing that you're
00:00
generally going to want to do when you're
00:00
running an ioping is let
00:00
this run 100 times to get an accurate representation
00:00
of the kind of performance you're
00:00
seeing on this file system,
00:00
or with this storage
00:00
backing on this file system, the disk latency.
00:00
We can do dash C,
00:00
that specifies count,
00:00
and we're going to specify 100.
00:00
Let me clear my screen, and then we'll hit
00:00
"Enter", and just let this run.
00:00
This is going to run 100 times.
00:00
Then once we've gotten this measurement,
00:00
once this has run through 100 times,
00:00
we can measure this against
00:00
other systems that have the same work load.
00:00
If we have a like for like
00:00
system running the same workload,
00:00
maybe running on the same storage,
00:00
we can see how the performance is over there.
00:00
Maybe we want to look at an
00:00
earlier baseline that we've run
00:00
this ioping command before on the system,
00:00
maybe we want to see that we don't have
00:00
some weird storage latency or failing hardware,
00:00
or we might want to run this as a point of
00:00
comparison against different types
00:00
of storage, or file systems.
00:00
Is it faster on in this type of disk,
00:00
or on this type of file system,
00:00
how's the performance differing from one to another?
00:00
But that's pretty much all we need to
00:00
know about the ioping command,
00:00
I'm just going to hit Control C,
00:00
and here we can see all the information,
00:00
the request completed, the generated 45 requests,
00:00
and get all the summary information here.
00:00
This is the information that you really
00:00
want to capture for your baseline.
00:00
Now, let's take a look at this next command,
00:00
which is the iostat command.
00:00
I'm going to go ahead and type clear.
00:00
You can also hit Control L
00:00
and remember to clear your screen.
00:00
Now, with iostat,
00:00
it's used to measure I/O wait times,
00:00
and really an I/O wait time is
00:00
just a statistic that's showing
00:00
the amount of time a processor
00:00
has to wait on the disk to return I/O.
00:00
This can indicate high latency on a system,
00:00
and by default,
00:00
iostat just provides a summary of
00:00
statistics since system boot time.
00:00
We can do an iostat and hit "Enter",
00:00
and what we'll see here is it's going to provide us
00:00
information about the system,
00:00
it's going to tell us about what kernel we're on,
00:00
the date the system was booted, the architecture,
00:00
X8664, it also gives us average CPU time,
00:00
and then down at the bottom actually
00:00
gives some device statistics.
00:00
There are a few other things you can use,
00:00
some other options with iostat.
00:00
For example, maybe you want to see
00:00
just the device statistics,
00:00
you can do iostat
00:00
[inaudible] care about this other stuff,
00:00
you can just do iostat dash d,
00:00
and now we see just the information about the device,
00:00
and then some information
00:00
>> about the system here remains.
00:00
>> One other thing we might want to do is
00:00
make this a little bit more human-readable,
00:00
like what are these values? What do they mean?
00:00
Well, we can do an iostat dash dh,
00:00
and now we can see that actually inside,
00:00
so 11.2 megs, 2.1 megs,
00:00
7.2 gigs, so on and so forth.
00:00
Now, the columns here merit mentioning as well,
00:00
because they don't really tell you
00:00
exactly what's going on.
00:00
TPS is transactions per seconds,
00:00
so this is the number of
00:00
transactions that have occurred,
00:00
we can also see the kilobytes read per second,
00:00
and these are the kilobytes of data that are
00:00
read from the device per second,
00:00
we can also see kilobytes written per second,
00:00
written to the device,
00:00
and then we can see kB_read and kB_written,
00:00
and these are just a total kB read and
00:00
total kB written from the device.
00:00
In this case, this is the SDA device,
00:00
which is the primary partition we're
00:00
using for pretty much everything on this VM.
00:00
Now, again, once we have this information,
00:00
we can take this and measure this against
00:00
other systems with the same workload,
00:00
maybe some earlier baselines taken,
00:00
or for comparison against
00:00
other different storage and different file systems.
00:00
But with that, we've reached the end of
00:00
this lesson, and in this lesson,
00:00
we covered how to use
00:00
this latency troubleshooting tools
00:00
such as ioping and iostat.
00:00
Thanks so much for being here,
00:00
and I look forward to seeing you in the next lesson.
Up Next