Okay, So for our next section in Tijuana three, we're gonna talk about text streams.
Let me see you quite a few commands here
that you have available to you to manipulate
standard output, standard input and
and the stream of data going in between programs.
And these are really handy tools
to be aware of
because it allows you to do lots of different things with files and,
uh, be able to kind of filter out information that you want right away without having to go hunting for it in a more manual sense.
Now, if you remember in the previous section we talked a little bit about standard input, standard output and standard error.
So these are the io streams. I can filter the input. I can filter the output. I could also filter the errors.
And I may want Thio do this. Rivera's reasons, because I'm manipulating information. Maybe I'm trying to merge log files or get some information out of a log file or sorts of information counted.
There's lots of different reasons why I would do some of these things.
let's look at some examples.
You've already seen me use the pipe character
and the greater than symbol
pipe will take standard output
and send it to the standard input of some other command, which follows the pipe character,
whereas the redirection
symbol of greater than will take the output that would normally go to standard out and send it to a file. Typically,
I could also use the less than simple
to use a file as input for standard input,
so I can do that in both directions
and sometimes the commands. We could get a little bit
difficult to to read because you've got a command. Maybe you're piping the output of that to something else, and then I've got redirection from standard input to something else. It can look a little bit confusing,
but as well as I'll show in some of the examples here, it's not not too difficult to understand what you practice a little bit.
So I've got a handful of commands here that will cover, and then
I'll go to another selection of commands that you need to know for the exam.
We start with the Cat Innate Command, which which have already used several times.
This basically takes some argument on the command line and send it to standard output.
Assuming it's a text file, this works as you would expect. If it's a binary file, you might get some strange characters on the screen. Your terminal Mike beep.
You cannot output a binder file to the screen that you normally would be able to see. A text file
can also run the OD Command to do an Octel dump.
This might be useful for programming purposes, debugging purposes.
I'll demonstrate it, but it's, um
not not a very commonly used to manned by any means.
Then we had to split command, which does just what it sounds like, allows me to take a file and split it into multiple pieces of a certain size.
This was typically done in the old days when you wanted to try to take a large file and fitted on some
removal immediate, like a floppy disk or a tape.
If it's too big, I could split into smaller pieces, copy those separately and get around that problem.
I've already used work counts several times.
This is very handy to be able to count lines count, words
we'll use had entailed toe. Look at the beginning section of a file and also the end section of the file, specifying the number of lines that we want
expanding and un expand.
I have a very easy to understand purpose. It takes a space and converted the tabs or converts tabs spaces.
Then we have translate,
translate. Let's we take
two different character sets and translate from one to the other.
Easy to understand examples. I've got a file that's all lower case. I want to make it all uppercase or the other way around,
and I can convert other characters as well, and we'll look at the details for that.
And then we have the printer command,
which takes a file and,
formats in such a way so that it's more conducive to printing.
Let's have a look
at some of these options.
You've already seen a pipe and the redirect, but I can still,
uh, do a quick example here. So I'm in my home directory
I'm going to, uh,
if I run the L. L Come in. I've got a bunch of files
now. I can ruin allow, and I can pipe it
to another command
you can pipe into sort.
We'll see sort here in just a minute.
Now you'll notice that
the output is different.
No, I just gave Sort as a
as a, uh without any without any arguments.
It sorts on the first field that encounters. So in this case, it actually sorted on
the provision strength.
And then then it will go to the next field. The next field, since the's first field's a little bit different, See how it's sort of a little bit? Their second and third fields and four fields are all the same.
And then it went
to the rest of listing here.
Now I could I could do something a little bit more interesting. Instead of piping this to sword, I can pipe it to grip.
And let's say I want to look for
string file one.
This is kind of a roundabout way of doing this. I could I could have more easily just
typed. L l file one.
Yeah, that same thing. I lost a shell
for one with a
with the wild card.
The nice thing about using the pipe is that I can take any command, send it the output to another command,
and I can also do something like this a zay ran this before. I can save the output
so I can, uh,
save this. An upward filed called
out got text.
So now I don't get anything to standard out.
But if I look in my shell, I've got a file called out dot tax which I just created, and I could cat this
extra character there
and I see there's the output of my command.
So saving the output of command is really useful
because I may need that later or maybe do using it as part of a script.
Now let's look at the Octel dump Command.
I can just run this
or rather, help pansy.
So my options are I can
specify various settings
writing the octo bites of a file doing man page on this real quick.
So lets me send the output of a follow in Octel to this to the standard out.
But it also supports other formats,
so decimal form and, um
But for a simple demonstration, I will just
run OD against
a nice by near That I know exists has been if config
and there's the config program and Octel format,
then we have our split command
and what I've done is I've copied var log messages
to my local director here, So I've got a nice large file to work with.
If I run split,
uh, dash, dash help,
I can see that I can split by the number of bites,
or I could split by the number of lines. So if I can't
messages text and pipe this
to workout dash out, I can see that it's got 19,419 lines.
It's pretty easy way to, uh,
to get that information.
No, not with the split command
I can run, since I've got a large file messages out text.
What I can do is decide to split this in various ways. Aiken say split
based on line numbers. I know it was 19,000.
I'm gonna put it into
5000 line chunks
now. If I do a long listing by default, it's It names the files, starting with a default naming of X. Aye aye. Then it goes to exit A, B, E X, A C and so on,
and you can change the naming. But for for illustration purposes,
this is fine.
So now contest to see if there's work. I can Cat x, a
pipe that toward count Dash l and it's 5000 lines.
I'm gonna go ahead and remove these. You notice I'm using the dash R F
This says, remove the files that I specify enforced removal. So I don't have to acknowledge each one.
I know that X ray will match those files. I can go ahead and remove those.
Maybe I want to split based on,
the number or so. That was a number of lines, but I might also want to split
based on the, uh
the size of each
So I've got
I want to split on
I actually think it could specify this in
Oh, it's our dead dish and is the number of chunks so I can put into even number.
Dash B is bite size,
and I think I can specify that. And
megabytes, every gun.
So these are the units
so split dash and five messages dot text
should split into five evenly sized items, which is really convenient. If I know that on my storage medium on Lee supported 400,000 bites, let's say
I could say, OK, split it based on that divider.
But then I can do it again with
Dash Pizza and I'll say
Oh, actually should erase these other ones first.
Now run split
100 mag increments
and it looks like it. Uh,
well, maybe it Maybe it Maybe I shouldn't have done 100 magnification on 10 minutes.
Let's try that again.
I'd only give me one file,
but you see the usefulness I can break it up by the number of lines or the size or divided into evenly sized chunks.
I've used the word count command several times.
Now we're count. Uh, also,
I typically use it with the, um,
dash l'd account lines, but I can also count characters. I can count bites and I can count actual words.
So if I can't
messages dot text
a pipe that toe work on bash elegant that 19,000.
But now, if I took count the number of words that's gonna be much larger. 227,000
and a word is some string of text
that's separated by delimit er. Could be a space could be a semi colon in this case, or some other cup,
the limiter and I can define which dl emitters I want to use as a separator.
Then we have our head and tail commands. If I run hey, head against messages dot text
by default, it gives me the 1st 10 lines
Very useful. Sometimes there's information at the beginning of file you want to look at
I can also run head
a a number
and it will always show me that number of lines. So head Dash five shows me the 1st 5 lines.
Similarly, I can run tail
on messages, not text
See the last 10 lines
Or if I specify something else,
I can see the light the last 20 lines.
So this is really useful
just to see the beginning of the end of the file.
And even more useful is to,
you the tale dash F option.
I've got a *** remove this.
So what I'm gonna do is run tail dash F
out dot text
Now tail gave me a text, only has 22
lines of data in it
and you'll notice I didn't get my prompt back. So tell Dash is waiting for more information.
So What I can do is I can open a new tab
who didn't like my password.
So now I'm gonna echo a new line of data
to out dot text.
Oops. Almost made a mistake there.
We were talking about redirection earlier.
A single redirect. We'll create a new file or override existing file. What? I want to actually do it. Upend my data. So I have to use a double redirection
in order to do this.
So now I've sent
a new line of data
to the file. That's what out dot text now contains.
But because I was I was running tail dash F
against this file. That new line that I just added shows up on my standard output.
So you might think, But why is this useful? The reason this is useful
is because there may be times when I want to
monitor log file.
For instance, if I if I look at it
var log messages
and I do something like open a new shell
or do something else that generates a new log entry, it'll spit out to the screen.
all right, there. We got another message.
So, starting fifth Section 51 that was the new shall. I just opened up here.
That's my new, uh,
new session that it was talking about. I'm gonna go ahead and close that,
and so this is useful because I can continue to monitor this file in a separate window without having a concert, do a tail command or open the file to look at it.
I could just do control. So you to get out of there
so expand and a nun expand. If I had files,
for instance, I have
my file one dot text which I created earlier. It has some items in it file to doubt text tests of items which will look at here in a little bit.
So there's there spaces there in, uh,
in the in the file.
So now if I created file
that has some tabs in it,
like file three dot texture, I've got four lines with tabs.
I could run the expand, command and tell it
maybe I don't want these tabs. I want thio
reduce each tab to a, uh instead of what looks like
five way. We have
seven characters. I wanna reduce each tab to two characters
or actually that should give me too. But,
oh, it's too, including the first line. So there's if I change into four,
So pretty useful for doing a little bit of four. Manning
our next command to consider is translate,
and the trips like to me is kind of used. So, for instance, if I have a
cold translate dot text, I just typed in the alphabet.
This is a real simple example.
But what I can do is I can
cat this file
pipe that to translate
and the sin texture is a little bit tricky. But basically what I'm saying is I want to count a translate all letters a through Let's say m
I want to translate those characters on Lee
to lower Case
So you notice
a through M
more upper case. Now they're lower case, and that could go backwards and forwards with this. Of course,
looking at the help command for translate,
I can do things like go back up here for a second.
I can delete characters so I can take,
uh, lower case letters and get rid of them all. Get rid of all my upper case letters
or I can translate translate
using these different,
sequences here, all letters and digits
or all letters. So, ALF in America, I can translate letters and digits in that same range.
I can translate all my control characters and non printable characters. Things like four feeds new lines and so on. I can get rid of spaces.
It's a little bit similar to the way that the Stream editor work said,
but I can do it in a little bit more of a
a precise manner to basically say, Take all these characters and convert them to these other characters.
Then I have the printer command, the PR command.
So if I run
PR against messages dot text, which we know is that 19,000
line long file,
I'll run a PR, you know, send it to more
and you'll notice
it gives me a nice header at the top, tells me what the date and time is
with the files called and the page number,
and as I hidden space, get my next page number.
So if I was printing this, this could be pretty handy, and you can obviously change the number of lines per printed page, according to your to your needs
So those are some good commands to be familiar with. We have some other ones to look at as well. Before we're done with this module,
I can number the lines in a file, which is very handy.
And I can also control the width of the file as I send it to San Standard out or just some other
We've got the sort command. This lets me manipulate data and sorted according to various
parameters that I specified.
Now the unique command cut is a little bit of overlap with sword because unique.
I want to remove duplicates. But I can also do this with sort command directly.
And we've got the cut command, which is very
useful when I've got a file that has delimit er's. I can easily go in and and only see certain pieces of that file.
The Pace command lets me take two files that have matching field definitions and paste them together.
Uh, this has limited usefulness for for your your typical daily tasks, but we'll see how it works. Nonetheless,
you saw the split command earlier wth economy verse of that is the joint command. I can take multiple files and join them
single file, if I wish.
Then we'll go through the stream editor or said
show a couple of quick examples for that.
And then finally, the diff command, which lets me compared to files together directly to see what the difference is might be
and thats useful when you're got two different config files and not sure what changes someone made. I can run the diff command to see where there
where they might be
differing from each other.
All right, so let's have a look.
First, we'll start with the number line.
So number number being the lines in the file. I've got some different options, but the basic function is just to run an L against some some file.
We know that this file messages, not text that I copied earlier has 19,000 lines 19,419. So a number each of those
now this just went to standard out. I could also redirect this
because we saw earlier
messages. That's call it numbered,
and I've saved that file
with the line numbers attached pretty handy,
and we have the format command
so if I format.
If I run for mayor against file one dot text
when it's doing is joining the lines together,
uh, that were
that were previously separate.
You can see this because I can catch the old file,
but when I might want to do to make this more useful is use the dash w
So I said, Okay, only give me a five character with So it'll give me five characters for the first column.
Five for the second. Even though it's longer than five, it'll still printed.
And so this way I can
tell the formatting how many Hewat I want to be. If I say 20 characters, it'll think up to 20 characters on each line,
and you could see why that might be useful for taking certain kinds of data and formatting it to be printed or to be sent to another program.
I don't really use format too much. The sort command, however, is
much more useful, much more regularly useful.
So what I've done is I've copied the password file
to my current directory and called and password dot Text is a good wanted to demonstrate some of these techniques.
So if I can't pass word dot text,
it's in the order that it's in. But maybe I don't want that. Maybe I want to sort it
now. It'll look at the first field and sort according to that. To that value.
There it is. I should have cleared the screen first to see that more easily. Let's do that again
and their pipe, this tomb or so we don't lose everything strolling off the screen.
So now we can see the sorting on that first field. Everything is alphabetical,
and I could I can save this output if I wantto have sort this file and
But let's look at a different file. I've got one here called a sort dot text, which I created,
and this is a series of numbers. These could be the first field of data Entry's could be anything, but it's It's a good way to see how this works
now. If I If I run sword against sword dot text,
it sorts them, but not in the way that we might expect.
This is what's called Lexa graphical sorting. So, um, 11 is greater than one. According to this sorting method 14 is greater than 1112 is also greater than one form.
This kind of sorting is not the way humans normally think of sorting, so it's a little bit unusual,
but it is still a
a way to sort characters according to a certain algorithm.
Now, if I want to sort these numerically Aiken type sort dash and
for this file
hopes, which has the wrong one sort dash end for sort of text.
Now the file shows me the numbers sorted in a order that's more predictable. This is the way people normally sort things.
But as you can see, I've got some
So what I can do
is add the dash, you option to say, sort according to a new miracle settings, but also remove the duplicates. So now I've got my numbers sorted correctly with the duplicate lines removed. That's very handy
Now. I can also run the unique command
you need command
Good because, well, let's let's do this again. So there's sort of a text,
a bunch of unsorted numbers with a bunch of duplicates. I run unique against this data.
I can get where the duplicates but it's still not sorted.
So as I mentioned before, more than one way to do things in UNIX. Lennox.
I can run unique sort
and then pipe this output to soar. Dash end
duplicates already removed, but I want it in numerical order. So now I've got the same result.
So I can do this in multiple ways. Personally, I think Soar dash and you makes more sense
because it's it's less typing. But you can see why.
Why? That might be
All right, now we have the cut command.
So since I've got my password filed,
that's, uh, that's the cop of the past reform. Maybe I only want to look at
a listing of the user name, which is the first field and the user i D. Number, which is the third field.
So what I can do is I can cat password dot text
and I can fight this to cut.
Now, If I preferred to come without any kid without any
it tells me I need to specify some parameters
so we look at cut
The main thing that we're concerned with is a delimit er
started the DL emitters here
in this case to delimit er is
The colon character. And then I want to tell which fields I want.
So I will cat password, text
pipe that to cut
and the Taliban. My delimit er is
the colon character.
And then I wanted look at Fields one and feels three.
I did exactly what I wanted.
I've got each user name and the third That's the first field in the third field. Waas the user i d number we know Route zero
and so on.
Maybe I also want to look at the home directory,
so that should be field number seven.
So I want to see 13 and seven
The problem with cut is that you need a
file that's got delimit er's and and and and and it's consistently formatted that way.
Otherwise, it's hard to tell the program where you want to do this. Thea cutting to display that information.
Then we have our paste command.
if I, uh capt
dot taxi and filed to dock text,
we can see that those are two files with six lines each.
I can paste them together, and this might be useful in certain cases
where I've got data that's very consistently formatted.
And I want to, uh,
take these two files like this and join up together, paste them together, side by side.
So now I can see I've got line one from both files pasted together, lying to and all the way through line six.
There are certain instances where this capability is is pretty useful.
I can also join files together.
So if I wanted to join, found one
with file, too?
these files, uh, share a common
ah field in the beginning,
which is the line number in this case,
it doesn't show this in the output, but I've joined the two files together.
Uh, this is joining it. Just two standard out, but I didn't actually create anything. I would have to save this
So remember, I ran split.
I could run split,
dash, and two for messages, not text.
And we see that I've just created these two files right
now. What I can do is I can I can join
x a with ex baby
and save the output as password
Oh, see, it's complained that they're not sorted. So
the joint Command does require
files to re sort it.
and so what I did. That's probably bad example. What I did earlier waas
created took file one and filed to that I was using and sorted those.
So if I can't find one
and fall to sorted,
I can join those together
now that that didn't send me any output to my screen because I'm,
sorting them together, but not actually showing the output to standard up.
If I had a really large file like, there is a way to to join things together like I just showed,
and I think what you have to do there is used the
the no check option which tells, sort or tell the joint command. Don't bother looking. If there's if the two files they're trying to join together our sordid already.
All right, moving on we have the said command
is a stream editor, so I can use regular expressions
to change words of modifying, do search and replace type options.
So I've got a file called said dot text,
and it's just contains a sentence, as this is a test using said.
What I can do is I can, Kat said dot text
pipe this to said,
And I can
change, For instance, the word
this to that. So use asked for a substitute.
I'll substitute this for that.
Who played, I forgot.
There's always some detail that you need. Thio specify. If I'm using regular expressions, I need to tell said with a dash e
Okay, so my message here is Tell me this. This is Unterman. Aidan. What I forgot to do was type in the trailing slash
So syntax is obviously very important.
What I'm doing is saying
And if I don't use the trailing slash before I use my single court and it doesn't know what I'm trying to accomplish
now, if I run it
Previously it said, this is a test now it says that is a test
and said has quite a few
for doing more advanced things like I can,
I can run this again,
and now I'll change this too.
Send the, uh, put back to said dot text.
I'm sorry. I'll call it Dust said
tue dot text.
So what I've done is is made a change to the file and then created a new file with that output.
I can also do this in a little bit more complex way.
I could say also substitute
test for experiment
All right, so I'm a little bit of a minor air there
and what I did wrong waas
and syntax again is kind of tricky. Sometimes we have to refer to a man, Paige, and double check our work.
So I'm using, said Dash E, which let's meet
sounds using regular expressions. But I'm any multiple replacements on the same line,
and I'll get rid of this.
So the, um,
far we're gonna operate on your jacket.
I'm ready. Catting it.
So this is a test using said That's what said dot text contains. Someone changed this to that and then changed test to experiment.
So now we have that as an experiment. Using said, instead of this is a test using set.
I did this using cat at the beginning of the command line.
I could have just as easily run, said directly,
and then specified,
said Don tux at the end of the line. So again multiple ways I can use cat to send the output through a pipe or run it this way
for more convenience.
And then we have to look at our def command.
So if I run,
uh, let's say I'm going to CAT
found one dot text
hand filed to dock text.
Each has six lines. They're all different except for the last line which matches.
And I did this on purpose in order to illustrate what how diff works.
So if I run def on File one and filed two,
it tells me that the 1st 5 lines
on the first file, which is the arrow pointing to the left that's filed the first following specified,
are different than
the next five lines in the second file
Lying six, which exists in both files. And I called it matched just for purposes to illustrate this. Since line six matches, it doesn't send me any output.
Now I can I can show on Lee those lines that match. If I wish
there's ways to do that, I can specify my, uh, I'll put a width of the of the one I'm looking at
and quite a few other things that are really useful. But the main thing with def is that I'm trying to
compared to files together and see where the differences are
in order to
be able to do some work with that information
so I could ignore matching lines.
I can ignore the case
dealing with Spaces and so on. Sometimes def will get a little bit confused when you're when you've got spaces and tabs.
All right, so it's a big, big list of commands that we looked at.
I recommend doing a little bit of practice with each of these,
looking at the man Paige and looking at help in order to get yourself better prepared for the exam.
All right, that ends our processing text dreams. Next, we will get into file management.
See you then. Thank you.