Azure Data Factory Demo

Video Activity
Join over 3 million cybersecurity professionals advancing their career
Sign up with

Already have an account? Sign In »

14 hours 28 minutes
Video Transcription
Hello, Siberians. Welcome to Lesson 3.7. Off Monetary off this constituted Is that reserva one Microsoft Azure architect design.
Here is another view off the demo that I'll be walking you true today
in Ha Jer. I currently have this to service is running have another blob storage and I have a series with father contents that in customer information already uploaded to that as a blob storage
I also have on I just see quote database created already in Hajer.
Let me go ahead and show you this story sauces on. Then I'll show you the rest off. What are we doing?
I'm currently on the Azure Pato. You can see my storage account he And that's my sequel, several that also has my sequel database for Great. And with Fresh Start, you can see my secret database of a day
if I clicked, connects to my sequel database and I click on Query. A. Ditto
So I probably did not have permission on the network level, so let's go ahead and give myself permission on the network level. To do that, I will need to go to my sever
and I need to go on the firewall and virtual networks and now needs to select the option tohave my client I p address on. I'll save that, so that should give me network connectivity access.
The other reason why I brought you to this pain is to point out this configuration to you allow. How's your services and resources to access this Ever. I need tohave this configuration and neighborhood on my secrets ever enough for Data Factory to be able to connect to heat.
So let's go back to my secret database, and I'm going to connect the Duchess and Query editor.
So now I'm logged in to my sequel database. If I explained that you see that I currently have no tables, so let's go ahead and create a table
that quiver is gonna create a simple table that's gonna have this different columns.
So go ahead and run the query,
and that succeeded
on If I go to refresh these, I should see my near table over here. So that's kid.
The other thing that I'll show you is my storage account on the CIA's. We found that I already have in it, so let's go back to my research group. Let's selects the storage account.
I have a container,
a blob container and called customer data. And within a block, continents. The CIA's with foul called customer data that C. S V.
If I click on that and if I click on edit, you can see the information in the essentially just contents for columns. Also, just
comma separated Just first name, last name, age and email.
So that's what I currently have. Let's see what I'm gonna be showing you the rest of this demo.
So the first thing that I'll be showing you is I'll be creating on as your data factory
out. Don't go ahead and creates a pipeline within that Did the factory,
then my pipeline out. I'll specify Link Service's boats to my azure blob storage and to Maja sequel database.
I'll specify the data set for the azure blob that was specified. The CIA's referral on the data set for your secret that was specified. The as your database to a secret database table. I just created
our then create a copy activity to link those together to copy the data from the sea every foul into my sequel database table on. Of course, all of that we execute within the integration. One time in Azure data factory,
We're very fight the prerequisites already. So go ahead and show you the creation off on Azure did a factory service.
This is official representation off. What are between? Let's go create the service
on back to the Azure Pato on. What I'll Do is I'll go back over here. I'll click and create a recess
on our type in data factory
on nightly. Condemn a factory. I click and creates,
and I'll specify in name for my data factory. I'll just call leads a the F
just 11 to 0. That's the Suffolk that I'm using for every resource that I'm creating for this particular lesson
are, put it in. Every such groups are positing the results Goodbye created earlier,
and I was specified the location as UK self. I'm not going to be using C I. C. D for NASA. I'm gonna disabled the option to just get
on our great and click on Creates.
It's a simple is that it's creating a Deere factory. So after we've created a did a factory, then we can go into beauty. I did the pipelines,
so data factory now successfully created That didn't take long to create a tower. It only took a few seconds, actually.
So what about this? Like Lincoln go to results. You can see the edge of the factory
resource over here.
So the next thing that I'll be showing you is how to create an azure did a factory pipeline.
And here's the visual representation of what we were doing. So before I quiet and creates anything like the links service is our first of all nature created pipeline in Azure did a factory.
So let's go ahead and do that now.
And back to the azure. Did a factory service in Azure Pato. What our click on is our click on water and monitor here.
So in order to create a new pipeline, I could eat it. I used the wizard option here,
but what I'm gonna be doing is I'm gonna be using the proper auto vin
option over here. So let's click on water on the left hand side on. I have the option to specify what I want to order
our guide and click on hot new results and or click on ads New pipeline.
So let's give the pipeline name.
I'm giving it the name of customer data as your blob toe Azure sequel
hand. That's all I need to do for now. Created a pipeline and give it in and giving it a name.
There's no need to click on seven anything for now because we're still gonna be working within this pipeline.
So the next one I'll be showing you is how to create link services for a job blob storage and I just sequel in the factory.
So back to my pipeline enough for me to create the Link Service is I go to the lower left corner over here and I click on connections.
Now, when I click on connections, it opens up this connection. Tub and I have linked Service is a guide and click on New for New Links Service on. I can specify different categories, and then I can pick the option they want. Now, if I go under the azure category,
I should be able to see as your blob storage so quiet and click on that and I'll click and continue.
I'll have to give this link service the name. Keep it simple. Our call it as your blood storage
on when it comes to the one time that this is going to execute in so that I don't creating a new one off Aladin Beard in a custom one. I'm just gonna let
used the default as your integration. One time
he's gonna ask me for the authentication method to connect. Stow this azure blob storage account.
I'm gonna leave that sets to account key right The most the better way may be to you something light managed identity are even part like it's actually a right. But for now, let's just use that counts. Kids keep it simple.
I was specified them not gonna be integrating with key vote. I'm just gonna be providing the connection string
on DDE. What option do I have to die once used to select the account method, I will say for my job description so that final quiet and select managers description and I'll select my storage accounts directly from here and that my storage account over there.
So once I've selected the storage accounts,
I have the option to click on test connection over here. So let's go ahead and click contest connection.
So that's successful. That's great. So click on Create
On That created the links Every Treasure blob storage. Let's do the same thing for as your sequel database
so under linked service is over here. I'll just go ahead and click on New and I'll go on the Hajer and I'll look for Azure
sequel database.
And here we go. I just see called Database Likely can continue
on dhe to keep it simple again. Our colleague as your secret database. Live it to run into default indication one time abusing the connection. String out quiet and select my subscription.
I'll select my seven name, which is this,
and I'll select my database name. Which is that the debatable day. So let's go ahead and select Start
on Dhe, the other indication type that are using abusing sequel authentication.
So I was specified my user name for connecting to my sequel database,
and I'll specify my password.
I can don't test the connection again,
and that's successful and a great and click on create. So now I jotted a factory knows how to connect to this true data sauces as a blob and as your sequel.
The next thing that I'll show you how to create the deter sets for the CIA's Reef Hour in blob storage and for the adjust sequel database table,
which would be our House put. So let's greatly did a set for the inputs and data set for the output.
That's quiet to do that now,
back in the AS O pato
Yeah, for me to be able to create the sets, I go back to my pipeline
over here
on within my pipeline. I can click on
New, and I can click on new data set here.
So let's click on New Data Sets on. I'm gonna be selecting data set for a job love stoppage and I just see quote database also So data set fires your bluff storage. Let's click on Continue
and you can see the supported for much types.
So, in her case, would be using C. S V so great and select. See as we and our click on continue
and I'll need to specify the name for that. So let's specify
cost Ahmadi to see as we are specified, links every star begins and that's all specified. The azure blob storage link service, like reading earlier on this is where I have to specify the import so I can specify. The container directory of foul is just ways I can just click on browse. I go on the customer data container on, I can see
my father. So let's select that.
So that automatically populated
on. I'll leave. The imports came in from the connection stops, so that's fine. Are quiet and click on. Okay,
on what This is done. Is it giving me the option to modify the name of a year? So I'll just leave this
the way that it is our show. Your future is actually
if I click under the connection section. Yeah, I can test the connection for my link service again, but here's the option to preview my data. So if I click on that option,
it was actually reach out and preview my data so I can see what my data looks like. Yes. Oh, that's great.
The other thing that we can also do here, which are pointed to use I can also do things like, let's say my column limiter is not a comma. It's something else. I can select what I'm getting for the limiting
and I can specify other option. The other thing that I want to select a deception that says Foster role as a that because my first role in this yes, every foul is indeed my head. So let's quiet and select that option.
So that's like That's so that's so good.
That's quiet and creates the data set for Azure sequel database table.
So what I'll do is I'll go back to my pipeline
and I've got two new
data sets.
This time around, I will look for data set for as your secret database out. We can continue
and I'll give it a name.
So what I'll do is I'll give you the name off azure sea quote, table out, put on my links. Every should be. I just secret database what I created earlier
on DDE.
I can specify my table name here, So that's the table that I created. Elias also likes that table
on the imports. Kimye will be from connection stops so that final go ahead and click okay to these also
so it can give us the information there. I can click on the connection. If I previewed data, there should be nothing there, so it's currently empty.
So the next thing that I required to show you its outward creates the copy activity which ties everything together, so to speak.
what? I'm gonna be doing this. I'm gonna be creating this copy activity. There's gonna be copying information for my input. The decides to the output data sets.
Let's go ahead and do that. So if I go back to my pipeline top over here
on dhe on the pipeline tab, you can see the different activities that I have feelings. And there's a lot of them, right? So the one I'll be using it will be under the category of movement transform,
um, category
on. I can drag this copy data, so that's one of the most popular activities. So if I drag, that's to this work in pain or to this workspace, the copy data option. It's gonna give me the information down here. So first of all, I have to give you the name.
Give it a name off customer data, C S V, two Sequels. So let's go ahead and put that in there. I can give it a description if I want to, but the main part is I need to specify the sauce. That's where I specify my source data set, which is the CIA's we found
on dhe. I can can again previewed data and I can make seven modifications there.
And then I have the sink, which the sink is also the destination so I can space for my destination. They decide who to be mad. I just said, Quote, table
on. I specify Dad's.
So it's it's gonna give me the option track. You can actually say automatically create table here, so I'm not gonna be using that. But the other important one is the mapping,
because in my cases, some of the others don't don't map directly. So if I go on the map in I click on imports, schemers is gonna impart skimmers from the source in the destination so I can map them to each other.
So here we go, does the sauce and dusted destination. So let's have a look at that very quickly
so you can see that under the source.
Um, you know the destination. I have the first name and last name columns, which is not maps to anything. So in this case, I have just first as they had a column self select, forced to be marched that on our select last to be mapped to last name. So all the hardest I'm matched already.
So once I have these, I can go ahead and test the pipeline.
So I've always recommend testing the pipeline before we go ahead and publish it. So that's always a good party. So that's where we're gonna be publishing something that doesn't work.
So if I quiet and click on the bog here,
Andi, it's gonna start running that
on, we're gonna be able to view the status and see what that looks like over here.
And by the way, White's going through this testing. What I'll do is I'll go back to Hajer.
Let's go back to my research group. Let's go in there. Just sequel so I can connect back to my database.
But in my database information on dhe. I can see my table over here, So that's good.
Let's see how the task of ST ST succeeded now. So that looks good. That's quiet and verify within my sequel
database here.
So what? I'm gonna be doing this. I'm gonna be
off for my table a great and run the squaring on there we go. You see all this information now populated, caused data factory has moved them across and I can actually check the count. I think I have Tru TV 100 users.
So let's go ahead and execute on the Querida counts. Time for great and execute. That's that's 300 years is already inserted into this table, so that's great.
The final thing that we can do is we can publish this pipeline. So that's something that we can reuse. So at the top, you can click on Publish all
when I click on Publish, our is gonna be the bullet's gonna deploy this pipeline. So the data factory
and what that means is I can now shadow. So let's go successfully published I can have a trigger. I can trigger this entire workflow either. Manually, I can treat guide on a shadow. So that's something that takes the data from time to time and inserts them from one source. Randall Destination.
I contribute based on an event,
so that's one of the advantages off actually the factory.
So that brings me to the hand off this particular demonstration. I hope you found it useful on. I'll see you in the next lesson.
Up Next
AZ-301 Microsoft Azure Architect Design

This AZ-301 training covers the skills that are measured in the Microsoft Azure Architect Design certification exam. Learn strategies to plan for the exam, target your areas of study, and gain hands-on experience to prepare for the real world.

Instructed By