To complete the Extract, Transform, and Load (ETL) process, engineers can depend on several tools and technologies. One of these tools is Azure Data Factory (ADF). At its core, ADF is a data pipeline orchestrator and ETL tool facilitating easy and streamlined data processing.
With ADF, we can transfer data at scale and with speed while creating bespoke data-driven workflows and schedule pipelines. Besides the flexibility of ADF for processing data, it also has a lower learning curve.
This makes Azure data factory a good solution for beginners in this field and when you need a reliable solution to complete the task quickly.
In this guide, we will go through the steps to begin working with the ADF. Look for the steps to;
Setup
Creating datasets
Creating a pipeline
Debugging the pipeline
Manual triggering of pipeline
Scheduled triggering
Monitoring the pipeline
Let’s get started.
Setting Up Azure Data Factory
Before moving to the setup process, make sure of a few things;
Get a subscription with Azure. You can make an account for free and get started with the basics right away.
Next, identify your role in the Azure account. To set up everything from scratch, take on the role of an administrator.
However, to work on the child resources (datasets, pipeline, triggers, etc.), you can take on the role of a Data Factory Contributor.
Continuing with creating an Azure data factory, here are the steps.
1. Launch Data Factory
You can use Microsoft Edge or Google Chrome to access your Azure account. Once in, navigate to Azure Portal, click on Create a Resource, and select Integration. From the options given, find and click on Data Factory.
2. Add Resource
From the window, you are seeing right now, look for a tab named Basics. Then select your Azure Subscription. This is important because the data set you are about to create will be attached to this subscription.
When prompted to choose Resource, use the drop-down list to select one or Create a new resource.
Follow the “What is Azure Resource Manager” guide to know more about creating a resource.
3. Select Region
These are the geographical regions, and the supported ones are listed on the platform. Basically, these will help you know where the IT Infrastructure Managed Services Azure data factory metadata will be stored. The supported regions are;
West US
East US
North Europe
4. Enter a Name and Version
A basic practice is to give a globally unique name to the data factory. For trial purposes, you can take ADTTutorialDataFactory or anything else you want. If the name is not unique, you will get an error message, which will be easy to resolve. With the name fixed, move to Version and select V2.
It is important to read and understand the Data Factory – naming rules to add the required names to the Data Factory Artifacts.
Check the image below for a better understanding.
5. Git Configuration and Review
In the last step of setting up the ADF, move to the next tab Git Configuration. Here click on Configure Git Later and click on Review & Create. Before hitting create, you will have to pass the validation test.
6. Azure Data Factory Studio:
Once you have created the ADF. Move to the main page, click on Go To Resource and select the name of your Data Factory Page. Towards the bottom, you will see Open Azure Data Factory Studio. This will open the data factory page on a new tab.
When you experience issues with getting authorized, try clearing the browser from third-party cookies and site data.
Original Source: Azure Data Factory Overview