Create A Copy Of Azure Data Factory Using Azure ARM Templates

Introduction

In day-to-day operations, we must have faced requirements to backup and restore or copy an Azure data factory from existing to new ones. In today's demo we will see how can we backup and restore the Azure data factory using ARM templates export/import option in azure data factory studio.

Steps

I will create a simple pipeline that will copy few rows of data from my azure blob storage into an Azure SQL database. For this, I am going to create a blob storage and add a file into it which we are going to use to upload records into the Azure SQL database. I have created a database and with it a table called financial which could hold the data that we are going to import from blob storage.

You could see from the above picture that currently there are no records in the table.

The pipeline has completed successfully. Now check and we can see the database table and the records that have been copied from blob to database table.

Backup Azure data factory

As stated earlier, we are going to utilize the ARM template Export/Import method to backup and restore the Azure Data Factory. Go to the ‘Manage’ tab on the left side pane and select ‘ARM template’ under source control. The idea is to export the ADF and restore it into a new ADF pipeline.

Once you click Export ARM template you will have a zip file downloaded into your folder which you have to unzip to see the highlighted JSON file is available. With this step we have completed the backup process. Now we have to use this file to restore it into a new ADF.

Restore Azure data factory

Before starting with the restore process we have to create a new data factory from azure portal. I have named the new data factory as adfdemo100222. Once created open the same in data factory studio and click on ‘manage’ tab from the left side pane, followed by the options highlighted in the image below to import ARM template. Ultimately we are going to load the ‘arm_template.json’ file from the folder which we have extracted from the zip previously.

Since we are loading the ARM template file from the backup we have created it is a custom deployment. So instead of choosing the common templates, let's build our own template from the editor.

In the next screen, you will see the edit template with a few lines of code, you can go ahead and delete that and click on ‘Load file’ option above to load the arm_template.json file that we have extracted from the zip file.

I have modified the name of the file which contained the old name to ‘adfdemo100222’ for identification.

After modifying the name now save the file where will be required to enter the connection string details with the resource groups and region.

For ‘blob storage connection string’ go to the storage and then access keys, click Show keys to copy the connection string details.

1. Data factory name
2. Connection string of blob storage where the sample files has been stored
3. Connection string of SQL database

For SQL database connection string details go to the databaseà connection string tab in the left side pane. You can copy the string available in the ADO.NET tab and make sure you replace the password before pasting into the create deployment page.

Once done, go ahead and make sure it is successfully deployed.

We could see the copy of our restored pipeline from the old data factory instance. So you can go ahead to see how the restored pipeline is working.

Go to the pipeline and just run the pipeline and check the count of the table records in the database table. The count should be increased as the data from the file should have been appended in the table. In my case, the file contained 99 rows and there were 4 existing rows already taking the count to 103.

We could see that the table count has been increased to 203 rows, 103 existing and 99 appended in the second run.

Summary

This is the simplest form of backup and restore for an azure data factory pipeline using ARM template. Hope the live demo will come handy in case you want to implement such methods as well as for your understanding.