Azure Data Factory – Manually Deploy to other Environments using ARM Templates

When we are working with Azure Data Factory (ADF), best is to setup a development environment with DevOps (Git) for CI/CD but sometimes you might want to deploy it manually. Deploying in ADF means moving the ADF Pipeline from one environment to other environments (development, test, prod). ADF does this by using the ARM Templates to store various ADF entities like pipelines, data flows, datasets, linked services etc.

An ARM template is a JSON file that defines the infrastructure and configuration for your project.

We can move ADF to another environment using below methods:

  • Manually upload ARM Template using Azure Portal
  • ADF integration with Azure Pipelines

How to configure Git repository?

In case, you want to take a look into Git setup.

In this post, we will do it using the manual method.

Also read: How to Create Azure Data Factory for Azure SQL DB to Azure File Storage Pipeline

Manually Deploy Azure Data Factory using ARM Templates

Below are the brief steps to export and import the ADF.

  • On Azure Portal, go to your development ADF environment and export the ARM template using Export ARM Template option.
  • Go to your Test / Prod Azure subscription, and open Deploy a Custom Template service (direct link). This will help us import the exported template.
Azure Portal: Search for “Deploy”

Select Build your own template in the editor to open the Resource Manager template editor.

Build your own template in the editor
  • Select Load File, and upload the exported ARM template file (arm_template.json) from step 1. You can see the code for your ADF is all about JSON. If you want, you can change any of the parameter values at this point or on the next screen so that it points to the TEST/PROD resources.

At this point, if you already have ADF in the target environment it will deploy ADF entities like pipelines, datasets etc., else it will fail with below error.

The Resource ‘Microsoft.DataFactory/factories/RSDataFactory’ under resource group ‘ResourceGroup’ was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix\

ERROR DETAILS

Create an Azure Data Factory using ARM template

The above error comes because arm_template.json doesn’t include the Data Factory creation configuration. But don’t worry, it is an easy fix. You can find a folder called “factory” in the ARM Template zip file we downloaded in step 1, which contains data factory ARM Template file. You can either use this template to create the factory first or modify the main arm_template.json file to import/create everything with one file.

Add factory creation and dependency in a single file

That’s it for manual ADF Pipeline move from one environment to another.

If you liked the story and want to see more of this kind of content, please follow us on twitter @CursorRun or FB page @CursorRun

Default image
Rahul Sharma
Rahul is a technology enthusiast, solutions architect, trainer, and blogger, working on various Microsoft and open source solutions with more than 18 years of industry experience. He specially takes interest in designing enterprise applications, cloud integrations, IoT, and other architecture rich business solutions. Rahul is a guest author with us and shares his knowledge through technology blog posts.

Subscribe

Enter your email address below to subscribe to our website. We will never flood your inbox with unwanted emails & you can always unsubscribe anytime.

Leave a Reply