Invoke Azure Data Factory Pipeline via MSFT Fabric

Problem Statement

Is it possible to Trigger Azure Data Factory (ADF) pipeline from MSFT Data Factory, as there is no out-of-the-box Activity for Azure Data Factory?

Prerequisites

  1. Azure Data Factory
  2. MSFT Data Factory
  3. Organizational Account /Service Principal
  4. APIs used

Solution

Note. To trigger an ADF pipeline, the Organizational Account / Service Principal must have the below minimum permission on the ADF.

/ Microsoft.DataFactory/factories/pipelines/createrun/action

1. Create a pipeline in MSFT Fabric Data Factory and add the parameters below to the pipeline.

Parameters

2. Drag and drop the Web activity with the below configurations.

Web

Where create a new connection in web activity with organizational account authentication and appropriate base uri and token audience, click sign in to proceed.

Connection

Where

Connection Setting

Relative

  • URL
    /subscriptions/@{pipeline().parameters.SubscriptionId}/resourceGroups/@{pipeline().parameters.ResourceGroupName}/providers/Microsoft.DataFactory/factories/@{pipeline().parameters.DataFactoryName}/pipelines/@{pipeline().parameters.PipelineName}/createRun?api-version=2018-06-01
  • Method: POST
  • Body: Since we are doing a POST request, we need to define the body and pass a space as the value.

3. The Trigger ADF API call is asynchronous. Hence, we do not know whether the pipeline has actually been completed. The successful execution of the web activity doesn’t mean that the ADF pipeline was a success. To check the status of the Pipeline, add another web activity with the below config.

Setting

Use the same connection as created in #2.

  • Relative URL
    /subscriptions/@{pipeline().parameters.SubscriptionId}/resourceGroups/@{pipeline().parameters.ResourceGroupName}/providers/Microsoft.DataFactory/factories/@{pipeline().parameters.DataFactoryName}/pipelineruns/@{activity('Trigger ADF Pipeline').output.runId}?api-version=2018-06-01
  • Method: Get

4. We have to add a polling pattern to periodically check on the status of the trigger until it is complete. We start with an until activity. In the settings of the until loop, we set the expression so that the loop executes until the output of the above web activity is not equal to "Failed" or "Succeeded".

Pipeline execution

Expression

@or(
    equals(activity('Get Pipeline Details').output.status, 'Failed'),
    equals(activity('Get Pipeline Details').output.status, 'Succeeded')
)

Within Until Activity

Canvas

Output

Output


Similar Articles