Azure DevOps Build Trigger: Integrating with Data Factory/Synapse Pipeline

Problem Statement 

The blog Trigger Azure DevOps Build Pipeline via Azure Data Factory / Synapse Pipeline describes how to trigger a DevOps build pipeline leveraging PAT authentication. As PATs are simply bearer tokens, meaning token strings that represent a user’s username and password, they're incredibly risky to use as they can easily fall into the wrong person’s hands.

Is it possible to trigger an Azure DevOps Build Pipeline via Azure Data Factory (ADF) / Synapse leveraging Managed Identity authentication for better security purpose.

Prerequisites 

1. Azure Data Factory / Synapse

Solution 

Azure DevOps supports Service Principal / Managed Identity authentications as stated in MSFT Doc Use service principals & managed identities

We would be leveraging Azure DevOps REST API : Runs – Run Pipeline to Trigger the Build pipeline run.

Github Code

Sign in to your organization ( https://dev.azure.com/{yourorganization} ) and select a project. Choose Project settings, and then choose Permissions. Then select Build Administrators > Members > Add. Enter the Managed identity you want to add to the permission, and then select Save.

Create 3 Pipeline parameters.

For which below are the values required

a) Organization: This represents the name of the Azure DevOps organization.

Login to the Azure DevOps and click on the Build Pipeline.

Sample URL format: https://dev.azure.com/organization/project/_apis/pipelines/pipelineId

The one highlighted in yellow represents the organization.

b) Project: The one highlighted in Green represents the Project.

c) PipelineId: The one highlighted in Red represents the PipelineId.

Trigger Build pipeline Config

• URL: @concat('https://dev.azure.com/',replace(pipeline().parameters.organization,' ','%20'),'/',replace(pipeline().parameters.project,' ','%20'),'/_apis/pipelines/',pipeline().parameters.pipelineId,'/runs?api-version=6.0-preview.1')
• Body: {"resources":{"repositories":{"self":{"refName":"refs/heads/main"}}}}
• Resource: 499b84ac-1321-427f-aa17-267ca6975798

The Trigger API call is asynchronous. Hence, you do not know whether the Build Pipeline has actually succeeded. The successful execution of the web activity does only mean that DevOps Pipeline was a success. To check the status of the Pipeline one can leverage Web activity to trigger refresh status via REST API.

• URL: @concat('https://dev.azure.com/',replace(pipeline().parameters.organization,' ','%20'),'/',replace(pipeline().parameters.project,' ','%20'),'/_apis/pipelines/',pipeline().parameters.pipelineId,'/runs/',string(activity('TriggerBuilldPipeline').output.id),'?api-version=7.0')
• Resource: 499b84ac-1321-427f-aa17-267ca6975798

We have to add a polling pattern to periodically check on the status of the refresh until it is complete. We start with an until activity. In the settings of the until loop, we set the expression so that the loop executes until the output of the above web activity is equal to Completed.

Expression: @equals(activity('GetBuildPipelineDetails').output.state,'completed')

The Final activity is the IF activity that checks the Refresh status and leverages a Fail activity to fail the pipeline in case of DevOps Pipeline failure.

Expression: @not(equals(activity('GetBuildPipelineDetails').output.result,'succeeded'))

Fail Message: @concat('Pipeline ',activity('GetBuildPipelineDetails').output.pipeline.name,' ',activity('GetBuildPipelineDetails').output.result)

Output