Automated Way to Get Latest Pipeline Execution Details of Synapse

Problem Statement

As of Dec 18, 2023, One can get the list of all execution details of a specific pipeline of an Azure Data Factory via the Powershell Module: Get-AzSynapsePipelineRun. But this module provides all execution details of the pipeline or the execution details of the specific pipeline runid.

Similarly, the Monitor section provides all execution details, and we have to scan for the latest execution per pipeline manually.

So, is there an automated way to Get the Latest Pipeline Execution Details of Synapse?

Solution

1. Open Powershell in admin mode and import the Azure SDK modules by executing the below commands.

Import-Module Az.Accounts
Import-Module Az.Synapse

2. Install the custom module by executing the below command.

Install-Module -Name AzSynapsePipelineLastExecution

3. Now, the environment is set for us to execute the commands to fetch the pipeline details.

To get the list of all pipelines in Synapse and their corresponding last execution details, execute the command below.

Get-AzSynapsePipelineLastExecution -WorkspaceName “<<WorkspaceName>>”

To get the details of a specific pipeline in Synapse and its corresponding last execution, execute the below command.

Get-AzSynapsePipelineLastExecution -WorkspaceName “<<WorkspaceName>>” -PipelineName “<<PipelineName>>”

To get the list of all pipelines in Synapse and their corresponding last execution details in a specific time range, execute the below command.

Get-AzSynapsePipelineLastExecution -WorkspaceName “<<WorkspaceName>>” -StartRange "<<StartDate>>" -EndRange "<<EndDate>>"

Output Results

1. Overall Synapse output

2. Specific Pipeline output

3. Specific Range output

Note. In case no values are provided to StartRange and EndRange, then default values would be assigned as 1900-01-01 and the latest datetime accordingly.

This would avoid manually identifying the latest execution details of a pipeline in Synapse.