Monitor Azure Synapse Analytics Using Azure Monitor

A few days ago, I was presenting my topic regarding synapse in ‘Global Azure Bootcamp – 2022’ when one of the participants questioned about how to get notified regarding the status of pipeline activities in a synapse pipeline. After the event, I explored some of the present and effective options that are available and decided to document them with a demo for each method.

Introduction

In Azure Synapse whenever dealing with data ETL/ELT activities, at any point of time the engineer might need to know the state and health of repositories by going through the metrics. The engineer or administrator can accomplish this by setting up notifications within the synapse workgroup for critical metrics which when crossing those thresholds, notification will be sent to the response group for corrective actions.

Create alerts

Once you click on ‘Create new alert’, you will see tabs with multiple options. The ‘Scope’ tab is to select the Synapse workspace if in case you have more than one, here we have only one as listed below.

In the condition tab click on ‘Add condition’ to create new criteria for the alert. Once you click you will see a pop-up with list of metrics and log values which you can select one or more based on your requirement. These are called as ‘Signals’ in azure diagnostics terminology.

Please note that the alert rule in azure is not a free service. $0.10 will be charged for each rule created

Let’s select the signal ‘Pipeline runs ended’ from the list. This will trigger the alert based on pipeline activity or its output, whatever the condition we specify in the coming steps. I will leave that with default hours in the chart period where it will check for the pipeline status in the last 6 hours.

Scroll down to use the next option Split by Dimensions which will let you monitor multiple metric time-series with a single alert rule. A metric time-series is a series of measurements (or "metric values") captured over a period of time. While I am not going to use the time-series monitoring now, just to show you I have selected the ‘Failure Type’ dimension and custom dimension values named Data error & Data Type Mismatch.

Further scrolling down will lead you to ‘Alert Logic’ where you can set the threshold to Static & Dynamic. Since for this demo I have selected pipeline run, I am selecting the Static threshold to alert if the pipeline run ends more than once.

An action group is a collection of notification preferences defined by the owner of an Azure subscription. Azure Monitor, Service Health and Azure Advisor alerts use action groups to notify users that an alert has been triggered. Various alerts may use the same action group or different action groups depending on the user's requirements.

Continuing towards the next step in creating the alert, you will be moved into ‘Actions’ tab to select an existing action group or create a new one.

I created a new action group and setup email notification in the next tab with my own email where I like to be notified. Once the action group is created successfully you will receive an email confirmation that you have been added to the action group.

Once this is all complete now go ahead and create alert rule with after providing it a name. Just visit the dashboard once the alert has been created to see how many notifications has been received and their severity. We don’t have any in the below screenshot as we haven’t run any pipeline yet. Click on the ‘Alert rules’ button above to view the alerts created for this workspace.

Run Pipeline

Now the alert and all the steps have been completed, it is time to test by running the pipeline. For this demo I have created a simple ‘Get Metadata’ activity to adls gen2 storage container where we will fetch some basic details. As per our plan the alert we have created should notify us if any pipeline runs more than once.

The pipeline was successful, and the output triggered has the ‘childitems’ list as per our selection in field list. Let’s run one more time but with ‘Item names’ parameter. This is purely optional, I edited it just to show you the difference, you can even go ahead and run the same pipeline twice without changing anything as our core agenda has nothing with the pipeline’s output.

The alerts have been triggered successfully in my mail and the alert dashboard as well.

Once after I ran the pipeline twice initially, I didn’t receive any alert in dashboard or in my mail. I desperately created another alert to check, hence there were two alert entries whereas we created only one earlier.

Conclusion

Using the azure alert metrics is one of the most effective and simple methods for setting up the alerts and triggers for a synapse pipeline, we will explore the other methods in future articles.