Alerting In Prometheus With AlertManager

In this article we will try to implement Alerting in Prometheus with AlertManager. We know Prometheus helps us to predict potential problem in our services by providing details and related metrics but Alerting helps us to notify the occured problem and allow developer to identify problems.

N.B. To have a better understanding in this article, I would recommend to go through my other article "Instrumenting ASP.NET Core Application for exporting metrics to Prometheus"

AlertManager handles alerts sent by the Prometheus server and notifies end-user through E-mail, Slack, or other tools. For this we need to define alert rules in Prometheus configuration and while scraping metrics Prometheus checks if any alert condition hits depending on the defined rules. If alerts occur then Prometheus pushes it to AlertManager.

AlertManager manages all alerts through its pipeline of Silencing, grouping, inhibition, and sending out notifications.

• Silencing - Silencing is to mute alerts for a given period of time.
• Grouping - Grouping groups alerts of similar nature into a single notification. This helps prevent multiple notifications simultaneously to the receivers.
• Inhibition - To suppress specific alerts if other alerts are already fired.

To Configure AlertManager we need to follow these steps,

Step 1 - Configure Alert Rule

Rule file holds the conditions depending on which Prometheus triggers an alert. Inside the "Prometheus" folder create a new file "alert.yml". And inside the "alert.yml" put the following,

Here this alert states that if any target is "down" for 1 minute it will trigger an alert.

Step 2 - AlertManager Config File

Create a new folder "alertmanager" and inside it create a new file "alertmanager_Mail.yml" that holds the configuration for Alert Manager.

route:
  group_by: [Alertname]
  # Send all notifications to me.
  receiver: email-me

receivers:
- name: email-me
  email_configs:
  - to: <receiver's email>
    from: <Sender's email>
    smarthost: smtp.gmail.com:587
    auth_username: <Sender's email>
    auth_identity: <Sender's email>
    auth_password: <Sender's mail password>
    send_resolved: true

Step 3 - Update prometheus.yml file to configure Prometheus server to talk with AlertManager Service

We need to update our previously created (Previous article) prometheus.yml file to configure the AlertManager.

Here we have configured the alertmanager and it will be running on port:9093.

Step 4 - Update docker-compose.yml File

We need to update our docker-compose.yml File with the following code. With this updated YML we will pull alertmanager image from dockerhub and will set the "alertmanager.yml" file as the config file for alertmanager.

version: '3'

services:
  prometheus:
    image: prom/prometheus:v2.21.0
    ports:
      - 9090:9090
    volumes:
      - ./prometheus:/etc/prometheus
      - prometheus-data:/prometheus
    command: --web.enable-lifecycle  --config.file=/etc/prometheus/prometheus.yml
  alertmanager:
    image: prom/alertmanager:latest
    container_name: alertmanager
    ports:
      - 9093:9093
    volumes:
      - ./alertmanager:/etc/alertmanager
    command:
      - '--config.file=/etc/alertmanager/alertmanager_Mail.yml'
      - '--storage.path=/alertmanager'
  pushgateway:  
    restart: always  
    image: prom/pushgateway:latest  
    ports:  
      - 9091:9091

volumes:
  prometheus-data:

Step 5 - Run the container

Now we can run the docker container.

After successful installation of alertmanager we can access the AlertManager UI by browsing "localhost:9093".

As we have configured the Prometheus server for AlertManager. Now if we navigate to the rule option we can see our available rules.

Triggering Alert

To observe the alerting Mechanism in Prometheus, we intentionally take one of our targets/service down. For this, we stop our service "app_prom" in docker with is basically an asp.net core application and it is dockerized.

Now if we navigate to the Prometheus Web UI and click on the "Target" option we can see that alert has been triggered and it will go from the "pending" state to the "Firing" state.

After firing the alert from the Prometheus if we navigate to the AlertManager UI browsing "localhost:9093" we can see that the triggered alert has been sent to the AlertManager.

Along with this, we will also receive an alert on our Mail.

Like this, we can define multiple alert rules like resource usages, memory usages, request observing. And if somehow these rules are triggered, immediately Prometheus will notify us via mail and we can take proper action for this. So through this, we can ensure the reliability of our services.

With this, our configuration for AlertManager finishes here. In my next article, I will try to discuss the Push Gateway of Prometheus.

Happy Coding!!!