System Monitoring And Alerting With Prometheus (Docker)

In this blog, we will try to discuss about Prometheus and the installation Process of Prometheus in docker.

When any services or software are running in production, we don't know too much about how our applications/services are performing. In the production model, it’s very important to have hardware or application-level insights (like Errors, Lack of Resources, Response Latency, Overloaded, etc.) of the deployed service.

Luckily, there are several tools that provide all sorts of metrics for these services. Prometheus is one of them and  is one of the most commonly used tools.

Prometheus is an open-source system monitoring and alerting toolkit. It is specially designed for large-scale environments, to monitor system health, CPU usages, Memory Usages, etc., and generate alerts if any unexpected behavior or crashes happens.

1. It collects multi-dimensional data models with time series data identified by metric name and key/value pairs.
2. Time series collection happens via a pull model over HTTP besides this, the Push model is also supported.
3. Provides flexible query language called PromQL.

Components of Prometheus ecosystem

1. Prometheus Server
   This is the main component of the Prometheus architecture. The main monitoring work is done by this server.
   
   This server is made up of three parts,
    
   • Timeseries Database(TSDB) - stores all the metrics data as time series. Data is stored on the HDD/SSD.
   • Data retrieval worker - pulls or gets metrics from the Applications or servers and stores them to the TSDB.
   • HTTP Server - It’s a server API that accepts queries over the stored data and returns the query to the visualization tool.
      
2. Client Libraries
   For instrumenting application code.
    
3. Push Gateway
   For supporting short-lived jobs.
    
4. Alertmanager
   Handles alerts.

What are Targets and Metrics?

While working with Prometheus, two terms we generally use are Targets and Metrics.

Targets

Basically, a Prometheus server monitors a particular service/application and this particular service/application is known as the Targets.

For example,

• Any single application
• Linux/Windows server
• Services like Database

Metrics

Every target has some Unit.

For example,

1. CPU Status
2. Memory/Disk space usages
3. Request Duration
4. Number of Exceptions

These units that we would like to monitor for a specific target are called Metrics.

Prometheus Metrics are categorized into 4 types

1. Counters
   Counter metrics type is used for any value that only increases. Such as Request count, Tasks completed, Error count.
    
2. Gauges
   Gauges metric type can be used for values that go up as well as down, such as Memory usages, queue size, number of requests in progress.
    
3. Histograms
   Histograms metrics type measures the frequency of value observation that falls into specific predefined buckets. such as Request count
    
4. Summaries
   Summaries and histograms share a lot of similarities. The key differences are: With histograms, quantiles are calculated on the Prometheus server but with summaries, they are calculated on the application server with a specific quantile. Summary data can not be aggregated from a number of application instances.

Install Prometheus In Docker(Linux Container)

Step 1 - Configure prometheus.yml file

To setup Prometheus, from the root directory create a new folder (here we have named it Prometheus), and inside it create a new file named “prometheus.yml” which holds the main config of Prometheus.

Here scrape_interval defines how often to check for new metrics. If a scrape takes longer than scrape timeout, Prometheus will cancel the scrape.

rule_files tells Prometheus where to search for the alert values.

scrape_config tells Prometheus where the application resides and from which endpoint Prometheus will scrape(default path is “/metrics”)

Step 2 - Configure docker-compose.yml file

In the root directory create a new file “docker-compose.yml” and put the following code into the file.

Here the volume “./prometheus:/etc/prometheus” mounts our Prometheus folder in right place for the image to pick up configuration.

“prometheus-data/prometheus” is used to store the scraped data so that they are available after a restart.

Here the command is optional. If we use –web.enable-lifecycle, we can reload configuration files without restarting Prometheus.

Step 3 - Start Prometheus

To start the Prometheus from the terminal navigate to the root directory.

After that, run the command “docker-compose up –d” which will finally start Prometheus.

Here I’m using WSL for running the command, you can use the command prompt then you don’t need to use “/mnt/”.

After successful installation, if we open http://localhost:9090 in the browser we can see the Prometheus web UI.