In today’s tutorial, we are going to take a look at one of the most popular monitoring stacks : Grafana and Prometheus.

Prometheus is a time series database, created in 2012 and part of the Cloud Native Computing Foundation, that exposes dozens of exporters for you to monitor anything.

On the other hand, Grafana is probably one of the most popular monitoring tools.

In Grafana, you create dashboards that bind to datasources (such as Prometheus) in order to visualize your metrics in near real-time.

Grafana & Prometheus natively bind together, so today we are going to see how you can setup Prometheus and Grafana on your Linux system.

This tutorial works for the latest versions of Ubuntu (19.04) and Debian (10 Buster).

Ready?

I – What You Will Learn

By following this tutorial, you are going to learn about the following concepts:

  • Installing a Prometheus server on your Linux server.
  • How to configure authentication and encryption on Prometheus with a reverse proxy
  • How to install and configure Grafana for Prometheus
  • How to install your first exporter in Prometheus to monitor Linux system metrics.
  • How to import your first monitoring dashboard on Grafana

II – Installing Prometheus on Ubuntu and Debian

First, we are going to see how you can install the latest version of Prometheus and configure it for your Linux server.

a – Download Prometheus

First, head over to the Prometheus downloads page.

Make sure to filter for your operating system and your CPU architecture (in my case Linux and amd64)

a – Download Prometheus prometheus-downloads-page

Make sure to select the latest stable version, and not the “rc” one, as it is not considered stable enough for now.

Download the archive on your system by clicking on the archive, or by running a wget command if you are using the Terminal.

$ wget https://github.com/prometheus/prometheus/releases/download/v2.11.2/prometheus-2.11.2.linux-amd64.tar.gz

You should now have the tar.gz file on your system.

Untar it to extract the files in the archive.

$ tar xvzf prometheus-2.11.2.linux-amd64.tar.gz

a – Download Prometheus tar-xvzf-prom

The archive contains many important files, but here is the main ones you need to know.

  • prometheus.yml: the configuration file for Prometheus. This is the file that you are going to modify in order to tweak your Prometheus server, for example to change the scraping interval or to configure custom alerts;
  • prometheus: the binary for your Prometheus server. This is the command that you are going to execute to launch a Prometheus instance on your Linux box;
  • promtool: this is a command that you can run to verify your Prometheus configuration.

We are not going to execute directly the Prometheus, instead we are going to configure it as a service.

It will bring more robustness and reliability in case our Prometheus server were to stop suddenly.

b – Configuring Prometheus as a service

First of all, for security purposes, you are going to create a Prometheus user with a Prometheus group.

$ sudo useradd -rs /bin/false prometheus

Make sure to move the binaries to your local bin directory.

I stored my binaries in a Prometheus folder, located on my home directory.

Here’s the command to move them to the bin directory.

$ cd Prometheus/prometheus-2.11.2.linux-amd64/ 
$ sudo cp prometheus promtool /usr/local/bin

Give permissions to the Prometheus user for the prometheus binary.

$ sudo chown prometheus:prometheus /usr/local/bin/prometheus

Create a folder in the /etc folder for Prometheus and move the console files, console libraries and the prometheus configuration file to this newly created folder.

$ sudo mkdir /etc/prometheus
$ sudo cp -R consoles/ console_libraries/ prometheus.yml /etc/prometheus

Create a data folder at the root directory, with a prometheus folder inside.

$ sudo mkdir -p data/prometheus

Give the correct permissions to those folders recursively.

$ sudo chown -R prometheus:prometheus data/prometheus /etc/prometheus/*
Great!

You are now set to create a Prometheus service.

Head over to the /lib/systemd/system folder and create a new file named prometheus.service

$ cd /lib/systemd/system
$ sudo touch prometheus.service

In order to see the different launch options for Prometheus, you can run the prometheus command with a h flag.

b – Configuring Prometheus as a service

$ prometheus -h
usage: prometheus [<flags>]

The Prometheus monitoring server

Flags:
  -h, --help                     Show context-sensitive help (also try --help-long and
                                 --help-man).
      --version                  Show application version.
      --config.file="prometheus.yml"
                                 Prometheus configuration file path.
      --web.listen-address="0.0.0.0:9090"

Edit the newly created file, and paste the following content inside.

b – Configuring Prometheus as a service 1

$ sudo nano prometheus.service

[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path="/data/prometheus" \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-admin-api

Restart=always

[Install]
WantedBy=multi-user.target

Save your file, enable your service at startup, and start your service.

$ sudo systemctl enable prometheus
$ sudo systemctl start prometheus<

b – Configuring Prometheus as a service prometheus-start

Now that we have our Prometheus server running, let’s connect to the Web UI to make sure that everything is okay.

Unless you modified it in the service file, your Prometheus server should be running at http://localhost:9090.

By default, this is what you should see on your screen.

prometheus-target

By default, Prometheus should start monitoring itself.

Click on Status, then Targets in top bar menu, and verify that you have one target : the Prometheus server itself.

prometheus-target-2

Great! The state is “UP” for the Prometheus target. Everything is running smoothly.

Now that our Prometheus server is running, it is time to configure a reverse proxy for authentication and encryption.

III – Setting up a reverse proxy for Prometheus

Prometheus does not natively support authentication or TLS encryption.

If you are running Prometheus locally, it doesn’t not matter that much.

But if you were to expose Prometheus endpoints to the outside world, you would want to setup a strong authentication for users to check Prometheus.

For this tutorial, I will be using NGINX as a reverse proxy, but there are many other proxies out there that you can choose from.

a – Install NGINX on Debian 10

The steps are pretty much the same for Ubuntu 18.04/19.04.

Simply install nginx by downloading it from the apt sources.

$ sudo apt update
$ sudo apt-get install nginx

This should create a nginx service on your server, and it should be started by default.

a – Install NGINX on Debian 10

$ sudo systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-07-05 06:01:12 UTC; 1 months 10 days ago
     Docs: man:nginx(8)
 Main PID: 15792 (nginx)
    Tasks: 3 (limit: 4704)
   CGroup: /system.slice/nginx.service
           ├─15792 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;

By default, NGINX will start listening for HTTP requests in the default http port, which is 80.

If you are not sure about the port used by NGINX, run the following command.

The last column will tell you the port used by NGINX.

$ sudo lsof | grep LISTEN | grep nginx
nginx     15792     root    8u     IPv4         1140223421       0t0        TCP *:http (LISTEN)
Great! Let’s do a quick browser check to see the default HTTP page distributed by NGINX.

nginx-default-page

b – Configuring NGINX as a reverse proxy

In this case, we are not interested in using NGINX as a standard web server.

Instead, we want to use NGINX as a reverse proxy.

As a reminder, a reverse proxy is an entity that proxies requests to other entities given a set of internal rules defined in the proxy configuration.

reverse-proxy

By default, the NGINX configuration file is located at /etc/nginx/nginx.conf.

In this file, you have to make sure that it includes virtual hosts included in the conf.d directory.

$ cat /etc/nginx/nginx.conf

##
# Virtual Host Configs
##

include /etc/nginx/conf.d/*.conf;
include /etc/nginx/sites-enabled/*;

The conf.d directory is where we are going to create our reverse proxy configuration file for Prometheus.

Create a new file in this directory called prometheus.conf, and start editing the file.

$ cd conf.d/
$ sudo touch prometheus.conf

Paste the following configuration in your configuration file.

server {
    listen 1234;

    location / {
      proxy_pass           http://localhost:9090/;
    }
}

Restart your NGINX server, and browse to http://localhost:1234.

$ sudo systemctl restart nginx
(if you have any errors)
$ sudo journalctl -f -u nginx.service

proxying-prometheus

Great!
Prometheus is now sitting behind a reverse proxy.

c – Changing Prometheus configuration for reverse proxying

Now that Prometheus is accessed by a reverse proxy, we need to change the configuration of Prometheus for it to bind to the reverse proxy. Head over to /lib/systemd/system and change your service configuration file.

c – Changing Prometheus configuration for reverse proxying

$ cd /lib/systemd/system 
$ sudo nano prometheus.service

ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path="/data/prometheus" \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.enable-admin-api \
  --web.external-url=https://localhost:1234

Restart your service, and make sure that your server is up and running correctly.

$ sudo systemctl daemon-reload
$ sudo systemctl restart prometheus
$ journalctl -f -u prometheus.service
Awesome!

Make sure that Prometheus is still reachable at http://localhost:1234.

d – Enable reverse proxy authentication

Before moving on with the reverse proxy, let’s have a quick look at how you can set up credentials for your Prometheus instance.

Prometheus does not handle built-in authentication, so we are going to rely on reverse proxy authentication for this.

Create a password file in the Prometheus configuration folder.

If you don’t have the htpasswd executable on your system, make sure to install it first.

$ sudo apt-get install apache2-utils
$ cd /etc/prometheus
$ sudo htpasswd -c .credentials admin

Choose a strong password, and make sure that the pass file was correctly created.

d – Enable reverse proxy authentication htpasswd-file

Now, it is time to import it into your NGINX configuration file.

Head over to /etc/nginx/conf.d and change your server configuration by adding your credentials.

d – Enable reverse proxy authentication

$ cd /etc/nginx/conf.d

server {
    listen 1234;

    location / {
      auth_basic           "Prometheus";
      auth_basic_user_file /etc/prometheus/.credentials;
      proxy_pass           http://localhost:9090/;
    }
}

Save your file, and restart NGINX

$ sudo systemctl restart nginx
$ journalctl -f -u nginx.service

If everything is okay, go to http://localhost:1234 and you should be prompted with user credentials.

d – Enable reverse proxy authentication auth-required

Provide the correct credentials, and you should be good to go!

e – Enabling HTTPS on the reverse proxy<

For the last chapter, we are going to enable the TLS protocol on our NGINX instance.

Using HTTPS will encrypt requests done to the server, but most importantly it will encrypt the server responses containing all the metrics.

Creating keys for Prometheus

For this chapter, we are going to use self-signed certificates.

However, if you have a certificate authority, you can issue a certificate request to them.

You will need the certtool in order to create certificates on your Linux instance.

Install the gnutls related packages (utils for Ubuntu and bin for Debian based distributions)

(Ubuntu)
$ sudo apt-get install gnutls-utils

(Debian)
$ sudo apt-get install gnutls-bin

Create a new directory in /etc/ssl in order to store your Prometheus keys.

$ cd /etc/ssl
$ sudo mkdir prometheus

Go into the prometheus directory, and start by generating a private key for Prometheus.

$ sudo certtool --generate-privkey --outfile prometheus-private-key.pem

Now that you have a private key, let’s generate a certificate (or a public key) for Prometheus.

$ sudo certtool --generate-self-signed --load-privkey prometheus-private-key.pem --outfile prometheus-cert.pem
  # The certificate will expire in (days): 3650
  # Does the certificate belong to an authority? (Y/N): y
  # Will the certificate be used to sign other certificates? (Y/N): y
  # Will the certificate be used to sign CRLs? (y/N): y
Great!

You now have two keys sitting in your /etc/ssl/prometheus.

Configuring HTTPS on NGINX

Now that our keys are created, it is time to import them into our NGINX configuration.

Go back to your /etc/nginx/conf.d directory, and edit your prometheus.conf file.

Configuring HTTPS on NGINX

$ cd /etc/nginx/conf.d
$ sudo nano prometheus.conf

server {
    listen 1234 ssl;
    ssl_certificate /etc/ssl/prometheus/prometheus-cert.pem;
    ssl_certificate_key /etc/ssl/prometheus/prometheus-private-key.pem;

    location / {
      auth_basic           "Prometheus";
      auth_basic_user_file /etc/prometheus/.credentials;
      proxy_pass           http://localhost:9090/;
    }
}

Again, restart your NGINX server, and your virtual host should now be switched to HTTPS.

$ sudo systemctl restart nginx
$ sudo journalctl -f -u nginx.service

Verify that Prometheus is now delivered via HTTPS

We are going to use cURL in order to verify that Prometheus is now served via HTTPS.

If you don’t own cURL on your instance, make sure to install it first.

$ sudo apt-get update
$ sudo apt-get install curl

$ curl -u admin -k https://localhost:1234/metrics
Enter host password for user admin:

From there, this is what you should see on your screen.

Configuring HTTPS on NGINX prometheus-https

Great! We are now ready to install Grafana to visualize Prometheus metrics.

IV – Installing Grafana

The Grafana installation was already covered extensively in our of our previous articles, so make sure to read it to setup your Grafana server.

a – Configure Prometheus as a Grafana datasource

First, head to the datasources panel by clicking on Configuration > Data sources via the left menu.

IV – Installing Grafana datasources-grafana

Click on “Add a datasource”

add-data-source-1

Select a Prometheus data source on the next window.

prometheus-datasource

Here is the configuration for Prometheus. Make sure to skip the TLS verification as you are using a self-signed certificate.

prometheus-config

Click on “Save and Test” at the bottom of your configuration window, and make sure that your data source is working properly.

data-source-is-working-1

V – Installing the Node Exporter to monitor Linux metrics

Now that your Prometheus data source is working, it is time to install your first exporter.

As a reminder, exporters are standalone entities that regularly aggregate metrics for a wide variety of targets : operating systems, databases, websites.

If you feel like you are missing some of the concepts about Prometheus, here’s a guide to understand them.

The Node Exporter is an exporter that gathers metrics abour your Linux system : the CPU usage, the memory usage as well as various statistics on your filesystems.

a – Downloading the Node Exporter

First of all, we are going to download the Node exporter on our system.

Head over to https://prometheus.io/download/ and select Linux operating systems for amd64 CPU architectures.

Scroll down, and find the section dedicated to the node exporter.

node-exporter-prometheus

Simply click on it, or copy the link and run a wget command.

$ wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

Extract the archive to access the files inside it.

$ tar xvzf node_exporter-0.18.1.linux-amd64.tar.gz

Inside your newly created directory, you should now see the node_exporter binary ready for use.

Again, move this executable to your /usr/local/bin folder.

$ cd node_exporter-0.18.1.linux-amd64/
$ sudo cp node_exporter /usr/local/bin

Create a new user for the node exporter, and change the permissions for the node exporter binary.

$ sudo useradd -rs /bin/false node_exporter
$ sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

b – Create a Node Exporter service

Again, we are going to run the node exporter as a service.

Go to the /lib/systemd/system folder and create a new file named node_exporter.service

$ cd /lib/systemd/system
$ sudo touch node_exporter.service

Paste the following content in your node exporter service.

By default, a lot of modules (cpu, mem, disk) are already enabled by default, so we only need to enable a few more.

b – Create a Node Exporter service

[Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=node_exporter
Group=node_exporter
ExecStart=/usr/local/bin/node_exporter \
    --collector.mountstats \
    --collector.logind \
    --collector.processes \
    --collector.ntp \
    --collector.systemd \
    --collector.tcpstat \
    --collector.wifi

Restart=always

[Install]
WantedBy=multi-user.target

When you are done, enable your service, and start it.

$ sudo systemctl daemon-reload
$ sudo systemctl enable node_exporter
$ sudo systemctl start node_exporter

(Make sure that everything is running smoothly)
$ sudo journalctl -f -u node_exporter.service

c – Configure the Node Exporter as a Prometheus target

Now that the node exporter is up and running, we need to tell Prometheus to scrape it periodically.

To do that, go to your Prometheus configuration file located at /etc/prometheus/prometheus.yml.

$ cd /etc/prometheus
$ sudo nano prometheus.yml

In the static_configs part of your configuration file, add a new entry for the node exporter.

static_configs:
            - targets: ['localhost:9090', 'localhost:9100']

Restart Prometheus for your changes to be applied.

$ sudo systemctl restart prometheus

Head over to https://localhost:1234/targets and make sure that your target is correctly scraped.

c – Configure the Node Exporter as a Prometheus target targets-prom

Great! Prometheus is now scraping metrics from the Node Exporter.

VI – Building a Grafana dashboard to monitor Linux metrics

The last step will be to import a Grafana dashboard that reflects the metrics that we are gathering with Prometheus.

The “Node Exporter Full” dashboard created by idealista seems to be a great candidate.

Looking to create your own Grafana dashboard using the UI or the API? Follow the guide!

a – Importing a Prometheus dashboard into Grafana

We don’t have to design the entire dashboard by ourselves, we can import idealista’s dashboard right into our Grafana server.

To do so, click on “Import” by hovering the “Plus” icon, and clicking on “Import.

import-grafana-dashboard

On the next window, simply enter the dashboard ID in the corresponding field (1860 in our case)

dashboard-id

The dashboard should be automatically detected by Grafana.

On the next window, select your Prometheus data source, and click on Import.

import-dashboard-2

You are done! Your dashboard is now up and running.

a – Importing a Prometheus dashboard into Grafana

(If you are not seeing metrics right away, make sure to tweak the time parameter located at the top right corner of the dashboard)

So, where should you go from there?

The possibilities are endless.

You can start by adding more exporters to your Prometheus instance.

The full list of Prometheus exporters is available here.

Also, if you are looking for examples for MongoDB and MySQL, I wrote some tutorials about those specific exporters.

If you are looking to continue with the Node exporter, there is a complete guide about it in the monitoring section.

I hope that you learned something new today.

Until then, have fun, as always.

Leave a Reply

Your email address will not be published. Required fields are marked *