Category: Guide

If you are a web application developer or a database administrator, your infrastructure likely relies on MongoDB in some ways. Monitoring MongoDB is very important to assure that you are not holding memory issues or performance issues with your database.

In previous years, you can find various ways to monitor your MongoDB. But now, we are going to discuss MongoDB Database Monitoring with Grafana and Prometheus.

Do Check: Complete MySQL dashboard with Grafana & Prometheus

Are you ready? Then check out the concepts available here which you will learn from the following tutorial:

• What your Grafana – Prometheus – MongoDB exporter will look like
• How to install Prometheus, a modern time-series database on your computer;
• How to configure import a MongoDB dashboard in seconds
• How to set up the MongoDB developed by Percona as well as binding it to MongoDB;

Note: Percona’s MongoDB exporter incorporates MongoDB stats for sharding and duplicate, as the development of David Cuadrado’s MongoDB exporter.

• MongoDB, Grafana, and Prometheus Architecture
• Process of Installing & Configuring Prometheus, MongoDB Exporter
• Installing Prometheus
• Installing the MongoDB exporter
• Enabling MongoDB authentication
• Creating a service for the MongoDB exporter
• Configure the MongoDB exporter as a Prometheus target
• Building Awesome MongoDB Dashboards
• Set Prometheus as a Grafana datasource
• Downloading Percona dashboards for Grafana
• Importing the MongoDB dashboard in Grafana
• Common Errors

MongoDB, Grafana, and Prometheus Architecture

Here’s an entire overview of what the final monitoring architecture looks like.

As a quick reminder, Prometheus scrapes targets. Targets may be instrumented applications (like instrumented Java apps for example), the Pushgateway, or exporters.

Exporters are a way to bind to an existing entity (a database, a reverse proxy server, an application server) to expose metrics to Prometheus.

The MongoDB exporter is one of them.

Prometheus will bind to the MongoDB exporters and store related metrics in its own internal storage system.

From there, Grafana will bind to Prometheus and display metrics on dashboard panels.

Easy, isn’t it?

At last, you have a great understanding of what we are trying to build, let’s install the different tools needed to monitor MongoDB.

• Complete MySQL dashboard with Grafana & Prometheus | MySQL Database Monitoring using Grafana and Prometheus
• Windows Server Monitoring using Prometheus and WMI Exporter | How to Install WMI Exporter in Windows?
• How to Setup Grafana and Prometheus on Linux

Process of Installing & Configuring Prometheus, MongoDB Exporter

Here, we come to the main topic that how to install, configure, set up the tools, and monitor the Mongodb easily:

Installing Prometheus

If you are still a beginner using Prometheus, you will find the complete Prometheus installation on this tutorial.

If you run the Prometheus installation entirely, you know have your Prometheus up and ready.

To verify it, head over to http://localhost:9090. If you have a web interface close to the one presented just below, it means that your installation went correctly.

No metrics are currently stored, except maybe Prometheus internal metrics.

Run a simple Prometheus query, such as prometheus_http_request_duration_seconds_sum, and make sure that you have some results.

Now that your Prometheus server is running, let’s install the MongoDB exporter to start monitor our MongoDB database.

Installing the MongoDB exporter

As explained before, the MongoDB exporter is available on Percona’s GitHub here.

The MongoDB exporter comes as a binary file in an archive, but as always, we are going to configure it as a service.

We are also going to configure it to run on a specific authenticated user dedicated to monitoring.

First, download the MongoDB exporter release file from one of the versions available here.

$ mkdir mongodb-exporter
$ cd mongodb-exporter
$ wget https://github.com/percona/mongodb_exporter/releases/download/v0.7.1/mongodb_exporter-0.7.1.linux-amd64.tar.gz

Note: as of June 2019, the MongoDB exporter version is 0.7.1.

Next, extract the downloaded archive in your current folder.

$ tar xvzf mongodb_exporter-0.7.1.linux-amd64.tar.gz

You should now have 4 files: mongodb_exporter, LICENSE, README.md, CHANGELOG.md.

All files are pretty self-explanatory, but we are only interested in the mongodb_exporter binary file here.

As we are going to configure the exporter as a service, create a dedicated user for Prometheus if not already existing.

$ sudo useradd -rs /bin/false prometheus
$ sudo mv mongodb_exporter /usr/local/bin/

Enabling MongoDB authentication

Every access to your MongoDB instance should be authenticated and authorized.

To ensure it, we are going to set up a basic MongoDB authentication for the MongoDB exporter.

MongoDB authentication is set using the –auth flag in the Mongo shell.

By default, mongod does not set this flag, so you should be able to connect to it via localhost.

$ ps aux | grep mongod
mongodb  13468  1.1  6.9 1490632 281728 ? Ssl  Jan05 2838:27 /usr/bin/mongod --unixSocketPrefix=/run/mongodb --config /etc/mongodb.conf

Connect to your MongoDB instance with mongo.

$ mongo --port 27017

Create an administrator account for your exporter with the cluster monitor role.

use admin
db.createUser(
  {
    user: "mongodb_exporter",
    pwd: "password",
    roles: [
        { role: "clusterMonitor", db: "admin" },
        { role: "read", db: "local" }
    ]
  }
)

You should see the following message

Successfully added user: {                        
        "user" : "mongodb_exporter",              
        "roles" : [                               
                {                                 
                        "role" : "clusterMonitor",
                        "db" : "admin"            
                },                                
                {                                 
                        "role" : "read",          
                        "db" : "local"            
                }                                 
        ]                                         
}

Before exiting, shut down your MongoDB instance, and restart it.

$ db.adminCommand( { shutdown: 1 } )
$ exit
$ sudo mongod --auth --port 27017 --config /etc/mongodb.conf &

Set your MongoDB URI environment variable, according to the changes that you made before.

$ export MONGODB_URI=mongodb://mongodb_exporter:password@localhost:27017

Creating a service for the MongoDB exporter

Similar to the MySQLd exporter, we are going to set up the MongoDB exporter as a service.

As usual, head over to /lib/systemd/system and create a new service file for the exporter.

$ cd /lib/systemd/system/
$ sudo touch mongodb_exporter.service

Parse the following configuration into your service file:

[Unit]
Description=MongoDB Exporter
User=prometheus

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/mongodb_exporter

[Install]
WantedBy=multi-user.target

From there, don’t forget to restart your system daemon and start your service.

$ sudo systemctl daemon-reload
$ sudo systemctl start mongodb_exporter.service

You should always verify that your service is working.

As a quick reminder, Percona’s MongoDB exporter runs on port 9216.

To ensure that everything is working correctly, run a simple curl command on port 9216.

$ sudo curl http://localhost:9216/metrics
# HELP go_gc_duration_seconds A summary of the GC invocation durations.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0
go_gc_duration_seconds{quantile="0.25"} 0
go_gc_duration_seconds{quantile="0.5"} 0
go_gc_duration_seconds{quantile="0.75"} 0
go_gc_duration_seconds{quantile="1"} 0
go_gc_duration_seconds_sum 0
go_gc_duration_seconds_count 0
...

Can you already see some Prometheus metrics that are being aggregated already?

Great! Your MongoDB exporter is working!

We only need to bind it to Prometheus, and we should be all set.

Configure the MongoDB exporter as a Prometheus target

Almost there!

As described in the schema shown in the architecture section, we are going to bind Prometheus to the new MongoDB exporter.

Head over to the location of your Prometheus configuration file (mine is located at /etc/prometheus/prometheus.yml) and edit it to add the MongoDB exporter as a target.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    static_configs:
            - targets: ['localhost:9090', 'localhost:9216']

Restart Prometheus, and go to http://localhost:9090/targets to verify that Prometheus is now bound to your newly added exporter.

Great! Everything is up and running now.

The last step will be to create a complete Grafana dashboard to have some insights on our metrics.

Looking for a tutorial to install Grafana? We got it covered in our last article.

Building Awesome MongoDB Dashboards

As explained before, we are going to use dashboards built by Percona to monitor our MongoDB example on Grafana.

Percona provides multiple existing dashboards such as:

• MongoDB Overview;
• MongoDB ReplSet;
• MongoDB RocksDB;
• MongoDB WiredTiger;
• MongoDB MMAPv1
• MongoDB InMemory

For this part, we are going to focus on importing the MongoDB Overview dashboards into our Grafana instance.

Set Prometheus as a Grafana datasource

If you followed our previous tutorials, you are probably a master at it right now.

If that’s not the case, here’s the configuration for Prometheus on Grafana.

Downloading Percona dashboards for Grafana

In Grafana, dashboards come as JSON files. When you create a dashboard, you can export it in this format and share it with the world.

Percona provides dozens of dashboards on its Github repository.

In this case, we are going to download the MongoDB Overview dashboard file.

Run a simple wget command to get the JSON file.

You can create a dedicated dashboards folder in your /etc/grafana folder to store your downloaded dashboards.

$ cd /etc/grafana
$ sudo mkdir dashboards
$ cd dashboards
$ sudo wget https://github.com/percona/grafana-dashboards/blob/master/dashboards/MongoDB_Overview.json

If you want all the dashboards available in the repository, simply clone the repository into your current folder.

$ sudo git clone https://github.com/percona/grafana-dashboards.git

Now that you have the JSON files available, it is time for us to import them into Grafana.

Importing the MongoDB dashboard in Grafana

For this example, we are going to import the MongoDB Overview dashboard for Grafana.

Head over to your Grafana instance, located at http://localhost:3000 (if you didn’t change any default ports in your configuration file)

On the left menu, hover the ‘Plus‘ icon with your mouse and click on Import.

From there, you should be taken to the Import page. Click on the Upload .json file option.

Given the operating system, you are working with, navigate to the /etc/grafana folder (where you stored your dashboard), and click on the MongoDB_Overview.json file.

Your dashboard should be imported automatically, with real-time updates of your MongoDB database!

Common Errors

If you carefully followed this tutorial, chances are that you have a fully functional dashboard right now.

However, you might encounter some errors during the process.

Here are some clues on how to solve them:

• Failed to get server status: not authorized on admin to execute the command

This error message is fairly simple to understand.

Your mongodb_exporter user does not have the necessary credentials to perform queries on the admin database.

Clue 1

To resolve it, connect to your instance, use the admin database, and make sure that you configured correctly the mongodb_exporter user (it must have the cluster monitor right on the admin database)

$ mongo --port 27017 (or --auth if you already have an admin account on your database)
$ use admin;
$ db.getUsers()
{
        "_id" : "admin.mongodb_exporter",         
        "user" : "mongodb_exporter",              
        "db" : "admin",                           
        "roles" : [                               
                {                                 
                        "role" : "clusterMonitor",
                        "db" : "admin"            
                },                                
                {                                 
                        "role" : "read",          
                        "db" : "local"            
                }                                 
        ]                                         
}

Clue 2

You didn’t properly set the MongoDB URI environment variable.

To verify it, launch the following command:

$ env  | grep mongodb
MONGODB_URI=mongodb://mongodb_exporter:password@localhost:27017

Clue 3

If this is still not working, set the MongoDB URI directly in the service file, restart your service, as well as your MongoDB instance.

[Service]
Type=simple
Restart=always
ExecStart=/usr/local/bin/mongodb_exporter --mongodb.uri=mongodb://mongodb_exporter:password@localhost:27017

$ sudo systemctl daemon-reload
$ sudo systemctl restart mongodb.service
$ sudo systemctl restart mongodb_exporter.service

• Could not get MongoDB BuildInfo: no reachable servers!

Clue 1

Your MongoDB database is either not launched, not it is not running on port 27017.

For the first option, just verify that your MongoDB service is running, or that your mongod background is running.

$ sudo systemctl status mongodb.service
or
$ ps aux | grep mongod

For the second option, verify the port used by your MongoDB instance. To do so, run the following command:

$ sudo lsof -i -P -n | grep LISTEN
grafana-s   642         grafana    7u  IPv6  998256601      0t0  TCP *:3333 (LISTEN)
mysqld_ex  3136            root    3u  IPv6  998030854      0t0  TCP *:9104 (LISTEN)
mongod     3688            root   10u  IPv4 1092070442      0t0  TCP 127.0.0.1:27017 (LISTEN)

If your MongoDB instance is not running on the default 27017 port, change your mongodb_exporter file for it to bind to your custom port.

[Service]
mongodb.uri=mongodb://mongodb_exporter:password@localhost:12345

Going Further

Now that you have a fully operational monitoring pipeline for MongoDB, it is time for you to dig a little deeper.

Here are the best resources if you want to know more about MongoDB monitoring.

First, here’s a great talk by Vadim Tkachenko from Percona about Monitoring MySQL and MongoDB instances. You will understand how Percona builds its own monitoring architecture and its own dashboards.

This is a more general talk about MongoDB monitoring using the built-in commands available in your MongoDB CLI such as the server status, the stats, or the total size of each of your collections.

A great talk if you are not into custom monitoring solutions, and if you want to focus on native and already implemented functions.

In the present world, technologies are updating day by day with plenty of advanced facilities. Likewise, the installation of InfluxDB on windows tutorials also may differ some time to time. So, here we have come up with a useful and handy tutorial on How To Install InfluxDB on Windows which is good and up-to-date among others.

These types of technologies may change and grow all the time so educational resources should adapt properly. The main objective of offering this tutorial is to have an insightful and up-to-date article on how to install it on Windows.

The tutorial of Install InfluxDB on Windows in 2021 covers the following stuff in a detailed way:

• How do you install InfluxDB on Windows in 2021?
• How to download InfluxDB on Windows
• How to configure InfluxDB on your machine
• How to Run InfluxDB as a Windows service using NSSM Tool?
• Most Common Mistakes In The Process

Be ready to follow all the required steps for a clean InfluxDB installation.

How do you install InfluxDB on Windows in 2021?

Check out this video tutorial on how to install InfluxDB on windows and moreover, you can learn the basic commands of influxdb, and integration with Grafana from here:

How to download InfluxDB on Windows?

By following the below two methods, you can easily download the InfluxDB on windows:

a – Downloading the archive

Downloading InfluxDB is very straightforward.

Head over to the InfluxDB downloads page. There, you will see the following four boxes.

What are those four boxes for?

They are part of the TICK stack. (Telegraf, InfluxDB, Chronograf, and Kapacitor).

Each of these tools has a very specific role: gathering metrics, storing data, visualizing time series or having post-processing defined functions on your data.

In this tutorial, we are going to focus on InfluxDB (the time series database component of TICK)

• How To Install InfluxDB 1.7 and 2.0 on Linux in 2021
• How To Install InfluxDB Telegraf and Grafana on Docker
• How To Create a Database on InfluxDB 1.7 & 2.0

So should you download the v1.7.6 or v2.0.0 version?

In my previous articles, I answered the main difference between the two versions, but here’s the main difference you need to remember.

As the 2.0 version is still experimental, we are going to go for the 1.7.6 version.

Click on the v1.7.6 button.

Another window will open with all operating systems. Scroll until you see Windows Binaries (64-bit).

Simply click on the URL in the white box, and the download will automatically start in your browser.

Store it wherever you want, in my case, it will be in the Program Files folder.

Unzip the archive using your favorite archive utility tool (7-Zip in my case) or run the following command in a Powershell command line.

Expand-Archive -Force C:\path\to\archive.zip C:\where\to\extract\to

Great! Let’s take a look at what you have here.

b – Inspecting the archive

Inside your folder, you now have 5 binaries and 1 configuration file:

• influx.exe: a CLI used to execute IFQL commands and navigate into your databases.
• influx_inspect.exe: get some information about InfluxDB shards (in a multinode environment)
• influx_stress.exe: used to stress-test your InfluxDB database
• influx_tsm.exe: InfluxDB time-structured merge tree utility (not relevant here)
• influxd.exe: used to launch your InfluxDB server
• influxdb.conf: used to configure your InfluxDB instance.

Relevant binaries were marked in bold.

Also Check: How To Create a Grafana Dashboard? (UI + API methods)

How to configure InfluxDB Server on your Machine

Before continuing, you have to configure your InfluxDB instance for Windows.

We are essentially interested in four sections in the configuration file.

a – Meta section

This is where your raft database will be stored. It stores metadata about your InfluxDB instance.

Create a meta folder in your InfluxDB directory (remember in my case it was Program Files).

Modify the following section in the configuration file.

[meta]
  # Where the metadata/raft database is stored
  dir = "C:\\Program Files\\InfluxDB\\meta"

b – Data section

InfluxDB stores TSM and WAL files as part of its internal storage. This is where your data is going to be stored on your computer. Create a data and a wal folder in your folder. Again, modify the configuration file accordingly.

[data]
  # The directory where the TSM storage engine stores TSM files.
  dir = "C:\\Program Files\\InfluxDB\\data"

  # The directory where the TSM storage engine stores WAL files.
  wal-dir = "C:\\Program Files\\InfluxDB\\wal

Important: you need to put double quotes in the path!

c – HTTP section

There are many ways to insert data into an InfluxDB database.

You can use client libraries to use in your Python, Java, or Javascript applications. Or you can use the HTTP endpoint directly.

InfluxDB exposes an endpoint that one can use to interact with the database. It is on port 8086. (here’s the full reference of the HTTP API)

Back to your configuration file. Configure the HTTP section as follows:

[http]
  # Determines whether HTTP endpoint is enabled.
  enabled = true

  # The bind address used by the HTTP service.
  bind-address = ":8086"

  # Determines whether HTTP request logging is enabled.
  log-enabled = true

Feel free to change the port as long as it is not interfering with ports already used on your Windows machine or server.

d – Logging section

The logging section is used to determine which levels of the log will be stored for your InfluxDB server. The parameter by default is “info”, but feel free to change it if you want to be notified only for “error” messages for example.

[logging]
  # Determines which log encoder to use for logs. Available options
  # are auto, logfmt, and json. auto will use a more user-friendly
  # output format if the output terminal is a TTY, but the format is not as
  # easily machine-readable. When the output is a non-TTY, auto will use
  # logfmt.
  # format = "auto"

  # Determines which level of logs will be emitted. The available levels
  # are error, warn, info, and debug. Logs that are equal to or above the
  # specified level will be emitted.
  level = "error"

e – Quick test

Before configuring InfluxDB as a service, let’s run a quick-dry test to see if everything is okay.

In a command-line, execute the influxd executable. Accept the firewall permission if you are prompted to do it.

Now that your InfluxDB server has started, start a new command-line utility and run the following command.

C:\Users\Antoine>curl -sl -I http://localhost:8086/ping
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 7dacef6d-8c2f-11e9-8018-d8cb8aa356bb
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.6
X-Request-Id: 7dacef6d-8c2f-11e9-8018-d8cb8aa356bb
Date: Tue, 11 Jun 2021 09:58:41 GMT

The /ping endpoint is used to check if your server is running or not.

Are you getting a 204 No Content HTTP response?

Congratulations!

You now simply have to run it as a service, and you will be all done.

How to Run InfluxDB as a Windows service using NSSM Tool?

As you guessed, you are not going to run InfluxDB via the command line every time you want to run it. That’s not very practical.

You are going to run it as a service, using the very popular NSSM tool on Windows.

You could use the SC tool that is natively available on Windows, but I just find it more complicated than NSSM.

To download NSSM, head over to https://nssm.cc/download.

Extract it in the folder that you want, for me, it will be “C:\Program Files\NSSM”.

From there, in the current NSSM folder, run the following command (you need administrative rights to do it)

> nssm install

You will be prompted with the NSSM window.

Enter the following details in it (don’t forget the config section, otherwise our previous work is useless)

That’s it!

Now your service is installed.

Head over to the services in Windows 10. Find your service under the name and verify that its status is “Running” (if you specified an automatic startup type in NSSM of course)

Is it running? Let’s verify one more time with curl.

C:\Users\Antoine>curl -sL -I http://localhost:8086/ping
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: ef473e13-8c38-11e9-8005-d8cb8aa356bb
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.6
X-Request-Id: ef473e13-8c38-11e9-8005-d8cb8aa356bb
Date: Tue, 11 Jun 2021 11:06:17 GMT

Congratulations! You did it!

You installed InfluxDB on Windows as a service, and it is running on port 8086.

Most Common Mistakes In The Process

The service did not respond in a timely fashion.

I encountered this error when I tried to set up InfluxDB as a service using SC. As many solutions exist on Google and on Youtube, I solved it by using NSSM.

Tried tweaking the Windows registry but it wasn’t very useful at all.

Only one usage of each socket address (protocol/network address/port) is normally permitted.

Simple, there is already a program or service listening on 8086. You should modify the default port in the configuration file and take one that is permitted and not used.

I don’t have the same curl response

A 204 response to the curl command is the only sign that your InfluxDB is running correctly. If you don’t get the same output, you should go back and double-check the steps before.

I have a parsing error in my configuration file!

Remember that in Windows systems backslashs have to be escaped. It’s double backslashs in the paths of your InfluxDB configuration file.

If your path contains some spaces, like “Program Files”, make sure to put your path into quotes.

One of the important components of any software or Operating system is Logs. In case, you deal mostly with a Linux system administration, then apparently you spend a lot of time browsing your log files to obtain appropriate data regarding prior events. Generally, it records users’ actions, network activity, system events, and many more based on what they are intended for.

Rsyslog is one of the most extensively used logging systems on Linux OS. It is a very powerful, secure, and high-performance log processing system that accepts information from various types of sources and outputs it into different formats.

In this definitive guide, we are going to explain all How to Centralized Logging with Syslog on Linux Systems. Not just in a brief way, we will be discussing it even more broadly with every single step that you need to put in place to build a reliable, secure, and functional centralized logging system.

Are you ready to experience it practically by referring to this useful tutorial? Go for it:

• What Is Centralized Logging?
• What Is syslog‐ng?
• What You Will Learn?
• How does Linux logging works on a single instance?
• a – General concepts of Linux logging
• b – Where are logs stored on a Linux filesystem?
• Designing a centralized logging architecture on Linux
• Configure rsyslog to forward logs to a centralized server
• a – Configure your rsyslog server
• b – Configure your rsyslog client
• Encrypting rsyslog messages with TLS
• a – Configuring your certificate authority
• b – Generating private/public keys for the server
• c – Generate private/public keys for the client
• d – Send generated keys to your hosts
• e – Configure your rsyslog server
• f – Configure your rsyslog client
• Sending log messages reliably with action queues
• a – Designing message reliability
• b – Demonstrating message reliability
• Common Errors
• Bonus: A word on back-pressure protocols
• a – Architectural improvements
• b – Push vs Pull logging systems
• Conclusion

What is Centralized Logging?

Centralized logging lets you store your Linux, UNIX, and Windows logs in a centralized repository. It also provides several advantages such as:

• Single location to check for system errors (ever had a disk die that disrupted local logging?)
• Security, especially when you need to put together timelines after a system compromise
• Often required for security compliance

What is Syslog‐ng?

Syslog‐ng is a flexible and robust open-source syslog implementation. Syslog-ng offers various advantages for users like as mentioned below:

• Logging via UDP or TCP
• Mutual authentication through digital certificates
• Messages can be parsed and rewritten ( this is especially useful for removing sensitive data from log messages)
• Encryption of log traffic via TLS
• Logs can be sent to a SQL database
• Filters can be used to sort traffic based on host, facility, log level, message contents, etc.

Do Check: Syslog: The Complete System Administrator Guide

What You Will Learn?

As always, let’s start by having a look at everything that you are going to learn if you follow the tutorial until the end.

With this tutorial, you are going to learn:

• How Linux logging works on a single instance, where to look after logs;
• What a centralized logging architecture looks like advantages and drawbacks of using it;
• How to setup rsyslog forwarding from a client to a centralized server;
• How to secure log forwarding using the TLS protocol;
• How to send logs using reliable mechanisms such as action queues.

That’s a long program, but feel free to skip some sections if you are already comfortable with some of the concepts explained in this article.

How does Linux logging works on a single instance?

Let’s start with a bit of history about Linux logging.

• Syslog: The Complete System Administrator Guide
• Linux Logging Complete Guide
• How To Install Logstash on Ubuntu 18.04 and Debian 9 | Tutorial on Logstash Configuration

a – General concepts of Linux logging

Before jumping into building our centralized logging architecture, there are some concepts about logging on a single instance that are mandatory to understand more complex concepts.

By default, your Linux operating system records logs about many events that are happening on your machine.

Linux uses the syslog protocol which defines a standard for every aspect of logging on an operating system (not only Linux but also Windows): defining what a message looks like, describing severity levels on messages, as well as listing the ports that syslog will be using.

Syslog can be used as a server (hosting the logs) or as a client (forwarding the logs to a remote server).

As a consequence, the syslog protocol also defines how to log transmission should be done, both in a reliable and secure way.

If you are using a modern Linux distribution (like a Ubuntu machine, a CentOS, or an RHEL one), the default syslog server used is rsyslog.

Rsyslog comes as an evolution of syslog, providing capabilities such as configurable modules that can be bound to a wide variety of targets (forwarding Apache logs to a remote server for example).

Rsyslog also provides native filtering as well as templates to format data to a custom format.

b – Where are logs stored on a Linux filesystem?

Long story short, logs are stored at /var/log/ on your filesystem.

At this location, you should see multiple log files, each one having a name describing what they actually store.

For a quick overview, you can execute the following command:

$ ls -l /var/log
-rw-r-----  1 syslog        adm               120999 Jul 24 18:04 auth.log
-rw-r--r--  1 root          root              127503 Jul 20 06:35 dpkg.log
-rw-r-----  1 syslog        adm                    0 Jul 15 06:25 kern.log
drwxrwxr-x  2 logstash      root                4096 Jul  8 18:33 logstash
drwxr-xr-x  2 root          root                4096 Sep 10  2018 lxd
drwxr-xr-x  2 mongodb       mongodb             4096 Jul  8 06:25 mongodb

As you can see, you have dedicated log files for authentication purposes or for kernel related logs.

/var/log does not contain only files, but it also contain dedicated folders that vendors create when the application is installed.

As you can see, I am running a MongoDB database on my instance.

As a consequence, I have a MongoDB folder in my log folder.

But why am I telling you all of this?

Because using this knowledge, we are going to lay the first brick of your centralized logging architecture.

Suppose that three machines are sending logs to my server, each machine is going to have its own own auth.log, kern.log or dpkg.log files.

As a consequence, you want logs to be stored in dedicated folders, one for each instance.

Here’s the folder architecture on the server side.

Designing a centralized logging architecture on Linux

Now that we have the basics of Linux logging, we are ready to design a centralized logging architecture.

As described in the first section, every machine in our pool is already writing logs via rsyslog.

However, natively, our machines are configured as client-server syslog instances.

They create logs, but they store them locally in the file system.

On our centralized logging architecture, client machines will be configured to use rsyslog as a client, and they will forward every single log to a remote rsyslog server, which is the central server.

To be complete, our architecture has to:

• Be secure : that’s why we are going to implement TLS in the fifth section;
• Be reliable : if the entire network goes down for a minute, are all logs for this period lost forever? Not with action queues.

What are the advantages of such an architecture?

• You can inspect all your logs from a single place : you don’t rely on connecting individually to every machine on your pool, you can directly see the logs from the same filesystem.
• You have access to logs, even if the host is not accessible : if one of the machines is having issues and it cannot be reached, you would not be able to go through the filesystem to look after logs. With a centralized logging server, you have the opportunity to look at logs, without any physical access to the client machine.
• You can set up a complete log monitoring infrastructure : if you are interested in visualizing your logs with tools such as Kibana, I wrote a complete guide about it.

On the other hand, what would be for us the drawbacks of such an architecture?

• You risk overloading your syslog server : with this architecture, you are pushing logs to a remote server. As a consequence, if one machine goes crazy and starts sending thousands of logs, you risk overloading the log server. (if you want a reliable back-pressure control, you can check Filebeat by Elastic)
• If your log server goes down, you lose the ability to look at all the logs sent by the clients. Moreover, if the server goes down, clients will start storing messages locally until the server is available again, thus filling up disk space on the client side.

Configure rsyslog to forward logs to a centralized server

Prerequisites

First, make sure that you have rsyslog on your instance.

$ sudo systemctl status rsyslog

If the service is unknown on your computer, you can install it by running:

$ sudo apt-get update && apt-get install rsyslog
$ sudo systemctl enable rsyslog
$ sudo systemctl start rsyslog

a – Configure your rsyslog server

First of all, you have to configure your rsyslog server for it to accept incoming logs on port 514.

We are going to use TCP for log transmission, but you can use UDP if you don’t care that much about reliability.

On the server, head to /etc/rsyslog.d folder.

This is the directory that stores templates as well as files that contain the rsyslog rules.

In this directory, there should be a 50-default.conf file.

We are going to create our own configuration file, prefixing it with a number lower than the configuration file one.

This way, our configuration file takes priority over the default one.

$ sudo touch 01-server.conf

# Listen for TCP
$ModLoad imtcp
# Listen on port 514
$InputTCPServerRun 514

$template RemoteServer, "/var/log/%HOSTNAME%/%SYSLOGFACILITY-TEXT%.log"
*.* ?RemoteServer

Our first rule!

With this syntax, our log files will be grouped by hostname (aka the computer name sending the log) and then by syslog facility (kern, user, auth, etc..)

Restart your rsyslog server, and make sure that it is now listening on port 514 for TCP

$ sudo systemctl restart rsyslog
$ netstat -tna | grep :514
tcp        0      0 0.0.0.0:514             0.0.0.0:*               LISTEN
tcp6       0      0 :::514                  :::*                    LISTEN<

Note:  With this configuration file, your server logs are now no longer directly stored in /var/log, but in /var/log/hostname, hostname is the current name of your host (in my case /var/log/schkn-ubuntu)

$ uname -a
Linux schkn-ubuntu 4.15.0-42-generic #45-Ubuntu SMP Thu Nov 15 19:32:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

b – Configure your rsyslog client

It is time to configure rsyslog as a client on our instance.

Now that we have rsyslog, head over to /etc/rsyslog.d.

Similar to the procedure done on the server-side create a 01-client.conf file.

$ sudo vi 01-client.conf

*.*                         @@distant-server-ip:514

The configuration on the client-side is pretty simple, plain simple forwarding.

Restart your rsyslog server, and make sure that:

• You have no errors on the client-side

$ sudo systemctl restart rsyslog
$ journalctl -f -u rsyslog

• Your server now has the logs from the client

My client machine is named “antoine-Virtualbox” in my current setup, so I am looking for /var/log/antoine-Virtualbox.

Awesome! Distant logs are now correctly stored on our server.

However, I have some bad news for you.

As we are using TCP for log transmission, it means that messages are not encrypted between the client and the server.

As a consequence, any hacker that sniffs the packets on your network will be able to see your logs, spy on their content, and perform attacks on your host machines.

This is what we call a man-in-the-middle attack.

As proof, here’s a Wireshark instance sniffing my network and listening for a very specific IP.

As you can see, the message is clearly visible and states: “Hi, this is a test message”.

Now imagine if the log message contained sensitive information about your host, that could be a complete security disaster.

This is why we are going to setup TLS, allowing us to encrypt the traffic between the client and the server.

At the end of the next chapter, I will provide you with a Wireshark screenshot, proving that the traffic is now encrypted.

Ready?

Encrypting rsyslog messages with TLS

In order for us to encrypt messages between the server and the client, we are going to use TLS and more precisely trusted certificates.

Without going into too many details, those trusted certificates will ensure that both client and server identities are verified.

With this protocol, the client and the server will first ensure that they are correctly talking to each other (as there is no one along the way with a fake identity), and then they will proceed to encrypt their messages.

As the client and the server exchanged their keys, they are able to decrypt messages on the fly.

In our setup, we will have a certificate authority signing the keys.

Communication between the hosts and the certificate authority is obviously encrypted too, so we need first to create a key pair for the certificate authority.

If you are just testing out, you can use your rsyslog server as a certificate authority.

a – Configuring your certificate authority

On the server, head over to /etc/ssl, and create a rsyslog directory.

$ sudo mkdir /etc/ssl/rsyslog
$ cd /etc/ssl/rsyslog

Install the gnutls-utils package (that might come as the gnutls-bin for some distributions) that enables SSL API on your server.

$ sudo apt-get install gnutls-utils
(or)
$ sudo apt-get install gnutls-bin

Generate a private key for the certificate authority:

$ sudo certtool --generate-privkey --outfile CA-key.pem
$ sudo chmod 400 CA-key.pem

Generate a public key on the certificate authority:

$ sudo certtool --generate-self-signed --load-privkey CA-key.pem --outfile CA.pem
# Common name: authority.devconnected.com
  # The certificate will expire in (days): 3650
  # Does the certificate belong to an authority? (Y/N): y
  # Will the certificate be used to sign other certificates? (Y/N): y
  # Will the certificate be used to sign CRLs? (y/N): y

You can leave the other questions blank as they will have no impact on the final result.

b – Generating private/public keys for the server

Now that our certificate authority has keys to sign other keys, let’s create one for the rsyslog server.

First, create a private key for your server.

$ sudo certtool --generate-privkey --outfile server-key.pem --bits 2048

Generate a certificate request for your server.

$ sudo certtool --generate-request --load-privkey server-key.pem --outfile server-request.pem

Generate the certificate for your rsyslog server and import trusted certificate authority keys into it.

$ sudo certtool --generate-certificate --load-request server-request.pem --outfile server-cert.pem --load-ca-certificate CA.pem --load-ca-privkey CA-key.pem

  # The certificate will expire in (days): 3650
  # Is this a TLS web client certificate? (Y/N): y
  # Is this also a TLS web server certificate? (y/N): y
  # Enter a dnsName of the subject of the certificate: server.devconnected.com

c – Generate private/public keys for the client

Similar to what we have done before, let’s build our key pair for the client.

Create a private key for the client.

$ sudo certtool --generate-privkey --outfile client-key.pem --bits 2048

Generate a certificate request for the client.

$ sudo certtool --generate-request --load-privkey client-key.pem --outfile client-request.pem

Create a certificate (a public key) for your client and import the certificate authority certificates inside.

$ sudo certtool --generate-certificate --load-request client-request.pem --outfile client-cert.pem --load-ca-certificate CA.pem --load-ca-privkey CA-key.pem

# The certificate will expire in (days): 3650
  # Is this a TLS web client certificate? (Y/N): y
  # Is this also a TLS web server certificate? (y/N): y
  # Enter a dnsName of the subject of the certificate: client.devconnected.com

Awesome! We now have key pairs for both our client and server.

d – Send generated keys to your hosts

Now that your keys are generated on the certificate authority, send your keys securely to the destination hosts.

For this, you can use scp.

For the server (simply switch to ‘client’ when you want to send keys to the client)

$ sudo -u root scp -i ~/.ssh/id_rsa CA.pem server-* root@server_ip_here:/etc/ssl/rsyslog/

e – Configure your rsyslog server

Now that both servers and clients contain keys for encryption, let’s configure hosts to conclude this chapter.

First, install the gtls driver on your server.

$ sudo apt-get install rsyslog-gnutls

As a reminder, we create a 01-server.conf file in the previous chapter.

Replace the content with the following configuration.

# Listen for TCP
$ModLoad imtcp
# GTLS driver
$DefaultNetstreamDriver gtls
# Certificates
$DefaultNetstreamDriverCAFile /etc/ssl/rsyslog/CA.pem
$DefaultNetstreamDriverCertFile /etc/ssl/rsyslog/server-cert.pem
$DefaultNetstreamDriverKeyFile /etc/ssl/rsyslog/server-key.pem
# Authentication mode
$InputTCPServerStreamDriverAuthMode x509/name
$InputTCPServerStreamDriverPermittedPeer *.devconnected.com
# Only use TLS
$InputTCPServerStreamDriverMode 1
# Listen on port 514
$InputTCPServerRun 514

$template RemoteServer, "/var/log/%HOSTNAME%/%SYSLOGFACILITY-TEXT%.log"
*.* ?RemoteServer

As you can see, we are adding an AuthMode, which is x509/name in this case.

This means that we are going to use a certificate that contains a “name” inside it.

Remember when we declare that we were “client.devconnected.com” or “server.devconnected.com” when we created the certificates?

This is exactly the information that the authentication is going to use, and you will get an error if the name does not match.

Don’t forget to restart rsyslog for the changes to be saved.

$ sudo systemctl restart rsyslog

f – Configure your rsyslog client

On the client, we had a simple forwarding rule in a 01-client.conf file.

Again, install the gtls driver on your client.

$ sudo apt-get install rsyslog-gnutls

Replace the content with the following configuration.

# GTLS driver
$DefaultNetstreamDriver gtls
# Certificates
$DefaultNetstreamDriverCAFile /etc/ssl/rsyslog/CA.pem
$DefaultNetstreamDriverCertFile /etc/ssl/rsyslog/client-cert.pem
$DefaultNetstreamDriverKeyFile /etc/ssl/rsyslog/client-key.pem
# Auth mode
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer server.devconnected.com
# Only use TLS
$ActionSendStreamDriverMode 1
# Forward everything to server.devconnected.com
*.* @@distant-server-ip:514

Restart your rsyslog client.

$ sudo systemctl restart rsyslog

Awesome! Your logs should now be sent to your remote server. Run this command and make sure that you have no errors (if you do, I have a section for it at the end)

$ $ journalctl -u -n 100 rsyslog

A promise is a promise, here’s my Wireshark between my server and my client, with a TLS encryption between them.

No way for you to decrypt the message.

Plot twist : I sent “Hi, this is a test message” again.

Sending log messages reliably with action queues

In the architecture that we designed before, there is one major issue that was not addressed before: reliability.

In our architecture, the client and the server are very tightly coupled.

What does it mean?

If the server goes down, your client will obviously not be able to send log messages, but they might also risk losing the logs messages forever.

Luckily for us, rsyslog has a contingency plan when it comes to bad forwarding: plain and simple disabling the faulty module.

This means that you won’t lose your messages (they still be stored locally), but you are losing your log forwarding altogether.

Would you go back to your host to re-enable your forwarding module every time the network goes down? I wouldn’t.

To solve this, we need to bring decoupling to this architecture.

If you read the Entreprise Integration Patterns book by Gregor Hohpe, reliability means queues.

a – Designing message reliability

First, let’s understand a couple of concepts related to rsyslog message queues.

In rsyslog, there are multiple ways for you to design queues, but eventually, it comes up the three main categories:

• Direct: this is kind of a “no queue” mode where messages are directly passed from the producer to the consumer. Definitely not a mode we want to be implemented in our case;
• Disk queues: in this case, as messages are passed to the queue, they are reliably written to the disk. In this case, you are safer when it comes to machine complete accidental shutdown, but the process is longer than memory queues. Writing to disk is more expensive than writing in the RAM;
• Memory queues: those queues will hold messages in memory for a given time and for a given amount of restricted space. After a while, they will be written to disk. This alternative is faster than disk queues.

For our architecture, we are going to use memory queues with a bunch of different options (all options are available here)

In your 01-client.conf file, append the following lines to your configuration file.

$ActionQueueType LinkedList
$ActionQueueFileName rsyslog_backup
$ActionQueueSaveOnShutdown on
$ActionResumeRetryCount -1
...
# Parameters from the TLS section
$ActionSendStreamDriverAuthMode x509/name

• LinkedList: we want memory allocation to be done on the fly, and not via a fixed array. This will create more overhead computation but you will benefit from it by gaining more space;
• Queue file name: this is the file where data is going to be written if data is eventually stored to disk. The path to it is $WorkDirectory/rsyslog_backup where WorkDirectory is the variable defined in your rsyslog.conf file;
• Save on shutdown: if the machine was to shut down, the memory logs would be written to disk first;
• Resume retry count: when losing the forwarding module, rsyslog will try to reconnect to it indefinitely.

You can tweak it more given the specifications you are working with, but I will go with that for this tutorial.

Restart rsyslog, and make sure that you are not having any errors.

$ sudo systemctl restart rsyslog
$ sudo journalctl -f -u rsyslog.service

b – Demonstrating message reliability

Okay, now that we have message reliability put in place, let’s test it out.

On the server-side, comment all the lines in your 01-server.conf file and restart rsyslog. From this point, your client should lose its connection to the server.

On the client:

$ sudo journalctl -f -u rsyslog.service

On the client, manually fire a log event.

$ logger -p 'auth.crit' 'Critial auth message, lost?'

Verify that you are not receiving anything at all on the server and that the client does not fire any preemptive message.

From there, your message is stored in memory, patiently waiting for the module to be back up.

Go to the server-side, remove comments for all the lines, and restart your rsyslog server.

From there, your client will automatically re-enable the forwarding module.

But, did our server receive the message that was stored in the client’s memory? It did!

Congratulations, you designed (secure!) message reliability between rsyslog clients and a rsyslog server.

Common Errors

Here is a list of the different errors that you may encounter during the process:

• omfwd: TCPSendBuf error -2027, destruct TCP connection the x.x.x.x

Solution: if you are having this error in the TLS section, your TLS configuration is probably wrong. Either you have badly configured the certificates, or you have the wrong server configuration.

• Cannot connect to x.x.x.x : network is unreachable.

Solution: make sure that the IP address can be accessed from the client host. You can ping it by example.

$ ping 142.93.103.142
PING 142.93.103.142 (142.93.103.142) 56(84) bytes of data.
64 bytes from 142.93.103.142: icmp_seq=1 ttl=64 time=0.023 ms
64 bytes from 142.93.103.142: icmp_seq=2 ttl=64 time=0.056 ms
64 bytes from 142.93.103.142: icmp_seq=3 ttl=64 time=0.033 ms
--- 142.93.103.142 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2047ms

• Peer name not authorized: not permitted to talk to it. Names CN: name.domain.com

In the rsyslog configuration file, make sure that you have the $ActionSendStreamDriverPermittedPeers correctly set.

Bonus: A word on back-pressure protocols

Before ending this guide, there is one point that I want to address, as I think that it might be something that could be improved in this architecture.

a – Architectural improvements

The architecture that we designed together can be seen as a “push” architecture. We essentially push logs from the client to the servers.

And that’s risky.

Why?

Because if the clients begin to send too many logs files to your server, you have the risks of overloading your entire system, preventing other clients to send data. From there, it is only a matter of time until your server goes down, or until you start losing logs files completely.

Here are some thoughts about how to handle this:

• Use the dequeueSlowDown rsyslog parameter to delay message execution, essentially putting a tempo on all the messages being sent. Here is a link to it.
• Use a back-pressure third-party product like Filebeat. Filebeat can be interesting if you are already working with the ELK stack all together. It implements a native back-pressure protocol for you to set a rate on when the logs are being sent.

b – Push vs Pull logging systems

We have already tackled the difference between push and pull systems at least when it comes to open source monitoring solutions.

We have seen the advantage of building an active monitoring solution, but couldn’t we apply the same principles when it comes to centralized logging systems?

As our log files are stored in disks, there is definitely a way for us to parse those files regularly and extract logs, at a rate that we decide on the server-side. This point removes the danger of having a client overloading our servers.

Are you aware of a system that is designed this way?

Conclusion

This last point ends this definitive guide about centralized logging for Linux systems.

In this guide, we went through pretty much all the aspects of centralized logging: from one single instance, to log forwarding to a central server in a secure and reliable way.

If you really want to go further on the subject, here are some videos that are related to the subjects, and that I found interesting.

Are you worried about monitoring your whole Linux system performance? Do you want to monitor your filesystems, disks, CPUs, also network statistics, likewise you would do with netstat? Also, you are looking for a complete dashboard that displays every single metric on your system efficiently.

This tutorial is the right choice for you all, where it takes a special glance at the Node Exporter, a Prometheus exporter trained in exposing Linux metrics right off the bat. In case, you are a beginner to Prometheus then we would suggest you go through this guide completely and grasp the fundamentals of the awesome time-series database.

After knowing the basics about Prometheus and before starting a build of our entire Linux monitoring system, you all have to know the concepts that will be learning today via this Complete Node Exporter Mastery with Prometheus Tutorial. 

• What You Will Learn
• Basics of Linux Monitoring
  • Existing Solutions
  • Node Exporter & Prometheus
• Installation of Required Tools
  • Installing the Node Exporter
  • Installing Prometheus
  • Installing Grafana
• Building a complete Grafana Dashboard for the Node Exporter
• Getting Inspiration for Dashboards
• Going Further
  • a – Additional Modules
  • b – TextFile Collector
  • c – Videos Resources

What You Will Learn

• Existing ways to monitor your Linux system: You will discover about the free and paid tools that you can use in order to quickly monitor your infrastructure
• What is the Node Exporter and how to properly install it as a service
• Bind your Node Exporter with Prometheus and start gathering system metrics
• Play with prebuilt Grafana dashboards to build 100+ panels in one click

After coming to the edge of this tutorial, you will be able to develop your own monitoring infrastructure and add many more exporters to it.

Basics of Linux Monitoring

Before building our entire monitoring architecture, let’s have a look at what are the currently existing solutions and what problems we are trying to solve with the Node Exporter.

Also Check: Complete MySQL dashboard with Grafana & Prometheus

Existing Solutions

As a system administrator, there are multiple ways for you to monitor your Linux infrastructure.

• Command-line tools

There are plenty of command-line tools for you to monitor your system.

• Monitoring Disk I/O on Linux with the Node Exporter
• How To Install Prometheus with Docker on Ubuntu 18.04
• How to Setup Grafana and Prometheus on Linux

They are very known to every system administrator and are often very useful to perform some simple troubleshooting on your instance.

Some examples of command-line tools may be top or htop for the CPU usage, df or du for the disks, or even tcpdump for an easy network traffic analysis.

Those solutions are great but they have major downsides.

Besides being very easy to use, they are often formatted in different ways, making it hard to export them in a consistent way.

Also, to run those commands, you sometimes need elevated privileges on the system which is not always the case.

With a complete monitoring system, you can have the security rules handled directly in your dashboarding system (for example Grafana) and you don’t need to provide direct access to your instance to whoever wants to troubleshoot outages.

• Desktop solutions

Desktop solutions provide a more consistent and probably more practical solution to system monitoring.

Some examples of those tools are the very established SolarWinds Linux Performance Monitoring Tool (that provides very complete dashboards for your Linux system) or Zabbix with the Metric Collection product.

The Linux Performance Monitoring Tools by SolarWinds is a paid tool, but if you want a solution ready to use very quickly, they are a great opportunity for your Linux monitoring.

Node Exporter & Prometheus

Now that we know what the existing tools for Linux monitoring are, let’s have a look at what we are going to use today: the node exporter and Prometheus.

As stated before, Prometheus scrapes targets and the node exporter is just one of them.

In your architecture, you will have the following components:

• A time-series database, in this case, Prometheus made to store the different metrics retrieved by the node exporter
• A Node Exporter run as a systemd service that will periodically (every 1 second) gathers all the metrics of your system.
• A dashboard solution, in this case, Grafana, displaying metrics gathered from Prometheus. We are not going to build every single panel by ourselves. Instead, we are going to use a very powerful feature of Grafana which is the dashboard import. (as a reminder, Grafana has a list of hundreds of dashboards that you import within the UI)

In our case, the whole stack will be run within the same instance, so there will be no need to configure any firewall rules.

The node exporter will run on port 9100 and Prometheus will run on port 9090

Note : as part of the configuration, Prometheus can actually monitor itself.

Now that you have an idea of what a monitoring architecture looks like, let’s install the different tools needed.

Installation of Required Tools

As a reminder, for our architecture, we are going to need Prometheus, the Node Exporter, and Grafana.

This tutorial focuses on installing the node exporter completely, but there’s also a quick installation for other tools.

Installing the Node Exporter

Before installing Prometheus, we need to install the Node Exporter as a service.

Head to and download the latest binary for the node exporter (here 0.18.1)

$ wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz

The archive contains a single binary which is node_exporter.

This is the binary we are going to launch to start gathering metrics on our system.

Now that you have the archive, extract it, and inspect its content it.

$ tar xvzf node_exporter-0.18.1.linux-amd64.tar.gz

Now to install it as a service, here are the instructions:

• Create a node exporter user

$ sudo useradd -rs /bin/false node_exporter

• Copy the binary to your /usr/local/bin folder.

$ cp node_exporter-0.18.1.linux-amd64/node_exporter /usr/local/bin

• Apply the correct permissions to your binary file.

$ chown node_exporter:node_exporter /usr/local/bin/node_exporter

Note: you will need to have a node_exporter user to run the binary.

• Navigate to /etc/systemd/system and create a new service file

$ cd /etc/systemd/systemd
$ sudo vim node_exporter.service

Then, paste the following configuration for your service.

[Unit]
Description=Node Exporter
After=network-online.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

• Exit vi, reload your daemon and start your service.

$ sudo systemctl daemon-reload
$ sudo systemctl start node_exporter

• Check your service by running the following command

$ sudo systemctl status node_exporter.service

Is your service running correctly?

• Enable your service for system startup

$ sudo systemctl enable node_exporter

• Verify that your node exporter is correctly up and running with a simple curl command

$ curl http://localhost:9100/metrics

Can you see the key-value pairs metrics of your system metrics?

We are done with the Node Exporter installation!

Installing Prometheus

It is not the first time we install Prometheus for our projects.

First, head over to and run a simple wget command to get the latest binaries.

$ wget https://github.com/prometheus/prometheus/releases/download/v2.10.0/prometheus-2.10.0.linux-amd64.tar.gz

For this tutorial, we are running the 2.10.0 version from May 2019.

You should now have an archive, extract it, and navigate inside the folder.

# tar xvzf prometheus-2.10.0.linux-amd64.tar.gz
# cd prometheus-2.10.0.linux-amd64/

Inside this folder, you have multiple elements:

• Prometheus: the executable file that launches a Prometheus server;
• prometheus.yml: the configuration file for your Prometheus server;
• promtool: a tool that can be used to check your Prometheus configuration.

In our case, we are first going to modify the Prometheus configuration file.

Navigate in it with vi:

# vi prometheus.yml

Then perform the following modifications:

global:
  scrape_interval:     1s # Set the scrape interval to every 1 second.

As a reminder, Prometheus scrapes targets. In our case, we want it to scrape our system metrics every one second.

Then, in the “scrape_configs” section, under the “static_configs” section, add the following lines.

static_configs:
            - targets: ['localhost:9090', 'localhost:9100']

It means that Prometheus will scrape the node exporter metrics, as well as its own metrics.

For now, simply launch Prometheus as a background process, and verify that it is correctly launched by pinging the Web UI interface.

> ./prometheus &

Can you see the web console?

We are done with the Prometheus setup!

Installing Grafana

The last part of our initialization section is about installing Grafana.

As a reminder, Grafana is an open-source dashboard monitoring solution that binds to databases in order to display metrics in a variety of ways.

To install Grafana, head over to https://grafana.com/grafana/download and download the latest binaries available for you.

$ wget https://dl.grafana.com/oss/release/grafana_6.2.4_amd64.deb

For this tutorial, we are using the 6.2.4 version of Grafana that includes the new bar gauges.

Extract the .deb file and Grafana should automatically start as a service on your computer.

$ sudo dpkg -i grafana_6.2.4_amd64.deb

You can verify that Grafana is running with the following command:

$ sudo systemctl status grafana-server
● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-06-22 10:43:12 UTC; 5 days ago
     Docs: http://docs.grafana.org

If the status is set to Active, and if no visible errors are shown in your console, then Grafana is now installed correctly on your machine.

By default, Grafana is running on port 3000, and the default credentials are

• username: admin
• password: admin

You will be asked to change them immediately on the first connection.

With a web browser, head over to http://localhost:3000, and follow the instructions until you see the welcome screen.
As described in the screenshot, click on ‘Add data source‘ and create a new Prometheus data source.

As described in our other tutorials, you can configure your data source with the following configuration:

Note: If you configured your Prometheus instance to run on another port, you have to change it in the configuration.

Building a complete Grafana Dashboard for the Node Exporter

Now that all your tools are set, there is not much work to do in order to have our dashboards.

For this, we are not going to build our dashboards by ourselves. Instead, we are going to use the “Import Dashboard” feature of Grafana.

In the top left menu, hover your mouse over the “Plus” icon and click on the “Import” element in the dropdown list.

You will be presented with the following window:

In this window, you have multiple choices. You can either:

• Type a dashboard URL or ID and it will be automatically imported into your Grafana instance.
• Upload a JSON file (as a quick reminder, Grafana dashboards are exported as JSON files and can be easily shared this way)
• Paste directly the raw JSON

In our case, we are going to use the first option by typing the dashboard id directly in the text field.

Getting Inspiration for Dashboards

We don’t have to build the dashboards all by ourselves.

This is especially true when you have dozens of metrics to look for.

You would have to spend a lot of time understanding the different metrics and building panels out of them.

We are going to use Grafana Dashboards for this. Grafana Dashboards is a repository owned by Grafana that stores hundreds of dashboards for you to choose from.

In our case, we are going to focus on Node Exporter dashboards.

Type “Node Exporter” in the search box and scroll until you reach the “Node Exporter Full” dashboard.

As you probably noticed it, the dashboard has the ID 1860 (the information is available into the URL of the website).

This is the ID that we are going to copy in order to monitor our system.

In the import, type “1860” in the Grafana.com dashboard text field, and hit “Load”.

You will be presented with a second screen to configure your dashboard.

Every field is filled automatically.

However, you will have to select the data source, in my case “Prometheus“. (you have to link it to the data source you created in section 2)

Hit “Import” when you are done.

In under a second, all the different panels will be built for you!

That’s 29 categories with over 192 panels created automatically. Awesome.

Here are some examples of how the dashboards look:

You can also have a look at all the options available in this whole panel.

Going Further

Mastering the Node Exporter is definitely a must-have skill for engineers willing to get started with Prometheus.

However, you can dig a little bit deeper using the Node Exporter.

a – Additional Modules

Not all modules are enabled by default, and if you run a simple node exporter installation, chances are that you are not running any additional plugins.

Here’s the list of the additional modules:

In order to activate them, simply add a –collector.<name> flag when running the node exporter, for example:

ExecStart=/usr/local/bin/node_exporter --collector.processes --collector.ntp

This should activate the processes and the ntp collectors.

b – TextFile Collector

A complete Node Exporter guide would not be complete without talking about the TextFile collector, at least for a small section.

Similar to the Pushgateway, the textfile collector collects metrics from text files and stores them right into Prometheus.

It is designed for batch jobs or short-lived jobs that don’t expose metrics in a continuous way.

Some examples of the textfile collector are available here:

• Using the text file collector from a shell script.
• Monitoring directory sizes with the textfile collector.

c – Videos Resources

Finally, I like to link to external videos that are closely related to the subject or particularly interesting.

A dashboard is a group of one or more panels structured and arranged into one or more rows. Grafana is one of the most awesome dashboards. Grafana Dashboard makes it easy to build the right queries, and personalize the display properties so that you can build a flawless dashboard for your requirement.

If you are looking to monitor your entire infrastructure or just your home, everybody helps by having a complete Grafana dashboard. In today’s tutorial, we are discussing completely how we can easily create a Grafana dashboard, what are the best practices, what the different panels are, about Dashboard UI, and how they can be used efficiently.

• Best practices for creating dashboards
• Dashboard UI
• Steps to create a Grafana dashboard using UI
• Steps to create a Grafana dashboard using API

Best practices for creating dashboards

This section will make you understand some best practices to follow when creating Grafana dashboards:

• At the time of new dashboard creation, ensure that it has a meaningful name.
  • If you want to create a dashboard for the experiment then set the word TEST or TMP in the name.
  • Also, include your name or initials in the dashboard name or as a tag so that people know who owns the dashboard.
  • After performing all the testing tasks on temporary experiment dashboards, remove all of them.
• If you build multiple related dashboards, consider how to cross-reference them for easy navigation. For more information on this take a look at the best practices for managing dashboards.
• Grafana retrieves data from a data source. A basic understanding of data sources in general and your precise is necessary.
• Withdraw unnecessary dashboard stimulating to diminish the load on the network or backend. For instance, if your data changes every hour, then you don’t need to set the dashboard refresh rate to 30 seconds.
• Perform the left and right Y-axes when displaying time series with multiple units or ranges.
• Reuse your dashboards and drive consistency by utilizing templates and variables.
• Add documentation to dashboards and panels.
  • To add documentation to a dashboard, add a Text panel visualization to the dashboard. Record things like the purpose of the dashboard, useful resource links, and any instructions users might need to interact with the dashboard. Check out this Wikimedia example.
  • To add documentation to a panel, edit the panel settings and add a description. Any text you add will appear if you hover your cursor over the small ‘i’ in the top left corner of the panel.
• Beware of stacking graph data. The visualizations can be misleading, and hide related data. We advise turning it off in most cases.

Also Check: Best Open Source Dashboard Monitoring Tools

Dashboard UI

• Zoom out time range
• Time picker dropdown: You can access relative time range options, auto-refresh options, and set custom absolute time ranges.
• Manual refresh button: Will let all panels refresh (fetch new data).
• Dashboard panel: Tap the panel title to edit panels.
• Graph legend: You can change series colors, y-axis, and series visibility directly from the legend.

Steps to create a Grafana dashboard using UI

• Hover the ‘Plus’ icon located on the left menu with your cursor (it should be the first icon)
• At that point, a dropdown will open. Click on the ‘dashboard’ option.

• Create a dashboard option in Grafana
• A new dashboard will automatically be created with a first panel.

In Grafana v6.0+, the query and the visualization panels are departed. It implies that you can easily write your query, and decide later which visualization you want to use for your dashboard.

• Complete MySQL dashboard with Grafana & Prometheus | MySQL Database Monitoring using Grafana and Prometheus
• MongoDB Monitoring with Grafana & Prometheus | Mongodb Prometheus Grafana Dashboard
• Windows Server Monitoring using Prometheus and WMI Exporter | How to Install WMI Exporter in Windows?

This is particularly handy because you don’t have to reset your panel every time when you want to change the visualization types.

• First, click on ‘Add Query’ and make sure that your data source is correctly bound to Grafana.
• Write your query and refactor it until you’re happy with the final result. By default, Grafana sets new panels to “Graph” types.

• Choose the visualization that fits your query the best. You have to choose between ten different visualizations (or more if you have plugins installed!)

• Tweak your dashboard with display options until you’re satisfied with the visual of your panel.

• Add more panels, and build a complete dashboard for your needs! Here is an example of what a dashboard could be with a little bit of work. Here is an example with a more futuristic theme on disk monitoring.

Steps to create a Grafana dashboard using API

Most of the API requests are authenticated within Grafana. To call the Grafana API to create a dashboard, you will have to get a token. If you don’t own the Grafana example, you have to ask your administrator for a token.

• Hover the ‘Configuration’ icon in the left menu and click on the “API Keys” option.

• Click on “Add API Key”. Enter a key name and at least an “Editor” role to the key.
• Click on “Add”

• A popup page will open and show you the token you will be using in the future. It is very important that you copy it immediately. You won’t be able to see it after closing the window.

• Now that you have your API key, you need to make a call to the /api/dashboards/db endpoint using the token in the authorization header of your HTTP request.

For this example, I will be using Postman.

• Create a new POST request in Postman, and type http://localhost:3000/api/dashboards/db as the target URL.
• In the authorization panel, select the ‘Bearer token’ type and paste the token you got from the UI.

• In the body of your request, select “Raw” then “JSON (application/json)”. Paste the following JSON to create your dashboard.

{
  "dashboard": {
    "id": null,
    "uid": null,
    "title": "Production Overview",
    "tags": [ "templated" ],
    "timezone": "browser",
    "schemaVersion": 16,
    "version": 0
  },
  "folderId": 0,
  "overwrite": false
}

Here’s the description of every field in the request:

• dashboard.id: the dashboard ID, should be set to null on dashboard creation.
• dashboard.uid: the dashboard unique identifier, should be set to null on dashboard creation.
• title: the title of your dashboard.
• tags: dashboard can be assigned tags in order to retrieve them quicker in the future.
• timezone: the timezone for your dashboard, should be set to the browser on dashboard creation.
• schema version: constant value that should be 16.
• version: your dashboard version, should be set to zero as it is the first version of your dashboard.
• folderId: you can choose to set a folder id to your dashboard if you already have existing folders
• overwrite: you could update an existing dashboard, but it should be set to false in our case as we creating it.

• Click on “Send”. You choose to see the following success message.

{
"id": 3,
"slug": "production-overview",
"status": "success",
"uid": "uX5vE8nZk",
"url": "/d/uX5vE8nZk/production-overview",
"version": 1
}

• Make sure that your dashboard was created in Grafana.

That’s it! You now have a complete idea of the two ways to create a Grafana dashboard in 2021.

If you have any comments on this content, or if you found that this guide has run out of date in the future, make sure to leave a comment below.

In this informative tutorial, we have covered complete details about InfluxDB like what exactly it is, why you use it, What value can developers design by fusing InfluxDB into their own environment? and many others.

Also, this guide can become an initial stage for every developer, engineer, and IT professional to understand InfluxDB concepts, use-cases, and real-world applications.

The main objective of curating this article is to make you an expert with InfluxDB in no time. So, we have designed the InfluxDB learning paths into diverse modules, each one of them bringing a new level of knowledge of time-series databases.

• How To Install InfluxDB 1.7 and 2.0 on Linux in 2021
• How To Create a Database on InfluxDB 1.7 & 2.0
• InfluxDays London 2021 Recap | Key Facts of InfluxDays London 2021

In this Definitive Guide To InfluxDB In 2021, firstly, you will gain some knowledge on the overall presentation of time-series databases, then with an in-depth explanation of the concepts that define InfluxDB, and at last, we explained the use-cases of InfluxDB and how it can be used in a variety of industries by using real-world examples.

Hence, step into the main topic and learn completely about InfluxDB Open Source Time Series Database, Key concepts, Use cases, etc. Let’s make use of the available links and directly jump into the required stuff of InfluxDB.

• What is InfluxDB?
• What are Time-Series Databases?
  • Why do we need time-series databases?
  • Specific concepts about time-series databases
  • A Whole Different Ecosystem
    • Data consumption
    • Data Production
• Illustrated InfluxDB Concepts
  • InfluxDB Query Language
  • Explained InfluxDB Key Concepts
    • Database
    • Measurement
    • Tags & Fields
    • Timestamp
    • Retention policy
    • Point
• InfluxDB Use-Cases
  • DevOps Monitoring
  • IoT World
  • Industrial & Smart Plants
• Your Own Imagination!
• Going Further

What is InfluxDB?

INfluxDB is definitely a fast-growing technology. The time-series database, developed by InfluxData, is seeing its popularity grow more and more over the past few months. It has become one of the references for developers and engineers willing to bring live monitoring into their own infrastructure.

Do Check: InfluxDays London Recap

What are Time-Series Databases?

Time Series Databases are database systems specially created to handle time-related data.

All your life, you have dealt with relational databases like MySQL or SQL Server. You may also have dealt with NoSQL databases like MongoDB or DynamoDB.

Those systems are based on the fact that you have tables. Those tables contain columns and rows, each one of them defining an entry in your table. Often, those tables are specifically designed for a purpose: one may be designed to store users, another one for photos, and finally for videos. Such systems are efficient, scalable, and used by plenty of giant companies having millions of requests on their servers.

Time series databases work differently. Data are still stored in ‘collections’ but those collections share a common denominator: they are aggregated over time.

Essentially, it means that for every point that you are able to store, you have a timestamp associated with it.

The great difference between relational databases and time-series databases

But.. couldn’t we use a relational database and simply have a column named ‘time’? Oracle for example includes a TIMESTAMP data type that we could use for that purpose.

You could, but that would be inefficient.

Why do we need time-series databases?

Three words: fast ingestion rate.

Time series databases systems are built around the predicate that they need to ingest data in a fast and efficient way.

Indeed, relational databases do have a fast ingestion rate for most of them, from 20k to 100k rows per second. However, the ingestion is not constant over time. Relational databases have one key aspect that makes them slow when data tend to grow: indexes.

When you add new entries to your relational database, and if your table contains indexes, your database management system will repeatedly re-index your data for it to be accessed in a fast and efficient way. As a consequence, the performance of your DBMS tends to decrease over time. The load is also increasing over time, resulting in having difficulties to read your data.

Time Series databases are optimized for a fast ingestion rate. It means that such index systems are optimized to index data that are aggregated over time: as a consequence, the ingestion rate does not decrease over time and stays quite stable, around 50k to 100k lines per second on a single node.

Specific concepts about time-series databases

On top of the fast ingestion rate, time-series databases introduce concepts that are very specific to those technologies.

One of them is data retention. In a traditional relational database, data are stored permanently until your decide to drop them yourself. Given the use-cases of time series databases, you may want not to keep your data for too long: either because it is too expensive to do so, or because you are not that interested in old data.

Systems like InfluxDB can take care of dropping data after a certain time, with a concept called retention policy (explained in detail in part two). You can also decide to run continuous queries on live data in order to perform certain operations.

You could find equivalent operations in a relational database, for example, ‘jobs’ in SQL that can run on a given schedule.

A Whole Different Ecosystem

Time Series databases are very different when it comes to the ecosystem that orbits around them. In general, relational databases are surrounded by applications: web applications, software that connects to them to retrieve information or add new entries.

Often, a database is associated with one system. Clients connect to a website, that contacts a database in order to retrieve information. TSDB is built for client plurality: you do not have a simple server accessing the database, but a bunch of different sensors (for example) inserting their data at the same time.

As a consequence, tools were designed in order to have efficient ways to produce data or to consume it.

Data consumption

Data consumption is often done via monitoring tools such as Grafana or Chronograf. Those solutions have built-in solutions to visualize data and even make custom alerts with it.

Those tools are often used to create live dashboards that may be graphs, bar charts, gauges or live world maps.

Data Production

Data production is done by agents that are responsible for targeting special elements in your infrastructure and extract metrics from them. Such agents are called “monitoring agents“. You can easily configure them to query your tools in a given time span. Examples are Telegraf (which is an official monitoring agent), CollectD or StatsD

Now that you have a better understanding of what time series databases are and how they differ from relational databases, it is time to dive into the specific concepts of InfluxDB.

Illustrated InfluxDB Concepts

In this section, we are going to explain the key concepts behind InfluxDB and the key query associated with it. InfluxDB embeds its own query language and I think that this point deserves a small explanation.

InfluxDB Query Language

Before starting, it is important for you to know which version of InfluxDB you are currently using. As of April 2019, InfluxDB comes in two versions: v1.7+ and v2.0.

v2.0 is currently in alpha version and puts the Flux language as a centric element of the platform. v1.7 is equipped with InfluxQL language (and Flux if you activate it).

The main differences between v1.7 and v2.0

Right now, I do recommend keeping on using InfluxQL as Flux is not completely established in the platform.

InfluxQL is a query language that is very similar to SQL and that allows any user to query its data and filter it. Here’s an example of an InfluxQL query :

See how similar it is to the SQL language?

In the following sections, we are going to explore InfluxDB key concepts, provided with the associated IQL (short for InfluxQL) queries.

Explained InfluxDB Key Concepts

In this section, we will go through the list of essential terms to know to deal with InfluxDB in 2021.

Database

A database is a fairly simple concept to understand on its own because you are applied to use this term with relational databases. In a SQL environment, a database would host a collection of tables, and even schemas and would represent one instance on its own.

In InfluxDB, a database host a collection of measurements. However, a single InfluxDB instance can host multiple databases. This is where it differs from traditional database systems. This logic is detailed in the graph below :

The most common ways to interact with databases are either creating a database or by navigating into a database in order to see collections (you have to be “in a database” in order to query collections, otherwise it won’t work).

Measurement

As shown in the graph above, the database stores multiple measurements. You could think of a measurement as a SQL table. It stores data, and even metadata, over time. Data that are meant to coexist together should be stored in the same measurement.

Measurement Example

Measurement IFQL example

In a SQL world, data are stored in columns, but in InfluxDB we have two other terms: tags & fields.

Tags & Fields

Warning! This is a very important chapter as it explains the subtle difference between tags & fields.

When I first started with InfluxDB, I had a hard time grasping exactly why are tags & fields different. For me, they represented ‘columns’ where you could store exactly the same data.

When defining a new ‘column’ in InfluxDB, you have the choice to either declare it as a tag or as a value and it makes a very big difference.

In fact, the biggest difference between the two is that tags are indexed and values are not. Tags can be seen as metadata defining our data in the measurement. They are hints giving additional information about data, but not data itself.

Fields, on the other side, is literally data. In our last example, the temperature ‘column’ would be a field.

Back to our cpu_metrics example, let’s say that we wanted to add a column named ‘location’ as its name states, defines where the sensor is.

Should we add it as a tag or a field?

In our case, it would be added as a.. tag! We definitely want the location ‘column’ to be indexed and taken into account when performing a query over the location.

In general, I would advise keeping your measurements relatively small when it comes to the number of fields. More and more fields often rhyme with lower performance. You could create other measurements to store another field and index it properly.

Now that we’ve added the location tag to our measurement, let’s go a bit deeper into the taxonomy.

A set of tags is called a “tag-set”. The ‘column name’ of a tag is called a “tag key”. Values of a tag are called “tag values”. The same taxonomy repeats for fields. Back to our drawings.

Timestamp

Probably the simplest keyword to define. A timestamp in InfluxDB is a date and a time defined in RFC3339 format. When using InfluxDB, it is very common to define your time column as a timestamp in Unix time expressed in nanoseconds.

Tip: you can choose a nanosecond format for the time column and reduce the precision later by adding trailing zeros to your time value for it to fit the nanosecond format.

Retention policy

This feature of InfluxDB is for me one of the best features there is.

A retention policy defines how long you are going to keep your data. Retention policies are defined per database and of course, you can have multiple of them. By default, the retention policy is ‘autogen‘ and will basically keep your data forever. In general, databases have multiple retention policies that are used for different purposes.

What are the typical use-cases of retention policies?

Let’s pretend that you are using InfluxDB for live monitoring of an entire infrastructure.

You want to be able to detect when a server goes off for example. In this case, you are interested in data coming from that server in the present or short moments before. You are not interested in keeping the data for several months, as a result, you want to define a small retention policy: one or two hours for example.

Now if you are using InfluxDB for IoT, capturing data coming from a water tank for example. Later, you want to be able to share your data with the data science team for them to analyze it. In this case, you might want to keep data for a longer time: five years for example.

Point

Finally, an easy one to end this chapter about InfluxDB terms. A point is simply a set of fields that has the same timestamp. In a SQL world, it would be seen as a row or as a unique entry in a table. Nothing special here.

Congratulations on making it so far! In the next chapter, we are going to see the different use-cases of InfluxDB and how it can be used to take your company to the next level.

InfluxDB Use-Cases

Here is a detailed explanation of InfluxDB Use-Cases:

DevOps Monitoring

DevOps Monitoring is a very big subject nowadays. More and more teams are investing in building fast and reliable architectures that revolve around monitoring. From services to clusters of servers, it is very common for engineers to build a monitoring stack that provides smart alerts.

If you are interested in learning more about DevOps Monitoring, I wrote a couple of guides on the subject, you might find them relevant to your needs.

From the tools defined in section 1, you could build your own monitoring infrastructure and bring direct value to your company or start-up.

IoT World

The IoT is probably the next big revolution that is coming in the next few years. By 2020, it is estimated that over 30 billion devices will be considered IoT devices. Whether you are monitoring a single device or a giant network of IoT devices, you want to have accurate and instant metrics for you to take the best decisions regarding the goal you are trying to achieve.

Real companies are already working with InfluxDB for IoT. One example would be WorldSensing, a company that aims at expanding smart cities via individual concepts such as smart parking or traffic monitoring system. Their website is available here :

Industrial & Smart Plants

Plants are becoming more and more connected. Tasks are more automated than ever : as a consequence it brings an obvious need to be able to monitor every piece of the production chain to ensure a maximal throughput. But even when machines are not doing all the work and humans are involved, time-series monitoring is a unique opportunity to bring relevant metrics to managers.

Besides reinforcing productivity, they can contribute to building safer workplaces as they are able to detect issues quicker. Value for managers as well as for workers.

Your Own Imagination!

The examples detailed above are just examples and your imagination is the only limit to the applications that you can find for Time Series databases. I have shown it via some articles that I wrote, but time-series can be even used in cybersecurity!

If you have cool applications of InfluxDB or time-series database, post them as comments below, it is interesting to see what idea people can come up with.

Going Further

In this article, you learned many different concepts: what are time-series databases and how they are used in the real world. We have gone through a complete list of all the technical terms behind InfluxDB, and I am confident now to say that you are to go on your own adventure.

My advice to you right now would be to build something on your own. Install it, play with it, and start bringing value to your company or start-up today. Create a dashboard, play with queries, setup some alerts: there are many things that you will have to do in order to complete your InfluxDB journey.

If you need some inspiration to go further, you can check the other articles that we wrote on the subject: they provide clear step-by-step guides on how to setup everything.

In this digital world, every small and big organization is coming up with their best services in a website form for a good reach into the audience. The rise of their volume and value of the data growing very frequently. Are you a bit worried to get more value out of your data? Not anymore, switch to the dashboard technique where it serves as an important tool to monitor and control the situation within an organization.

While storing data in a time-series database, usually, you need to visualize and analyze it to have a more precise idea of trends, seasonalities, or unexpected changes that may be anomalies. This is when the open-source dashboard monitoring tools come into play.

In this tutorial, we are going to concentrate mainly on the 4 best open source dashboard monitoring tools in 2021 along with what is dashboard & dashboard software with their key aspects. However, we will also discuss what their qualities are, the industries they are linked to, and how they vary from each other.

• What is Dashboard?
• What is Dashboard Software?
• Types of Dashboard
• How to find the perfect Dashboard?
• Key Aspects of Dashboard Software
• What is an Open Source Dashboard Monitoring Tool?
• Advantages of Dashboard Management Software
• Best Free Open Source Dashboard Monitoring Tools in 2021

What is Dashboard?

A dashboard is a tool that performs all administration KPIs (key performance indicators) and crucial data points at a particular place that assists in monitoring the strength of the business or department. The dashboard analyzes the complex data sets by making use of data visualization, which in turn supports users to gain knowledge of the current performance at a glance. The user can visualize data in the form of charts, graphs, or maps.

• How To Create a Grafana Dashboard? (UI + API methods) | Best Practices for Creating Dashboard
• Complete Node Exporter Mastery with Prometheus | Monitoring Linux Host Metrics WITH THE NODE EXPORTER
• Monitoring Linux Processes using Prometheus and Grafana | Prometheus & Grafana Linux Monitoring

What is Dashboard Software?

Dashboard software serves as an automated tool that analyzes complex data sets and assists in revealing the patterns of data processing. With the help of dashboard management software, users can easily access, interact, and analyze up-to-date information at a centralized location. The usage of this technology is very huge, you can utilize it in different business processes like marketing, human resource, sales, and production. Mainly, it helps business people to monitor their business performance at a glance.

Also Check: 6 Tips To A Successful Code Review

Types of Dashboard

Present in the market, you can discover different dashboard types that are depending on where they are utilized like for large enterprises or for small-scale industries. The types of the dashboard are as follows:

1. Tactical Dashboards: Managers who need a deeper knowledge of a company’s actions make use of it.
2. Operational Dashboards: It is applied in sales, finances, services, manufacturing, and human resources.
3. Strategic Dashboards: Senior executive uses this type to monitor the progress of the company striving to reach strategic goals.

How to find the perfect Dashboard?

By following these tips, we can easily discover the perfect dashboard for your firm or business:

• Ease of Use
• Customization
• Scalability
• Integration
• Extendable
• Modularity
• Security Management
• Exporting Options

Key Aspects of Dashboard Software

Remember that your dashboard software tool should include all required features for your business so that you can obtain the most out of your data:

• Global Dashboard Filters
• Dynamic Images
• Multiple Sharing Options
• Embedded Analytics
• Dashboard tabs
• Visual Representations
• 24/7 Dashboard Access
• Predefined Dashboard Templates
• Printing Bounds
• Public Links

What is an Open Source Dashboard Monitoring Tool?

Open source dashboard monitoring tools are designed to provide powerful visualizations to a wide variety of data sources. Often linked with time-series databases, they can also be linked with regular relational databases.

Advantages of Dashboard Management Software

The benefits of Dashboard Management Software are provided in the following shareable image:

Best Free Open Source Dashboard Monitoring Tools in 2021

The following four main free and open-source dashboard software tools provide high-quality options free of cost.

• 1. Grafana
• 2. Chronograf
• 3. Netdata
• 4. Kibana

Check out the advantages, limitations, uses, and many more about them from the below modules:

1. Grafana

Grafana is by far one of the most popular dashboard monitoring systems in use.

Released in 2013 and developed by Grafana Labs, Grafana plugs into a wide variety of data sources and provides a ton of panels to visualize your data.

One of the most common usages of Grafana is plugging into time series databases in order to visualize data in real-time. For certain panels, Grafana is equipped with an alerting system that allows users to build custom alerts when certain events occur on your data.

Gauges, world maps, tables, and even heatmaps are concrete examples of panels that you are going to find in Grafana.

New panels are released very frequently: as we write this article, Grafana just announced v6.2 which is shipping the brand new bar gauge panel.

As described previously, Grafana plugs to many different data sources: InfluxDB or Prometheus are examples of time series databases available; for relational databases, you can easily plug to MySQL or PostgreSQL databases (or TimescaleDB). Indexes are also available via the ElasticSearch connector.

In my opinion, Grafana remains a reference for open-source dashboard monitoring. Their latest additions (such as the ‘Explore’ function or the new panels) emphasize their ambition to bring a global tool for monitoring, alerting, and analyzing data.

For curious developers, you can check Torkel Ödegaard’s great talk at GrafanaCon 2019 where he described Grafana’s roadmap and future projects.

2. Chronograf

Developed by InfluxData for many years, Chronograf is a solid alternative to Grafana when it comes to visualizing and exploring your data for InfluxDB data sources.

Chronograf exposes similar panels but there is one major difference with Grafana: Chronograf really focuses on exploring, querying, and visualizing data using InfluxQL and the Flux language. If you’re not familiar with what the Flux language is, you can check the article that I wrote that unveils the different capabilities of this new programming language.

So should you use Grafana or Chronograf?

In the end, it all comes down to your needs.

If you’re dealing a lot with InfluxDB in your infrastructure, then you should use Chronograf as it is specifically designed to handle InfluxDB databases.

On the other hand, if you have a variety of data sources, you should use Grafana. Those tools have similar abilities but Chronograf is more Influx-centered than Grafana.

As Tim Hall mentioned in his “Chronograf – Present & Future” talk in InfluxDays 2018: the answer is to try both!

The UI aspect of Chronograf is very decent and modern: I think that you should try it at least once if you’re dealing with InfluxDB databases.

Would you link to see what a Chronograf dashboard look like? Head over to my ‘Monitoring systemd services in real-time using Chronograf‘ article!

3. Netdata

Netdata is a tool that tracks performance and monitors health for a wide panel of systems and applications.

Netdata is configuration-based and runs as a daemon on the target machine.

Furthermore, Netdata is plugin-based.

When defining your daemon, you can choose from a panel of plugins that are either internal or external.

When you are set, there are two ways for you to retrieve and visualize data:

• “Pull” method: you can set Netdata to run on individual nodes and plug your dashboards directly into it. This way, you can scale your node to your needs and you are not concerned about the scaling of different nodes. Also, storage is scoped to what’s really needed by a particular node thus more efficient;
• “Push” method: Similar to what you would find in Prometheus with Pushgateway, you can ‘push’ metrics to a centralized place. You may find this handy for jobs that have a small lifespan such as batch jobs.

With Netdata, you can easily configure streaming pipelines for your data and replication databases.

This way, you can scale slave nodes depending on your needs and adapt to the actual demand.

Netdata’s website is available here: https://my-netdata.io/

4. Kibana

Any dashboard monitoring ranking wouldn’t be complete without mentioning Kibana.

Kibana is part of Elastic’s product suite and is often used in what we call an ELK stack: ElasticSearch + Logstash + Kibana.

You probably know ElasticSearch, the search engine based on the Lucene language.

If you’re unfamiliar with Elastic products, ElasticSearch provides a REST-based search engine that makes it fast and easy to retrieve data. It is often used in companies that are looking to speed up their data retrieval processes by providing fast interfaces to their end-users.

Logstash can be defined as a log pipeline. Similar to rsyslog, you can bind it to an extensive list of data sources (AWS, databases, or stream pipelines such as Kafka). Logstash will collect, transform data and insert it into ElasticSearch. Finally, Kibana will be used to visualize data stored in ElasticSearch.

As you guessed it, Kibana is suited for log monitoring and has nothing to do with the direct network or DevOps monitoring (even if you could store logs related to servers or virtual machines!)

Wrapping Up

After referring to the above data, you get an idea of the best free and open-source dashboard monitoring tools. Now, it’s time for applying them to your company’s tech stack. Check out and analyze how do you plan on adding them to your company tech stack?

Well, are you utilizing them already? please let us know what were you able to accomplish with them? How did they specifically add value to your business?

Thank you for reading this article, I hope that you found it insightful. Until then, have fun, as always, and also visit our site for better knowledge on various technologies.

Learning the subject from various modern world options like podcasts, videos, blogs, expert classes, etc. can be your wishlist but reading a good book is the final order where people enjoy & gain knowledge without any loss. Hence, find the best software engineering books & kickstart your learnings.

Discovering the top-most Software engineering textbooks in 2021 can be difficult for everyone. But people who have viewed our article can find it very easily and effortlessly. As we are going to give you a compiled list of the best books on software engineering subjects where recommended by a few experts.

Before going to review all these top 10 best software engineering books in 2021 that are available in this tutorial, we want to suggest you view and remember a few factors that help you select the right book for your further learnings. They are as fashioned:

• High Recommendations
• Editor Reviews
• Hardcover/paperback
• Pricing

This tutorial completely focuses on the best software engineering books available for software engineers, developers, and project managers.

• 4 Best Open Source Dashboard Monitoring Tools In 2021
• 6 Tips To A Successful Code Review | Best Practices for Effective Code Review
• How To Install Git On Debian 10 Buster | Debian Git Repository | Debian Buster Git

Best New Software Engineering Books To Read in 2021

Software engineering is described as a process of analyzing user requirements and then designing, building, and testing software applications to fit those requirements. Guys who are beginners, or excited to learn coding, or expert ones can check the top 10 list of the best software engineering books 2021 below:

1. Clean Code by Robert Martins
2. Design Patterns: Elements of Reusable Object-Oriented Software by Eric Gamma
3. Patterns of Enterprise Application Architecture by Martin Fowler
4. Enterprise Integration Patterns by Gregor Hohpe
5. The Mythical Man-Month by Frederick Brooks
6. Code Complete by Steve McConnell
7. Git for Teams by Emma Hogbin Westby
8. Refactoring: Improving the Design of Existing Code by Martin Fowler
9. The Art of Unit Testing by Roy Osherove
10. Soft Skills: The Software Developer’s Life Manual by John Sonmez

1 – Clean Code by Robert Martins

Probably one of the greatest books about software engineering and programming. Every engineer, developer, or programmer should have read this book, at least one time.

In this book, Robert Martin provides clear and concise chapters about:

• How to write high-quality and expressive code;
• How to name your functions, your variables, essentially conveying your intent in your coding style;
• How to unit test properly, why it matters, and how to do it properly;
• How to choose relevant data structures and why they can make or break a piece of code;
• How to write comments but most importantly how NOT to write comments;
• How error handling works and how to properly engineer an exception handling workflow through your application or program

The book also provides real-life examples written in Java, so if you are familiar with object-oriented programming, that should not be an issue at all.

This book really helps to build code maturity. It actually helps you going from “How do I technically do this” to “How do I properly technically do this?” which is most of the time a neglected point by engineers.

Oh and for those who are wondering, what did the book from the introduction become?

I gave it to an aspiring Java engineer at my current job!

This book is ideal for junior developers and young software developers or engineers.

2 – Design Patterns: Elements of Reusable Object-Oriented Software by Eric Gamma

This software engineering book is a great follow-up to the Clean code manual.

As Clean Code gives you the foundations of programming, Design Patterns teaches you recipes to write manageable and scalable code.

For small or large programs, thinking about how to design it from the get-go is one of the mandatory skills of a good software engineer.

Most of the time, when designing a project, you don’t have to reinvent the wheel. You can open your design pattern book and pick one that fits your needs.

From there you have the guarantee that your project will be able to scale, and you are also given tools to scale it properly.

As examples, here are some design patterns that are taught in the book (and that I use on a daily basis)

• Abstract Factory: that lets you abstract object creation and decouples concrete objects from the business logic where they might be used;
• Observer: builds links between objects that allow them to be notified when a certain event occurs in one of them. Very useful for real-time applications or in industrial programs;
• Iterator: that enables developers to iterate on objects without knowing the implementation details of those data structures.

This book is really ideal for people willing to become either senior software engineers or solution architects.

Looking to master design patterns? Here’s where to get Design Patterns by Eric Gamma

3 – Patterns of Enterprise Application Architecture by Martin Fowler

Now that you know how to code, as well as how to design your code, it is time for you to know how to structure applications on an entreprise level.

Applications grow over time, and very often, they grow to a size that no one could have predicted.

However, you need to have concepts of entreprise architecture when you are building an application.

Are you correctly layering your application? If you are building a web application, are you aware of all the different presentational designs that you can choose from?

How are you accessing your data and how are you making sure that you are efficiently decoupling data from the applications that are trying to access them?

This book helps you master those concepts, and they can really play a big role in the life of an application.

This book, among other themes, teaches the following concepts :

• How to organize your domain and your business logic in your application;
• Techniques on how to access data in an application and how to build solid object-relational mappings for your databases;
• How to handle concurrency in applications and what patterns to use to avoid deadlocks;
• Web Presentations Patterns: MVC, MVVM, templates, are all equally useful in a world dominated by Javascript front-end frameworks.
• Data source architectural patterns: how to efficiently architecture your application depending on the data source that is residing behind it.

4 – Enterprise Integration Patterns by Gregor Hohpe

Even if you are working for startups, it is very unlikely that you will write programs as standalone tools, without any dependencies to other applications or without even communicating with them.

Applications do exchange data, they share information and they need to communicate in reliable ways.

Think about it, if you are withdrawing money at an ATM, how many different servers and databases will be contacted for every operation that you perform?

Probably a lot. And it needs to be fast and secure.

Those are the concepts taught in the book :

• What messaging patterns are and how they help to solve issues that were described right above;
• How to design messaging systems properly;
• An extensive list of individual messaging components (content-based router for example) that helps you build a complete architecture tailored to your needs;
• Real-life examples of how a banking system for example would actually be designed.

With this book, you will definitely get to know more about the capabilities of what we call an engineering architect or an entreprise architect. 

Do you even own the book? I have my very own version of it!👽

Tip: for some of my interviews, I actually got asked questions related to concepts described in this book, especially how to handle system reliability in case of failure.

Probably one of the best software engineering books when it comes to system design.

5 – The Mythical Man-Month by Frederick Brooks

If you are following the project management path of your engineering carrier, this is probably the book you should read.

The Mythical Man-Month discusses productivity, essentially tackling one of the myths that the time taken by one engineer can be equally divided if you hire more engineers to do the job.

This is of course false, and Frederick Brooks explains several project management concepts related to this myth :

• Silver bullet concept: stating that there are no project management techniques able to solve current inherent problems of software development;
• How to handle delay in project delivery and what role projects owners have to endorse when it comes to their clients;
• How to communicate efficiently as a project leader, and what your team expects from you;
• Most importantly, how to manage project iteration and how to prevent the “second-system” effect.

In software engineering, even with the best developers, most of the project success relies on being able to manage your team efficiently.

Project management is a whole different skill set, and you are trying to succeed in this field, this is probably the book you should read.

This project management masterpiece is available right here.

6 – Code Complete by Steve McConnell

This book is seen as one of the references for software developers as it teaches all the basics that you should know in this field.

This is a very lengthy book, as it goes over 900 pages and sometimes in a lot of details.

With this book, you will cover :

• How to code and how to debug: including how to write programs for people first, and for computers second;
• Divide your code in terms of domains: the design of a high-level program is very different from the design (and implementation) of a low-level program;
• Master human qualities of top coders: this is very big in an industry where everybody thinks it has the ultimate answer to a question. Build humility, curiosity, but most importantly, keep your ego in check;
• Pick a process and stick to it: from the planning to the development, until the delivery, pick a process that guarantees project quality and prosperity.

7 – Git for Teams by Emma Hogbin Westby

For the seventh book, I chose a book about Git, the most used version control software in the world.

Why did I put this book in the list?

Because I believe that there can’t be a successful project without using version control, or without defining a good version control workflow.

If you are working alone, you may have not encountered issues that come with multiple people working on the same codebase at the same time.

However, without a proper workflow, the codebase can become quite a mess, and there is a very high chance that you will experience regressions.

This book teaches:

• What git is and how to use the different commands efficiently.
• How to define a custom git workflow for your team, given its size and what your project is about.
• How to conduct code reviews and why they matter in software integration.
• How to pick the best branching strategy for your team
• How to define roles in your team, who should be a contributor, a reviewer, who manages the codebase, and so on.

Do you need a guide on how to conduct a code review? Here are the 6 best code review tips for you to know.

8 – Refactoring: Improving the Design of Existing Code by Martin Fowler

As a software engineer, you spend a lot of time writing code and thinking about new algorithms in order to achieve your expected goal.

However, as your project grows, your codebase becomes larger and larger, you often find yourself writing duplicate functions, or having code parts that are very similar to one another.

As your project grows, you often feel like you are missing some points on function reusability and factorization. 

Refactoring by Martin Fowler is a book that helps you synthesizing and factorizing your codebase.

The book is built on study cases, focusing on seventy different refactoring cases.

On those seventy refactoring cases, Martin Fowler describes how to perform them properly, in a safe way for the code base, as well as the role of unit testing in refactoring.

9 – The Art of Unit Testing by Roy Osherove

A software engineering book list would not be complete without a book focused on unit testing.

Unit testing is not important, it is crucial and essential if you want to deliver a good and qualitative piece of software to the end-user.

Not every functionality or line of code has to be tested, but you have to provide a reasonable amount of unit tests for crucial parts of your codebase.

Unit tests save lives.

When your codebase is rather small, you can’t foresee the immediate benefits of having extensive unit test coverage.

However, as your codebase grows, sometimes you may want to tweak a small and harmless part of your code.

Harmless? Never. I speak from experience, even when I could swear that my modifications had no impacts on the software, in reality, they had huge impacts on existing functionalities.

The Art of Unit Testing provides core competencies on how to unit test, how to scope it, and what to unit test.

The chapters focus on :

• What are the basics of unit testing, and how it differs from integration testing;
• What are stubs and mocks in unit testing frameworks;
• How to write loosely coupled unit tests in terms of dependencies;
• Understanding isolation frameworks extensively;
• How to work with legacy code from a testing perspective

Unit testing is crucial, and this is probably all you need to know to get your copy.

10 – Soft Skills: The Software Developer’s Life Manual by John Sonmez

I have followed John Sonmez from simpleprogrammer.com for a long time, and I respect John as an authoritative figure when it comes to soft skills designed for software engineers.

In a software engineering career, you spend most of your time coding, designing, and building software.

But as your responsibilities grow, you are sometimes given the opportunity to interact with clients, to gather their needs, or to actually showcase your advancement on its project delivery.

Interaction often means social skills, the ability to speak with confidence, the ability to choose the correct language given your audience, or the ability to negotiate.

Software engineering isn’t only about coding, it is also about maintaining a good work-life balance, having hobbies, exercising often, and eating properly.

Jon Sonmez helps you find and keep the right balance that you need to be an efficient and creative engineer, for a long time.

The books focus on:

• Productivity tips: how to build the right habits for you to triple down your productivity;
• Self-marketing tips: essentially how to sell yourself and how to increase your own perceived value;
• Fitness tips: how working out correlates with a good and healthy software engineering career, how it can benefit you on a daily basis;
• Financial advice: John explains how you can analyze your paycheck and make the best investments out of it.

Software engineering is not only about coding, get to know how to be more productive and have a great work-life balance.

Conclusion

Time to read and Time to practice are the best times to gain any knowledge you want.

Before ending this tutorial, there is one point that I want to make very clear when it comes to all the books.

True mastery comes from a reasonable amount of theory, and a tremendous amount of practice.

When practicing, you will get ten times more familiar with the concepts that you are reading about, and there are really no shortcuts to mastery in software engineering.

One of the greatest ways to keep learning when you are not at work, work on side projects!

Be patient, be humble, but also be confident that given the time, you will become a software engineer that delivers tools that really help people.

Experience is not taught in books. 

Until then, have fun, as always.