Nick Cook

Finding all monitoring platforms installation on Docker is very tricky but we are yet with this magical tutorial where you all gonna learn How To Install InfluxDB Telegraf and Grafana on Docker. Not only installation, configuration, and setup but also basics about three platforms such as InfluxDB Telegraf and Grafana.

InfluxDB is one of the most used time-series databases nowadays and was created in 2013 by InfluxData. Telegraf is a plugin-driven server agent for collecting and sending metrics and events from databases, systems, and IoT sensors. Grafana is a free and open-source (FOSS/OSS) visualization tool utilized on top of a variety of diverse data stores but is most commonly used together with Graphite, InfluxDB, Prometheus, and Elasticsearch

On the other hand, Docker is a virtualization environment that provides an easy way to create, manage and delete containers on the fly.

• How To Setup Telegraf InfluxDB and Grafana on Linux | TIG Stack Setup on Linux
• InfluxDays London 2021 Recap | Key Facts of InfluxDays London 2021
• How To Install InfluxDB 1.7 and 2.0 on Linux in 2021

All these were the most widely used for monitoring and dashboarding in the DevOps industry by many successful companies in the world. If you are seeking help to install any one of these tools on docker then follow this tutorial till to the end and make installation of InfluxDB Telegraf and Grafana on Docker easily. 

Note: InfluxDB is currently shifting to Influx 2.0, providing a single platform to manage all the components of the TICK stack. As a result, another tutorial will be available for InfluxDB 2.0.

• Prerequisites
• Installing InfluxDB 1.7.x on Docker
  • a – Prepare InfluxDB 1.7.x for Docker
  • b – Verifying your InfluxDB configuration for Docker
  • Running the InfluxDB container on Docker
  • Testing your InfluxDB container
  • Enabling authentication on InfluxDB for Docker
  • Create an administrator account with docker exec
• Installing Telegraf on Docker
  • Prepare Telegraf for InfluxDB and Docker
  • Modify your Telegraf configuration file
  • Running the Telegraf container on Docker
  • Visualizing Telegraf metrics in Grafana
• Installing Grafana on Docker
  • Configuring Grafana for InfluxDB
  • Importing a Grafana dashboard
• How to setup telegraph, influxdb, and grafana with Docker compose?
• Conclusion

Prerequisites

The requirements that you should do prior to install InfluxDB Telegraf and Grafana on Docker are as follows:

• Sudo privileges
• Installation & Configuration of Docker Correctly
• Designing the network strategy for InfluxDB

Sudo privileges

First of all, you need to have sudo rights on your Linux machine, otherwise, you won’t be able to install InfluxDB on your host.

To verify it, run the following command

$ sudo -v

If no error messages are shown on your terminal, you are good to go.

Next, you want to make sure that Docker is correctly installed on your system.

Installation & Configuration of Docker Correctly

To install Docker on Ubuntu and Debian, you can follow this tutorial. It provides great and extensive details on how to correctly set up Docker on Linux.

Again, to verify that Docker is correctly installed, you can run the following command.

$ docker --version
Docker version 19.03.1, build 74b1e89

Now that Docker is ready, let’s have a quick look at the networking strategy we are going to use for our containers.

Also Check: 

• How To Install Prometheus with Docker on Ubuntu 18.04
• How To Install Docker on Windows 7/8/10 Home and Pro

Designing the network strategy for InfluxDB

Before starting, it is important to have a few words about networking.

By default, containers are created to run on the bridge network stack.

By default, you will install InfluxDB, which will expose useful ports (like the 8086 one) to your current network stack.

Later on, you will bind Telegraf to it, but Telegraf does not have to expose any ports to your current host stack.

As a consequence, we can run InfluxDB on the default bridge network, and have Telegraf running in the same stack as InfluxDB.

Later on, we will add Grafana to our bridge network in order to visualize metrics gathered by Telegraf

Now that we have seen the network strategy we are going to use, let’s install the InfluxDB container for Docker.

To install InfluxDB on Docker, you have two ways of doing it.

You can either prepare your filesystem manually and run the InfluxDB on a Docker container with no initialization scripts.

This is the simplest way of initializing InfluxDB.

This method should be used if you plan on running InfluxDB on a single instance, if your initial InfluxDB configuration is very simple, or if you prefer to have full control over your containers.

However, there is a way to initialize InfluxDB with scripts (either bash scripts, or InfluxQL scripts).

This is the version that you should use if you are automating a lot of servers with InfluxDB (with Chef or Puppet for example), and you want to have the same initial setup on all your instances.

Installing InfluxDB 1.7.x on Docker

https://www.youtube.com/watch?v=0CpHwszFjUY

The official InfluxDB image for Docker is named: influxdb.

It is part of the Docker Official Images, so you can check that you are running an official version of InfluxDB on your system.

Moreover, the other tools of the TICK stack (Telegraf, InfluxDB, Chronograf, and Kapacitor) belong to the Docker Official Images.

The InfluxDB image is going to install the InfluxDB server responsible for storing time-series metrics on your system.

If you are familiar with Docker, you already know that you can map volumes from your local filesystem to your container in order to manipulate data easier in your container.

This is exactly what we are going to do in this tutorial.

Configuration files, as well as directories storing actual data, will be stored on our local filesystem.

a – Prepare InfluxDB 1.7.x for Docker

If you carefully followed the tutorial on setting up InfluxDB on Ubuntu, you know that you are going to create a specific user for your InfluxDB database.

$ sudo useradd -rs /bin/false influxdb

In your etc directory, create a new folder for your InfluxDB configuration files.

$ sudo mkdir -p /etc/influxdb

Creating a configuration file for InfluxDB and Docker

Luckily, you don’t have to create an InfluxDB configuration file by yourself.

To create an InfluxDB configuration file using Docker, run the following command.

docker run --rm influxdb influxd config | sudo tee /etc/influxdb/influxdb.conf > /dev/null

As a quick explanation, the “influxd config” command will print a full InfluxDB configuration file for you on the standard output (which is by default your shell)

As the –rm option is set, Docker will run a container in order to execute this command and the container will be deleted as soon as it exists.

Instead of having the configuration file printed on the standard output, it will be redirected to our InfluxDB configuration file.

Next, reassign the folder permissions for your newly created file, otherwise, your container won’t be able to interact with it properly.

$ sudo chown influxdb:influxdb /etc/influxdb/*

Creating a lib folder for InfluxDB and Docker

As stated in the documentation, InfluxDB stores its data, metadata as well as the WAL (for write-ahead log) in the /var/lib/influxdb folder by default.

As a consequence, you have to create this folder if it is not currently existing.

$ sudo mkdir -p /var/lib/influxdb

Again, make sure that the permissions are correctly set for your container to write into this folder.

$ sudo chown influxdb:influxdb /var/lib/influxdb

Now that our folders are ready, let’s see how we can initialize InfluxDB with custom scripts.

Preparing initialization scripts for InfluxDB on Docker (optional)

With the InfluxDB image, there is a way to automate database initialization on your containers.

As an example, we will instruct our Docker container to create an administrator account, a regular user account (for Telegraf), and a database with custom retention via a custom InfluxQL script.

Anatomy of the InfluxDB image

On container boot, the entrypoint.sh script is executed, it is set as the entrypoint of your Docker container.

The entry point can be executed in two ways.

You can execute the entry point script in order to launch a simple InfluxDB instance on your container.

This is for example what we have done in the previous section. We specified the configuration flag and it was used in order to set your InfluxDB server initialization.

However, there is a second way to execute the entrypoint script: by executing the init-influxdb script.

The init-influxdb script is made of two parts :

• First, it will watch for environments variables passed to your docker command, and it will execute commands accordingly
• Next, if you have a docker-entrypoint-initdb.d directory at the root directory of your container, it will execute either bash scripts or IQL scripts in it.

We are going to use that information to create our InfluxDB container.

First, make sure that no folders are already created in your /var/lib/influxdb folder.

$ ls -l /var/lib/influxdb
total 0

Execute the following command for the meta folder (in the influxdb folder) to be updated with the correct information.

As a reminder, we want an admin account and a regular account for Telegraf (named telegraf)

Creating initialization scripts on your host

In order for the initialization scripts to be run on initialization, they have to be mapped to the docker-entrypoint-initdb.d folder in your container.

First, create a scripts folder on your host wherever you want.

In my case, it is going to be created in /etc/influxdb.

$ sudo mkdir -p /etc/influxdb/scripts

Edit a new script file on your newly created folder, and make sure to give it a .iql extension.

$ sudo touch influxdb-init.iql

CREATE DATABASE weather;
CREATE RETENTION POLICY one_week ON weather DURATION 168h REPLICATION 1 DEFAULT;

This simple initialization script will create a database for weather data, and it will assign a one-week retention policy for it.

Great!

The last step will be to prepare our meta folder for InfluxDB initialization.

Creating/updating the InfluxDB Meta Database

In order to update your meta database, run the following command

$ docker run --rm -e INFLUXDB_HTTP_AUTH_ENABLED=true \
         -e INFLUXDB_ADMIN_USER=admin \
         -e INFLUXDB_ADMIN_PASSWORD=admin123 \
         -v /var/lib/influxdb:/var/lib/influxdb \
         -v /etc/influxdb/scripts:/docker-entrypoint-initdb.d \
         influxdb /init-influxdb.sh

Note: Setting the INFLUXDB_HTTP_AUTH_ENABLED to true does not mean that authentication is enabled on your InfluxDB server. Authentication is enabled in one of the next sections, this parameter is only used for the initialization script.

A couple of logs should be printed on your terminal.

If this is not the case, make sure that you specified the correct environments variables for your container.

If you chose to create initialization scripts for your container, you should also have a log line for it.

As the last verification step, you can inspect your meta.db file in your meta folder to make sure that the changes were correctly written.

$ cat /var/lib/influxdb/meta/meta.db | grep one_week

Now that your InfluxDB files are prepared, let’s head over to some configuration verifications.

b – Verifying your InfluxDB configuration for Docker

If you used the configuration command detailed in the section above, you should be presented with a simple configuration file in the /etc/influxdb folder.

Open your file and verify that everything is correct.

HTTP Interface

Head over to the [http] section of your configuration and make sure that it is enabled.

Verify the bind-address that should be 8086 by default.

This is the port that you are going to use to send some commands to your InfluxDB database, like creating a database or adding a user for example.

By default, authentication and encryption are disabled. However, sections of this tutorial explain how you can set up authentication in depth.

Data, meta, and WAL configurations

By default, your configuration file should have the paths that we created in the first section, so you don’t have to change anything.

However, you should check that your paths are correct.

[meta]
  dir = "/var/lib/influxdb/meta"

[data]
  dir = "/var/lib/influxdb/data"
  wal-dir = "/var/lib/influxdb/wal"

Running the InfluxDB container on Docker

We are going to use the InfluxDB image from the official Docker repositories.

As a quick reminder, you need to use the docker container run command in order to start a Docker container.

First, make sure that nothing is running on port 8086.

$ sudo netstat -tulpn | grep 8086

If you remember correctly, we configured our folders to be accessible by the InfluxDB user (belonging in the InfluxDB group).

As a consequence, we will need the user ID of the InfluxDB user in order to run our container.

To find the InfluxDB user ID, head over to the passwd file on your host and run

$ cat /etc/passwd | grep influxdb
influxdb:x:997:997::/var/lib/influxdb:/bin/false

As you can see, the user ID for my InfluxDB user is 997.

Note: The user ID will surely be different on your system, and you should modify it accordingly when running the docker command.

To start InfluxDB on Docker, run the following command.

docker run -d -p 8086:8086 --user 997:997 --name=influxdb \ 
-v /etc/influxdb/influxdb.conf:/etc/influxdb/influxdb.conf \ 
-v /var/lib/influxdb:/var/lib/influxdb \ 
influxdb \ 
-config /etc/influxdb/influxdb.conf

Testing your InfluxDB container

In order to test if your InfluxDB container is correctly running, you can check that the HTTP API is correctly enabled.

$ curl -G http://localhost:8086/query --data-urlencode "q=SHOW DATABASES"

You can also check that your InfluxDB server is correctly listening on port 8086 on your host.

$ netstat -tulpn | grep 8086
tcp6    0    0 :::8086      :::*       LISTEN       -

Awesome!

Your InfluxDB container is correctly running on Docker.

By default, your InfluxDB server does not contain any databases except for the _internal used, as its name describes, internal metrics about InfluxDB itself.

However, if you created initialization scripts for your InfluxDB database, make sure that your databases and retention policies are correctly assigned.

$ influx
Connected to http://localhost:8086 version 1.7.8
InfluxDB shell version: 1.7.7

> SHOW USERS

user    admin
----    -----
admin   true

> SHOW DATABASES
name: databases
name
----
weather

Enabling authentication on InfluxDB for Docker

In order to enable authentication for InfluxDB 1.7.x, you are going to create an administrator account for your InfluxDB database (if you didn’t use initialization scripts)

Create an administrator account with docker exec

You don’t have to create an administrator account if you initialized your InfluxDB image with environment variables in the previous sections.

This is only necessary if you choose a fully customized InfluxDB image that you configure yourself.

To create an administrator account, connect to a bash process in your container and run the influx utility by yourself.

To achieve that, run the following commands

$ docker container ls

Note: Your container is not appearing here? Run this command with the -a (for all) flag to make sure that your container hasn’t crashed.

Identify the container ID of your InfluxDB container, and run the following command to have a bash in your container.

$ docker exec -it <container_id> /bin/bash

As a reminder, the docker exec is used in order to run a command in a running container.

Here are the options specified with it :

• -i: for interactive, it will keep the standard input open even if not attached

• -t: to allocate a pseudo-TTY to your current shell environment.

Right now, you should have a shell prompt, similar to this :

In your container, run the influx utility to create your administrator account.

$ influx
Connected to http://localhost:8086 version 1.7.8
InfluxDB shell version: 1.7.8
> CREATE USER admin WITH PASSWORD 'admin123' WITH ALL PRIVILEGES
> SHOW USERS
user  admin
----  -----
admin true

Now that you have an administrator account, you can enable HTTP authentication for your database.

Enable HTTP Authentication in your configuration file

To achieve that, exit your container, and head to the configuration folder you created for InfluxDB.

Ctrl + D (to exit your container)

$ sudo nano /etc/influxdb/influxdb.conf

[http]
  enabled = true
  bind-address = ":8086"
  auth-enabled = true

Save your file and restart your container for the changes to be applied.

$ docker container restart <container_id>

To make sure that your changes are effective, try querying the HTTP API again.

You should be unable to execute a query without specifying the correct credentials.

$ curl -G http://localhost:8086/query --data-urlencode "q=SHOW DATABASES"
{"error":"unable to parse authentication credentials"}

Great!

Authentication is correctly enabled.

Let’s try to execute the InfluxQL query again with the correct credentials.

$ curl -G -u admin:admin123 http://localhost:8086/query --data-urlencode "q=SHOW DATABASES"

{"results":[{"statement_id":0,"series":[{"name":"databases","columns":["name"],"values":[["_internal"]]}]}]}

With this curl command, we made sure that our credentials were correctly set up for our InfluxDB server.

Now that your time-series database is up and running, it is time to install our metrics collection agent: Telegraf.

Installing Telegraf on Docker

For those who are not familiar with Telegraf, Telegraf is a plugin-driven agent that periodically collects metrics about a variety of different systems.

The metrics are pushed to InfluxDB and they can be later on analyzed in Chronograf or Grafana.

Luckily, Telegraf also belongs to the official Docker images.

In this section, we are going to configure Telegraf for it to gather system metrics on our Linux host (in this case, a Debian 10, but it works in the same way for Ubuntu-based ones)

Prepare Telegraf for InfluxDB and Docker

Likewise to our InfluxDB setup, we are going to create a Telegraf user for our host. It ensures that correct permissions are set for our future configuration files.

$ sudo useradd -rs /bin/false telegraf

In your etc directory, create a new folder for your Telegraf configuration files.

$ sudo mkdir -p /etc/telegraf

Creating a configuration file for Telegraf and Docker

Again, we don’t have to create a Telegraf configuration file by ourselves.

The Telegraf Docker image is built very close to the InfluxDB one.

As a consequence, it is able to run a simple telegraf config command to generate a configuration on the fly.

By default, the Telegraf configuration file has the following defaults:

• interval: 10 seconds. Telegraf is going to gather and send metrics to InfluxDB every 10 seconds.
• round_interval: true. The agent is going to collect metrics on :00, :10, or :(00 + n*interval)
• The InfluxDB output plugin is enabled by default.
• The CPU, disk, diskio, kernel, memory, processes, swap, and system inputs plugins are enabled. As those inputs use the /proc mountpoint to gather metrics, we will have to remap volumes on the container.

To create a Telegraf configuration file using Docker, run the following command.

docker run --rm telegraf telegraf config | sudo tee /etc/telegraf/telegraf.conf > /dev/null

Next, reassign the correct permissions to your Telegraf configuration folder.

This will ensure that only Telegraf itself and the root account are able to write to the configuration file.

$ sudo chown telegraf:telegraf /etc/telegraf/*

Modify your Telegraf configuration file

With Telegraf, most of the time, you will want to send metrics directly to InfluxDB itself.

This is why the InfluxDB output is enabled by default in your Telegraf configuration file.

By default, Telegraf will send metrics to a database named “telegraf” on InfluxDB.

This is a customizable parameter, however in this case we are only going to specify the InfluxDB authentication parameters.

Edit your Telegraf configuration file, and locate the [[outputs.influxdb]] section.

In this configuration file, locate the “HTTP Basic Auth” section and modify the credentials accordingly.

## HTTP Basic Auth
username = "admin"
password = "admin123"

You are of course free to create a dedicated administrator account for Telegraf by using the method we described above (using docker exec)

Save and exit your file, now it is time to run the container.

Running the Telegraf container on Docker

As stated before, Telegraf enables system inputs by default.

As a consequence, we will have to remap the /proc host folder to the /host folder on our Docker image.

This is made to ensure that Telegraf is not gathering metrics from the Docker container itself and that the container filesystem is not altered in any way.

To achieve this, unless you gave a name to your InfluxDB container, run this command to get your InfluxDB container ID.

It will be used to connect Telegraf and InfluxDB to the same virtual network.

$ docker container ls | grep influxdb
1939ba611410   influxdb   "/entrypoint.sh -conf..."   24 minutes ago    Up 30 minutes    0.0.0.0:8086->8086/tcp    ecstatic_moore

Isolate your Telegraf user ID by running the following command.

$ getent passwd | grep telegraf
telegraf:x:998:998::/etc/telegraf:/bin/false

Next, to run the Telegraf Docker image, run the following command.

$ docker run -d --user 998:998 --name=telegraf \
      --net=container:<influx_container_id> \
      -e HOST_PROC=/host/proc \
      -v /proc:/host/proc:ro \
      -v /etc/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf:ro \
      telegraf

Note: The net option can be replaced by –net=influxdb if you chose to create your InfluxDB container with a name.

Great!

To make sure your Telegraf instance is running correctly, run the following command.

$ docker container logs -f --since 10m telegraf

Telegraf seems no to raise any error messages, but let’s double-check the correctness of our setup by inspecting the InfluxDB database.

$ docker exec -it <container_id> influx -username admin -password admin123

InfluxDB shell version 1.7.8
> SHOW DATABASES

name: databases
name
----
weather
_internal
telegraf

> USE telegraf
> SELECT * FROM cpu WHERE time < now() - 1m

If you are seeing data points, congratulations!

Your Telegraf instance is correctly sending metrics to your InfluxDB server.

Now that all metrics are stored in Telegraf, for one week, we can install a modern dashboarding tool in order to visualize them: Grafana.

Visualizing Telegraf metrics in Grafana

For those who are unfamiliar with Grafana, Grafana is a dashboarding tool that binds to a wide variety of different datasources in order to create beautiful graphs.

It can bind to traditional SQL databases, but it can also bind to time series databases, which is what we are going to do in this section.

Installing Grafana on Docker

The Grafana docker image is stored under the Grafana repository.

To create a Grafana container, run the following command on your host.

$ docker run -d --name=grafana -p 3000:3000 grafana/grafana

A Grafana server container should now be up and running on your host. To make sure of it, run the following command.

$ docker container ls | grep grafana

You can also make sure that it is correctly listening on port 3000.

$ netstat -tulpn | grep 3000

Configuring Grafana for InfluxDB

With your web browser, head over to

You should be redirected to the Grafana home page.

The default credentials for Grafana are admin/admin.

Immediately, you are asked to change your password. Choose a strong password and click on “Save“

You should now be redirected to the Grafana default Web UI.

Click on “Add data source” to add an InfluxDB datasource.

Next, select the InfluxDB option and click on “Select“.

For this tutorial, we are not using InfluxDB images as Docker services. As a consequence, we have to isolate InfluxDB public IP on our bridge network.

To do that, run the following command

$ docker network inspect bridge | grep influxdb -A 5
"Name": "influxdb",
"EndpointID": "7e4eb0574a346687efbb96b6b45",
"MacAddress": "02:42:ac:11:00:04",
"IPv4Address": "172.17.0.2/16",
"IPv6Address": ""

Copy the IPv4 address and paste it into the InfluxDB configuration for Grafana.

Select the Basic Auth option, specify your administrator credentials, and fill in the details about your InfluxDB database.

Here is the final configuration.

Click on “Save and Test” to make sure that your configuration is working properly.

Awesome!

Now that everything is set up in Grafana, let’s import a dashboard designed for Telegraf host metrics.

Importing a Grafana dashboard

To import a Grafana dashboard, select the “Plus” icon in the left menu, and click on “Import“.

In the import text box, put 1443 as a dashboard ID.

In the next window, make sure that the information is correct and bind it to your newly created datasource.

Click on “Import” and your dashboard should be up and running!

Do Check Some Other Grafana Tutorials from here: 

• How To Create a Grafana Dashboard? (UI + API methods)
• MongoDB Monitoring with Grafana & Prometheus
• Complete MySQL dashboard with Grafana & Prometheus

How to setup telegraph, influxdb, and grafana with Docker compose?

Conclusion

Congratulations, you learned how to install InfluxDB 1.7.x, Telegraf and Grafana using Docker.

As you can realize that this tutorial concentrates on a fully customizable installation of your images. In case you want to automate your container setups, it might be a good idea to use docker-compose.

Furthermore, ensure to read through the entire list of Telegraf input plugins to start getting ideas about what to monitor. Until then, have fun & enjoy the work with discovering our docker tutorials, as always.

InfluxDays London is an annual convention held by InfluxData on the 13th and 14th of June 2019, where you get to know some general insights into the recent developments of the company and the industry by InfluxDays.

Not only recent development insights but also highlights various guest speakers whose feedbacks simply regarding real-world use cases and reality checks. In 2019, this convention was hosted in Brewery, London. This was specialized in hosting corporate venues in the Finsbury district of London.

Given the very fast pace of changes in the industry with new languages, new APIs, new partnerships, and a complete shift from 1.x to 2.x, expectations were high.

• How to Check Python Version
• How To Install InfluxDB 1.7 and 2.0 on Linux in 2021
• How to Install Eclipse IDE on Ubuntu 18.04

I wanted to have an appropriate section for the recent industry collaborations and a community chapter to end this recap.

So, in this tutorial, you will be going to have an entire look at all the latest & innovative features declared for the future of InfluxDB. Also, you guys can keep your eye on What are the key facts that were announced and what’s in the future for developers? from here.

Without wasting another single minute, let’s check out some key facts of InfluxDays London 2021 Recap prevailing below:

• From InfluxDB 1.x to InfluxDB 2.x
  • a – InfluxDB as a single platform
  • b – Emphasis on Flux
  • c – Tasks
    • What about alerts and notifications?
  • d – Giraffe and Clockface
    • Giraffe
    • Clockface
  • e – User Packages
• Recent Industry Collaborations
  • a – Grafana
  • b – Prometheus & Flux Transpilation
• Real World Use Cases
  • a – Playtech
  • b – Worldsensing
• Community
  • a – InfluxAces
  • b – Community Links
• A Closing Word

I – From InfluxDB 1.x to InfluxDB 2.x

If you have been working with InfluxDB for quite some time, you already know about the TICK stack, composed of Telegraf, InfluxDB, Chronograf, and Kapacitor.

All those components cover a very specific use case, whether it is about visualization with Chronograf, stream processing and analysis with Kapacitor, or metrics gathering using Telegraf.

However, some of those components showed some limitations, in addition to the inability of InfluxQL to scale up besides being a great ramp for the platform at the beginning.

That’s the reason why InfluxData invested in InfluxDB 2.0, unveiling the Flux language and revamping the whole platform thus inherently putting an end to the TICK stack as we used to know it.

The announcements made in InfluxDays London confirmed the transition.

a – InfluxDB as a single platform

Influx is much more than just InfluxDB. Even if the product is called InfluxDB 2.0, the single UI platform that InfluxData is building is doing way more than that.

Right now, Chronograf is integrated directly into the InfluxDB 2.0 platform. You have a real coexistence of the two products and you don’t have to rely on two different products to visualize data anymore.

As you can see, InfluxDB is stated to be a time series database visualization (Chronograf aspect of it), query, and task engine (tasks will be explained a little bit later).

Having a single platform for everything is interesting.

Developers are often hesitant about installing many different tools as it means that it will create maintenance costs for every single tool in the stack.

With two different products and Telegraf remaining an entity on its own, the data visualization and manipulation are done in a single place.

Opinion

Having a single platform for pretty much everything is definitely interesting.

It is definitely a great UI/UX challenge as it can be hard to design tools that are functionally that ambitious.

However, from the live demonstrations that I saw on Friday (the workshop day), the process seems smooth enough.

InfluxData definitely delivers on the “Time To Awesome” feature. Besides being a great marketing catchword, it is actually reinforced by the predefined templates (for your system, a Docker or a Kubernetes instance) that can create a complete dashboard in seconds.

Links

InfluxDB 2.0 documentation

InfluxDB 2.0 Announcements and Future Ahead – by Paul Dix

b – Emphasis on Flux

As stated before, Flux is a new data scripting and query language built by InfluxData.

Flux comes as a replacement of InfluxQL and provides an API, built on the concepts of piping, i.e sequencing, operations in order to manipulate data. You can for example send data to third-party services, pivot it or perform join operations on it.

For those interested, I wrote an article on the difference between Flux and SQL.

But Flux isn’t only a language. It has also a query planner, a query optimizer as well as a CLI (or a REPL) that one can use in order to directly test Flux functions.

The major evolution here is that Flux is seen as a first-class citizen, and not a subpart of InfluxDB itself.

In a way, Flux should exist on its own.

InfluxData, via its CEO voice Paul Dix, clearly stated that they want Flux to integrate with other systems and that other languages might be transpiled to Flux in the future.

Maybe we could see an efficient transpilation from SQL to Flux in the future? Opinions diverge.

Opinion

For those following the blog, you know that I advocate technologies that are designed to reach the widest possible audience. Languages should not be created for elite programmers especially when they are aimed at data manipulation and analysis.

Flux is definitely designed that way.

With an expressive API, that one can easily understand, it should not be too hard for non-tech people to manipulate it and bring value easily.

Even if Flux is theoretically impressive, and well documented, I believe that its potential will be confirmed by real use cases and popularized by developers becoming way more productive than with other languages.

c – Tasks

In short, tasks are scheduled jobs that one can run in order to manipulate, transform or transfer data.

They can be seen as cron jobs following a very special syntax close to JSON.

The goal here is to have small units that perform tasks on their own.

The task contains Flux scripts and should be composable to perform multiple operations.

Administrators are able to create tasks templates that can be used by all the users afterward. I believe this is a great point when you want to reuse some of the work you have done previously.

Tasks can be managed via the InfluxDB UI, the CLI, and with the new API that InfluxData is building very specifically for tasks.

This API will provide a set of endpoints (such as /create or /delete) for you to easily perform some administrative operations on your tasks.

Tasks are versioned, so you could theoretically go back to a specific version if your new task isn’t satisfying enough.

Finally, you have tasks runs and tasks logs directly in the interface to see when your task was run and if it was run successfully.

What about alerts and notifications?

Alerts and notifications channels are still a very big part of InfluxDB 2.0. Special APIs were created for alerts and notifications and you should theoretically be able to create, modify or delete alert rules easily.

Opinion

In my opinion, the real challenge with tasks is about defining the best practices around them.

Having tasks that perform small operations is interesting for debugging purposes. Also, with runs and logs, you can have feedback on how your actions perform.

However, having small and independent units can lead to a very large pool of tasks to maintain, as well as code duplication probably leading to bug duplication.

There is a big challenge for developers to create an efficient data flow that is not too convoluted. Tasks are probably easy to create, but they should not be seen as the ultimate option to perform every single operation on your database.

For example, do you find tasks adapted for data cleansing? Wouldn’t it be more adapted to perform it before inserting your data into the platform?

Do’s and don’ts will definitely be needed.

d – Giraffe and Clockface

Giraffe and Clockface are two products that were announced in InfluxDays 2021 London.

Right now, InfluxDB 2.0 provides a wide panel of visualizations for your metrics.

Graphs, scatter plots, single stats, histograms, and many more are provided by default in the interface.

However, InfluxData wants developers to be able to build their own visualizations.

Following the recent moves made by Grafana, moving from Angular panels to React panels, InfluxData created two libraries that allow developers to build their own panels and share them.

Giraffe

Giraffe is a library that is leveraging React components to provide a complete API for developers to use.

Pretty much like Grafana plugins, we can imagine that InfluxData is going to build a place for people to share their panels.

For now, the library is still in the pre-alpha stage, but I’ll share more info about it as soon as it becomes available to the public.

Clockface

At first, I had some trouble understanding the actual difference between Giraffe and Clockface.

Giraffe is designed for Chronograf-like panels and Clockface is designed to build visualization applications.

In short, it allows you to tweak the existing interface to add actions that are not existing natively in the platform.

Let’s say that you want to create a button on InfluxDB 2.0 that hides a certain panel that you find not that useful.

You would do it using Clockface, creating a button, and performing some Javascript operations to hide the panel.

e – User Packages

User packages are another big part of the next features of InfluxDB 2.0. Pretty much like NPM packages, Influx packages are meant to allow developments of any kind of logic into the platform without having to perform a pull request.

Proposals were made for interacting with Influx packages, such as a CLI with custom functions such as:

> influx package init
> influx package publish

You can define types and rules inside a configuration file that is very similar to a package.json file in Node ecosystems.

Warning: Influx user packages are different from Flux packages.

Flux packages are used to import Flux functions that were already coded and that you would want to use in a new Flux

II – Recent Industry Collaborations

In this industry, I believe that collaboration is one of the ways to succeed.

Most of the existing products (at least Grafana, InfluxDB, and Prometheus) are working towards improving the interoperability of their respective products.

What do we mean by interoperability?

Even if those tools are not accepting the same metrics format, and not dealing with the same APIs, there is a work in progress in accepting various different formats in InfluxDB.

a – Grafana

Even if InfluxDB 2.0 definitely intersects some of the features of Grafana, Grafana still provides new ways to monitor your InfluxDB metrics.

Both InfluxDB and Grafana are working on data exploration features.

Grafana wants to become an observability platform. It means that Grafana can be used to monitor real-time data, but it also can be used to have a closer look at certain metrics, and even compare them with metrics from other buckets or databases.

b – Prometheus & Flux Transpilation

We know that Prometheus and Flux work quite differently when it comes to data querying.

As a reminder, I explained how Prometheus querying via PromQL worked in my Prometheus guide.

One of the news that was unveiled at InfluxDays London 2021 was the transpiling work done from Prometheus to Flux.

The goal of this project is to have InfluxDB as long-term storage for Prometheus. Prometheus queries would be run against this transpiler to produce Flux code.

Flux code would then be interpreted by the dedicated Flux interpreter and run on the Flux engine.

Julius Volz revealed his advancements on this project, explaining how he designed the data mapper (with metric names as fields, the measurement being ‘Prometheus’, label names being column names, and so on).

He also explained how he used abstract syntax trees to map PromQL functions to a Flux pipeline.

Remarks: there are still some concerns regarding the speed performances of the transpiling process.

InfluxData and Prometheus are working on a way to make joint operations more efficient, thus inducing faster transpiling operations.

Do Check Related Articles on Grafana & Prometheus: 

• How To Create a Grafana Dashboard? (UI + API methods)
• MongoDB Monitoring with Grafana & Prometheus
• Complete MySQL dashboard with Grafana & Prometheus

III – Real World Use Cases

During InfluxDays London, many of the talks were done by guest speakers. Guests came from very different industries and showcased on they used InfluxDB for their own needs.

This is in my opinion the most interesting part of the convention, as it featured many different use cases that highlighted how InfluxDB is solving problems in the real world.

a – Playtech

This talk was presented by Alex Tavgen, technical architect for Playtech which is a popular gambling software development company.

Alex Tavgen unveiled his views on architecture monitoring solutions and provided very concrete examples of how they should be designed.

His talk was centered around the following points :

• “Managing a zoo” is hard: when developers are free to choose their stacks, it often leads to having teams with many different stacks which leads to obvious maintenance issues;
• Abusive use of dashboards: it is tempting to record everything, just in case. More is not better in monitoring, and metrics should not be monitored for the sake of it. We need to bring actual sense in monitoring;
• Are dashboards useless? No, but humans need to know what they have to look for in a dashboard. Dashboarding needs guidance and documentation;
• Anomaly detection to preserve client trust: you should tell the client that he has an issue, and not the other way around. Many investments were made to improve actual anomaly detection, not just outlier detection. Alex Tavgen explained how he used machine learning to create a predictive engine that raises alerts when it truly matters. (they also plan to open source their Java engine).

b – Worldsensing

This talk was presented by Albert Zaragoza, CTO and Head of Engineering at WorldSensing.Worldsensing is a company that provides monitoring solutions and intelligent components, leveraging IoT and time series databases to build smart cities. They provide intelligent parking solutions, as well as traffic flow solutions among other products.

Worldsensing produces client-facing applications, providing real-time feedback with extensive use of monitoring dashboards. As part of their infrastructure, Worldsensing uses Kapacitor and Flux in order to provide fast and reliable information to their end clients.

I will provide a link to the video as soon as it is available. However, if you use Postgres databases or if you are interested in providing Spatio-temporal feedback to your clients, you should definitely check what they are doing at Worldsensing.

IV – Community

InfluxDays London 2021 was also a great moment for us to talk and share our experiences with InfluxDB and the other products of its ecosystem.

a – InfluxAces

InfluxAces share their knowledge in a variety of ways, online and offline, through blogging, podcasting, attending meetups, answering community questions, building courses.. the list goes on!

And I am part of them now! Thank you!

As part of the InfluxAces program, I want to encourage creators and builders to submit their original articles to us as long as it is related to the subjects detailed in the article, and of course to InfluxDB.On my side, I will keep on writing about those technologies because I believe that they will shape the world of tomorrow.

b – Community Links

Meetups and conventions are great to connect with people. But they are not the only way to do it.

If you are interested in InfluxDB, you should:

• Join the Slack community available here.
• Ask your questions in the community forum (and have your questions answered by MarcV)
• Tweet to or about InfluxDB on Twitter.

A Closing Word

On a technological level, it was truly inspiring to see how fast those technologies change and evolve in such a short amount of time.

The progress made so far is huge and it looks very promising for the future.

At InfluxDays London, I was lucky to be able to connect with passionate and humble people from all over the world. I was able to share my views on the future and to listen to what the others had to say about it.

Sure, I learned more about InfluxDB, but I learned even more about the people behind the scenes, whether they are working for InfluxData or not.

Engineers, developers, managers, or just curious individuals, all share the same ambition to create, learn and give back to the community.

The open-source community is definitely an amazing community.

See you next year!

Elasticsearch is a platform for distributed search and data analysis in real-time. It offers a multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents with a simple installation.

By Elasticsearch, you can execute and combine various types of searches giving you the like Kibana, Logstash, X-Pack, etc., Elasticsearch can collect and monitor Big Data at a large scale. This Elasticsearch cluster includes three data nodes and with this, we could avoid a split-brain and have a quorum of master-eligible nodes.

• Monitoring Linux Logs with Kibana and Rsyslog | Using Kibana and Rsyslog to monitor Linux logs
• How To Install Logstash on Ubuntu 18.04 and Debian 9 | Tutorial on Logstash Configuration
• 4 Best Open Source Dashboard Monitoring Tools In 2021

In this free and ultimate tutorial, we will be going to learn how to install and configure a 3 Node Elasticsearch Cluster on Ubuntu 18.04 and with this you can go through some API examples on creating indexes, ingesting documents, searches, etc.

• What is ElasticSearch?
• Basic Concepts of Elasticsearch
• Note on Master Elections
• Prerequisites
• Installing Elasticsearch on Ubuntu
• Configuring Elasticsearch
• Start Elasticsearch
• Using Elasticsearch Restful API
• View Cluster Health
• View the Number of Nodes in your Cluster
• Create your first Elasticsearch Index
• Replicating a Yellow Cluster Status
• Ingest Data into Elasticsearch
• Search Queries
• Delete your Index

What is ElasticSearch?

ElasticSearch is a highly scalable open-source analytics engine, RESTful search engine built on top of Apache Lucene and issued under an Apache license. It is the most famous search engine and is generally used for full-text search, log analytics, security intelligence, business analytics, analyze big volumes of data faster and in near real-time, etc. Also, Elasticsearch is Java-based and can search and index document files in different formats.

Before we get to the main topic, let’s cover some basics about Elasticsearch from below: 

Basic Concepts of Elasticsearch

• An Elasticsearch Cluster is made up of a number of nodes;
• Each Node includes Indexes, where an Index is a Collection of Documents;
• Master nodes are subjective for Cluster related tasks, creating/deleting indexes, tracking of nodes, allocate shards to nodes;
• Data nodes are liable for hosting the actual shards that have the indexed data also handles data related operations like CRUD, search, and aggregations;
• Indexes are split into Multiple Shards;
• Shards exist of Primary Shards and Replica Shards;
• A Replica Shard is a Copy of a Primary Shard that is used for HA/Redundancy;
• Shards get placed on random nodes throughout the cluster;
• A Replica Shard will NEVER be on the same node as the Primary Shard’s associated shard-id.

Note on Master Elections

The least number of master eligible nodes that want to join a newly elected master in order for an election is configured via the setting discovery.zen.minimum_master_nodes. This configuration is very powerful, as it makes each master-eligible node informed of the minimum number of master-eligible nodes that must be visible in order to form a cluster.

Without this setting or incorrect configuration, this might lead to a split-brain, where let’s say something went wrong and upon nodes rejoining the cluster, it may form 2 different clusters, which we want to avoid at all costs.

From consulting elasticsearch documentation, to avoid a split brain, this setting should be set to a quorum of master-eligible nodes via the following formula:

(master_eligible_nodes / 2) + 1
# in our case:
(3/2) + 1 = 2

It is advised to evade having only two master eligible nodes since a quorum of two is two. To read more on elasticsearch cluster master election process, take a look at their documentation

Prerequisites

We have to set the internal IP addresses of our nodes to either our hosts’ file or DNS server. To keep it easy & straightforward, I will add them to my host file. This needs to apply to both nodes:

$ sudo su - 
$ cat > /etc/hosts << EOF
127.0.0.1 localhost
172.31.0.77 es-node-1
172.31.0.45 es-node-2
172.31.0.48 es-node-3
EOF

Now that our host entries are set, we can start with the fun stuff.

Installing Elasticsearch on Ubuntu

The following instructions and directions should be implemented to both nodes.

Get the Elasticsearch repositories and update your system so that your servers are aware of the newly added Elasticsearch repository:

$ apt update && apt upgrade -y
$ apt install software-properties-common python-software-properties apt-transport-https -y
$ wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
$ apt update

Elasticsearch relies on Java, so install the java development kit:

$ apt install default-jdk -y

Verify that java is installed:

$ java -version
openjdk version "11.0.3" 2019-04-16
OpenJDK Runtime Environment (build 11.0.3+7-Ubuntu-1ubuntu218.04.1)
OpenJDK 64-Bit Server VM (build 11.0.3+7-Ubuntu-1ubuntu218.04.1, mixed mode, sharing)

Install Elasticsearch:

$ apt install elasticsearch -y

Once Elasticsearch is installed, repeat these steps on the second node. Once that is done, move on to the configuration section.

Configuring Elasticsearch

For nodes to join the same cluster, they should all share the same cluster name.

We also need to specify the discovery hosts as the masters so that the nodes can be discoverable. Since we are installing a 3 node cluster, all nodes will contribute to a master and data node role.

Feel free to inspect the Elasticsearch cluster configuration, but I will be overwriting the default configuration with the config that I need.

Make sure to apply the configuration on both nodes:

$ cat > /etc/elasticsearch/elasticsearch.yml << EOF
cluster.name: es-cluster
node.name: \${HOSTNAME}
node.master: true
node.data: true
path.logs: /var/log/elasticsearch
path.data: /usr/share/elasticsearch/data
bootstrap.memory_lock: true
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.unicast.hosts: ["es-node-1", "es-node-2"]
EOF

Important settings for your elasticsearch cluster is described on their docs:

• Disable swapping
• Increase file descriptors
• Ensure sufficient virtual memory
• Ensure sufficient threads
• JVM DNS cache settings
• Temporary directory not mounted with noexec

Increase the file descriptors on the nodes, as instructed by the documentation:

$ cat > /etc/default/elasticsearch << EOF
ES_STARTUP_SLEEP_TIME=5
MAX_OPEN_FILES=65536
MAX_LOCKED_MEMORY=unlimited
EOF

Ensure that pages are not swapped out to disk by requesting the JVM to lock the heap in memory by setting LimitMEMLOCK=infinity.

Set the maximum file descriptor number for this process: LimitNOFILE and increase the number of threads using LimitNPROC:

$ vim /usr/lib/systemd/system/elasticsearch.service
[Service]
LimitMEMLOCK=infinity
LimitNOFILE=65535
LimitNPROC=4096
...

Increase the limit on the number of open files descriptors to user elasticsearch of 65536 or higher

$ cat > /etc/security/limits.conf << EOF
elasticsearch soft memlock unlimited
elasticsearch hard memlock unlimited
EOF

Increase the value of the map counts as elasticsearch uses maps directory to store its indices:

$ sysctl -w vm.max_map_count=262144

For a permanent setting, update vm.max_map_count in /etc/sysctl.conf and run:

$ sysctl -p /etc/sysctl.conf

Change the permissions of the elasticsearch data path, so that the elasticsearch user and group has permissions to read and write from the configured path:

$ chown -R elasticsearch:elasticsearch /usr/share/elasticsearch

Make sure that you have applied these steps to all the nodes before continuing.

Start Elasticsearch

Enable Elasticsearch on boot time and start the Elasticsearch service:

$ systemctl enable elasticsearch
$ systemctl start elasticsearch

Verify that Elasticsearch is running:

$ netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp6       0      0 :::9200                 :::*                    LISTEN      278/java
tcp6       0      0 :::9300                 :::*                    LISTEN      278/java

Using Elasticsearch Restful API

In this section we will get comfortable with using Elasticsearch API, by covering the following examples:

• Cluster Overview;
• How to view Node, Index, and Shard information;
• How to Ingest Data into Elasticsearch;
• Who to Search data in Elasticsearch;
• How to delete your Index

View Cluster Health

From any node, use an HTTP client such as curl to investigate the current health of the cluster by looking at the cluster API:

$ curl -XGET http://localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "es-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 0,
  "active_shards" : 0,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}

As you can see the cluster status is Green, which means everything works as expected.

In Elasticsearch you get Green, Yellow and Red statuses. Yellow would essentially mean that one or more replica shards are in an unassigned state. Red status means that some or all primary shards are unassigned which is really bad.

From this output, we can also see the number of data nodes, primary shards, unassigned shards, etc.

This is a good place to get an overall view of your Elasticsearch cluster’s health.

View the Number of Nodes in your Cluster

By looking at that /_cat/nodes API we can get information about our nodes that is part of our cluster:

$ curl -XGET http://localhost:9200/_cat/nodes?v
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.31.0.77              10          95   0    0.00    0.00     0.00 mdi       -      es-node-1
172.31.0.45              11          94   0    0.00    0.00     0.00 mdi       -      es-node-2
172.31.0.48              25          96   0    0.07    0.02     0.00 mdi       *      es-node-3

As you can see, we can see information about our nodes such as the JVM Heap, CPU, Load averages, node role of each node, and which node is master.

As we are not running dedicated masters, we can see that node es-node-3 got elected as master.

Create your first Elasticsearch Index

Note that when you create an index, the default primary shards are set to 5 and the default replica shard count is set to 1. You can change the replica shard count after an index has been created, but not the primary shard count as that you will need to set on index creation.

Let’s create an Elasticsearch index named myfirstindex:

$ curl -XPUT http://localhost:9200/myfirstindex
{"acknowledged":true,"shards_acknowledged":true,"index":"myfirstindex"}

Now that your index has been created, let’s have a look at the /_cat/indices API to get information about our indices:

$ curl -XGET http://localhost:9200/_cat/indices?v
health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   myfirstindex xSX9nOQJQ2qNIq4A6_0bTw   5   1          0            0      1.2kb           650b

From the output you will find that we have 5 primary shards and 1 replica shard, with 0 documents in our index and that our cluster is in a green state, meaning that our primary and replica shards have been assigned to the nodes in our cluster.

Note that a replica shard will NEVER reside on the same node as the primary shard for HA and Redundancy.

Let’s go a bit deeper and have a look at the shards, to see how our shards are distributed through our cluster, using the /_cat/shards API:

$ curl -XGET http://localhost:9200/_cat/shards?v
index        shard prirep state   docs store ip             node
myfirstindex 1     r      STARTED    0  230b 172.31.0.77    es-node-2
myfirstindex 1     p      STARTED    0  230b 172.31.0.48    es-node-3
myfirstindex 4     p      STARTED    0  230b 172.31.0.48    es-node-3
myfirstindex 4     r      STARTED    0  230b 172.31.0.77    es-node-1
myfirstindex 2     r      STARTED    0  230b 172.31.0.45    es-node-2
myfirstindex 2     p      STARTED    0  230b 172.31.0.77    es-node-1
myfirstindex 3     p      STARTED    0  230b 172.31.0.45    es-node-2
myfirstindex 3     r      STARTED    0  230b 172.31.0.48    es-node-3
myfirstindex 0     p      STARTED    0  230b 172.31.0.45    es-node-2
myfirstindex 0     r      STARTED    0  230b 172.31.0.77    es-node-1

As you can see each replica shard of its primary is spread on different nodes.

Replicating a Yellow Cluster Status

For a yellow cluster status, we know that it’s when one or more replica shards are in an unassigned state.

So let’s replicate that behavior by scaling our replica count to 3, which would mean that 5 replica shards will be in an unassigned state:

$ curl -XPUT -H 'Content-Type: application/json' \
http://localhost:9200/myfirstindex/_settings -d \
'{"number_of_replicas": 3}'

Now we have scaled the replica count to 3, but since we only have 3 nodes, we will have a yellow state cluster:

$ curl -XGET http://localhost:9200/_cat/indices?v
health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   myfirstindex xSX9nOQJQ2qNIq4A6_0bTw   5   3          0            0      3.3kb           1.1kb

The cluster health status should show the number of unassigned shards, and while they are unassigned we can verify that by looking at the shards API again:

$ curl -XGET http://localhost:9200/_cat/shards?v
index        shard prirep state      docs store ip             node
myfirstindex 1     r      STARTED       0  230b 172.31.0.45    es-node-2
myfirstindex 1     p      STARTED       0  230b 172.31.0.48    es-node-3
myfirstindex 1     r      STARTED       0  230b 172.31.0.77    es-node-1
myfirstindex 1     r      UNASSIGNED
myfirstindex 4     r      STARTED       0  230b 172.31.0.45    es-node-2
myfirstindex 4     p      STARTED       0  230b 172.31.0.48    es-node-3
myfirstindex 4     r      STARTED       0  230b 172.31.0.77    es-node-1
myfirstindex 4     r      UNASSIGNED
myfirstindex 2     r      STARTED       0  230b 172.31.0.45    es-node-2
myfirstindex 2     r      STARTED       0  230b 172.31.0.48    es-node-3
myfirstindex 2     p      STARTED       0  230b 172.31.0.77    es-node-1
myfirstindex 2     r      UNASSIGNED
myfirstindex 3     p      STARTED       0  230b 172.31.0.45    es-node-2
myfirstindex 3     r      STARTED       0  230b 172.31.0.48    es-node-3
myfirstindex 3     r      STARTED       0  230b 172.31.0.77    es-node-1
myfirstindex 3     r      UNASSIGNED
myfirstindex 0     p      STARTED       0  230b 172.31.0.45    es-node-2
myfirstindex 0     r      STARTED       0  230b 172.31.0.48    es-node-3
myfirstindex 0     r      STARTED       0  230b 172.31.0.77    es-node-1
myfirstindex 0     r      UNASSIGNED

At this point in time, we could either add another node to the cluster or scale the replication factor back to 1 to get the cluster health to green again.

I will scale it back down to a replication factor of 1:

$ curl -XPUT http://localhost:9200/myfirstindex/_settings -d '{"number_of_replicas": 1}'

Ingest Data into Elasticsearch

We will ingest 3 documents into our index, this will be a simple document consisting of a name, country and gender, for example:

{ 
  "name": "james", 
  "country": "south africa", 
  "gender": "male"
}

First, we will ingest the document using a PUT HTTP method, when using a PUT method, we need to specify the document ID.

PUT methods will be used to create or update a document. For creating:

$ curl -XPUT -H 'Content-Type: application/json' \
http://localhost:9200/myfirstindex/_doc/1 -d '
{"name":"james", "country":"south africa", "gender": "male"}'

Now you will find we have one index in our cluster:

$ curl -XGET http://localhost:9200/_cat/indices?v
health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   myfirstindex xSX9nOQJQ2qNIq4A6_0bTw   5   1          1            0     11.3kb          5.6kb

Since we know that the document ID is “1”, we can do a GET on the document ID to read the document from the index:

$ curl -XGET http://localhost:9200/myfirstindex/people/1?pretty
{
  "_index" : "myfirstindex",
  "_type" : "people",
  "_id" : "1",
  "found" : false
}

If we ingest documents with a POST request, Elasticsearch generates the document ID for us automatically. Let’s create 2 documents:

$ curl -XPOST -H 'Content-Type: application/json' \
http://localhost:9200/myfirstindex/_doc/ -d '
{"name": "kevin", "country": "new zealand", "gender": "male"}'

$ curl -XPOST -H 'Content-Type: application/json' \
http://localhost:9200/myfirstindex/_doc/ -d '
{"name": "sarah", "country": "ireland", "gender": "female"}'

When we have a look again at our index, we can see that we now have 3 documents in our index:

$ curl -XGET http://localhost:9200/_cat/indices?v
health status index        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   myfirstindex xSX9nOQJQ2qNIq4A6_0bTw   5   1          3            0       29kb         14.5kb

Search Queries

Now that we have 3 documents in our elasticsearch index, let’s explore the search APIs to get data from our index. First, let’s search for the keyword “sarah” as a source query parameter:

$ curl -XGET 'http://localhost:9200/myfirstindex/_search?q=sarah&pretty'
{
  "took" : 9,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "myfirstindex",
        "_type" : "_doc",
        "_id" : "cvU96GsBP0-G8XdN24s4",
        "_score" : 0.2876821,
        "_source" : {
          "name" : "sarah",
          "country" : "ireland",
          "gender" : "female"
        }
      }
    ]
  }
}

We can also narrow our search query down to a specific field, for example, show me all the documents with the name kevin:

$ curl -XGET 'http://localhost:9200/myfirstindex/_search?q=name:kevin&pretty'
{
...
  "hits" : {
    "total" : 1,
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "myfirstindex",
        "_type" : "_doc",
        "_id" : "gPU96GsBP0-G8XdNHoru",
        "_score" : 0.2876821,
        "_source" : {
          "name" : "kevin",
          "country" : "new zealand",
          "gender" : "male"
        }
      }
    ]
  }
}

With Elasticsearch we can also search with our query in the request body, a similar query as above would look like this:

$ curl -XPOST -H 'Content-Type: application/json' \
'http://localhost:9200/myfirstindex/_search?pretty' -d '
{
  "query": {
    "match": {
      "name": "kevin"
    }
  }
}'

{
...
        "_index" : "myfirstindex",
        "_source" : {
          "name" : "kevin",
          "country" : "new zealand",
          "gender" : "male"
        }
...
}

We can use wildcard queries:

$ curl -XPOST -H 'Content-Type: application/json' \
'http://172.31.0.77:9200/myfirstindex/_search?pretty' -d '
{
  "query": {
    "wildcard": {
      "country": "*land"
    }
  }
}'

{
...
    "hits" : [
      {
        "_index" : "myfirstindex",
        "_type" : "_doc",
        "_id" : "cvU96GsBP0-G8XdN24s4",
        "_score" : 1.0,
        "_source" : {
          "name" : "sarah",
          "country" : "ireland",
          "gender" : "female"
        }
      },
      {
        "_index" : "myfirstindex",
        "_type" : "_doc",
        "_id" : "gPU96GsBP0-G8XdNHoru",
        "_score" : 1.0,
        "_source" : {
          "name" : "kevin",
          "country" : "new zealand",
          "gender" : "male"
        }
      }
    ]
...
}

Have a look at their documentation for more information on the Search API

Delete your Index

To wrap this up, we will go ahead and delete our index:

$ curl -XDELETE http://localhost:9200/myfirstindex

Going Further

If this got you curious, then definitely have a look at this Elasticsearch Cheatsheet that I’ve put together and if you want to generate lots of data to ingest to your elasticsearch cluster, have a look at this python script.

Our other links related to ELK:

• How To Install Logstash on Ubuntu and Debian
• Monitoring Linux logs using Kibana and rsyslog