Prometheus: how to enable https schema

John Smith

• 3 min read
Prometheus: how to enable https schema
Photo by Kaffeebart / Unsplash

We all love Grafana and nice charts we can get with it. However, we rarely think about backends and how the data travels around. Internet was not designed with the security in mind, so http was a default protocol used (and it was a big breakthrough back then). Eventually, we have witnessed an evolution of http into https, which is basically a way to wrap traffic in secured tunnel until it reaches its destination. Today, https is de-facto is standard for traffic travel the public Internet.

Prometheus is Grafana's backend and is used to scrape metrics from various data sources. By default, Prometheus is using http schema, which is obsolete and not secure, so let's re-configure it to use https instead. Before we do it, a small disclaimer - I do not know how you installed Prometheus and this is important. In order not to break anything, I would expect you to more or less understand Linux bash command prompt and be able to amend the command below to match your setup.

This manual is based on this article.

  1. First, let's enable https on node_exporter. First, we will create a self-signed SSL certificate:
sudo mkdir /etc/node_exporter
sudo openssl genrsa -out /etc/node_exporter/node_exporter.key 2048
sudo openssl req -new -key /etc/node_exporter/node_exporter.key -out /etc/node_exporter/node_exporter.csr (this will prompt some question, feel free to answer them or to skip by hitting enter)
sudo openssl x509 -req -days 365 -in /etc/node_exporter/node_exporter.csr -signkey /etc/node_exporter/node_exporter.key -out /etc/node_exporter/node_exporter.crt
  1. Let's now create node_exporter config where we are going to indicate that we want https:
sudo nano /etc/node_exporter/config.yml
add the following text (be careful of indentations, this is yaml baby!):

tls_server_config:
  cert_file: node_exporter.crt
  key_file: node_exporter.key
  1. Now we need to amend our service file in order to tell node_exporter we are going to use https:
sudo nano /etc/systemd/system/node_exporter.service
in the ExecStart section we need to add the following variable:
--web.config.file="/etc/node_exporter/config.yml"
sudo systemctl daemon-reload
sudo systemctl restart node_exporter

This will enable https on node_exporter, which is good, have a cup of tea now to celebrate ☕.

  1. Let's move on to enable https on Prometheus:
you will need to copy node_exporter.crt we created in step 1 to your server with machine with Prometheus, I keep keys in /etc/prometheus. Once done, make sure you have got the correct permissions:

sudo chown prometheus:prometheus /etc/prometheus/node_exporter.crt
  1. Let's edit Prometheus's config to enable https:
sudo nano /etc/prometheus/prometheus.yml
add the following text:

scheme: https
tls_config:
  ca_file: /etc/prometheus/node_exporter.crt
  insecure_skip_verify: true
  1. Install apache-2-utils and hash your password:
sudo apt-get update && sudo apt install apache2-utils -y

htpasswd -nBC 12 "" | tr -d ':\n'

(make sure to correctly copy the hash only!)
  1. Add the following to config.yml
basic_auth_users:
    YOUR_USERNAME: <YOUR_PASSWORD>

change YOUR_USERNAME to username of your choice. You can add as many username as possible, one per line. Change YOU_PASSWORD to the hash from step 6.
  1. Restart node_exporter service
sudo systemctl restart node_exporter
  1. After than you can check your Prometheus's target website (IP:9099/targets) and in the Error section you should see 401 Unauthorized error code, this is fine. On you machine with Prometheus add the following to prometheus.yml:
basic_auth:
  username: <YOUR_USERNAME>
  password: <YOUR_PASSWORD>
  1. Restart Prometheus and you should be good to go:
sudo systemctl restart prometheus

Here is my prometheus.yml:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:

- job_name: "NYM Node"
  scheme: https
  tls_config:
    ca_file: /etc/prometheus/node_exporter.crt
    insecure_skip_verify: true
  basic_auth:
    username: something
    password: something
  static_configs:
    - targets: ["IP_OR_DOMAIN:9100"]

- job_name: "Localhost"
  scheme: https
  tls_config:
    ca_file: /etc/prometheus/node_exporter.crt
    insecure_skip_verify: true
  basic_auth:
    username: something
    password: something
  static_configs:
    - targets: ["IP_OR_DOMAIN:9100"]

You can add more jobs or have everything under one job name but add more targets.

Your Grafana should still be able to received data from Prometheus without any extra config, provided both Grafana and Prometheus are installed on the same machine.

Enjoy!