Consul-based Auto Discovery Mechanism for Prometheus Monitoring

Tue, 26 Sep 2023 00:00:00 +0000

Prometheus Comprehensive Monitoring Platform - Consul-Based Auto Discovery

Background

Consul Documentation | Consul | HashiCorp Developer

The Prometheus configuration file prometheus-config.yaml contains numerous scraping rules, which are mostly manually managed by DevOps teams. When new nodes or components are added, manual modification of this configuration and a hot reload of Prometheus are required. Is it possible to dynamically monitor microservices? Prometheus provides multiple dynamic service discovery mechanisms, and this guide uses Consul as an example.

Consul-Based Auto Discovery

Consul is a distributed key-value database and a service registry component. Other services can register with Consul, including Prometheus. By leveraging Consul’s service discovery, we can avoid manually specifying large numbers of targets in Prometheus.

The workflow for Prometheus’ Consul-based service discovery is as follows:

Register or deregister services (monitoring targets) in Consul.
Prometheus continuously monitors Consul. When changes to services meeting the criteria are detected, Prometheus updates its monitoring targets accordingly.

Service Discovery Mechanisms Supported by Prometheus

Prometheus configuration for data sources falls into two main categories: static configuration and dynamic discovery. Commonly used mechanisms include:

1) static_configs:       # Static service discovery  
2) file_sd_configs:      # File-based service discovery  
3) dns_sd_configs:       # DNS-based service discovery  
4) kubernetes_sd_configs: # Kubernetes service discovery  
5) consul_sd_configs:    # Consul service discovery

In Kubernetes monitoring scenarios, frequently updated resources like Pods and Services best demonstrate the advantages of Prometheus’ auto-discovery capabilities.

Working Principles

Prometheus queries the configuration information stored in Consul’s KV storage through the Consul API, then extracts service metadata from it;
Prometheus uses this information to construct target service URLs and adds them to the service discovery target list;
When services are deregistered or become unavailable, Prometheus will automatically remove them from the target list.

Containerized Consul Cluster

For testing and validation purposes only. Not suitable for production use! Production environments must undergo comprehensive cluster-based deployment validation with service process guardianship and monitoring.

Create a single-node Consul cluster:

# docker run -id -expose=[8300,8301,8302,8500,8600] --restart always -p 18300:8300 -p 18301:8301 -p 18302:8302 -p 18500:8500 -p 18600:8600 --name server1 -e 'CONSUL_LOCAL_CONFIG={"skip_leave_on_interrupt": true}' consul agent -server -bootstrap-expect=1 -node=server1 -bind=0.0.0.0 -client=0.0.0.0 -ui -datacenter dc1

Parameter Explanations:

    -expose: Exposes Consul's required ports: 8300, 8301, 8302, 8500, 8600
    --restart: always ensures automatic container restart upon failure
    -p: Establishes host-container port mappings
    --name: Container name
    -e: Environment variable for Consul configuration
    consul: Refers to the Consul image name, not the command
    agent: Command executed in the container. Parameter details:
    -server: Designates the node as a server type
    -bootstrap-expect: Specifies the number of server nodes required to trigger leader election (1 for single-node)
    -node: Node name
    -bind: Internal cluster communication address (default 0.0.0.0)
    -client: Client interface address (default 127.0.0.1)
    -ui: Enables Consul web UI
    -datacenter: Data center name

Validation test:

Access the web UI at, for example: http://192.10.192.109:18500/

`1`	`# curl localhost:18500`

Register Host to Consul

Example: Register node-exporter on a virtual machine to consul.

## Format
$ curl -X PUT -d '{"id": "'${host_name}'","name": "node-exporter","address": "'${host_addr}'","port":9100,"tags": ["dam"],"checks": [{"http": "http://'${host_addr}':9100/","interval": "5s"}]}' http://192.10.192.109:18500/v1/agent/service/register

## Example
$ curl -X PUT -d '{"id": "sh-middler2","name": "node-exporter","address": "192.10.192.134","port":9100,"tags": ["middleware"],"checks": [{"http": "http://192.10.192.134:9100/metrics","interval": "3s"}]}' http://192.10.192.109:18500/v1/agent/service/register

Parameter Explanation

id : Registration ID (must be unique in consul)
name : Service name
address: Binding IP for auto-registration
port: Binding port for auto-registration
tags: Registration tags (multiple allowed)
checks : Health checks
http:  Data source for check
interval: Check interval
http://192.10.192.109:18500/v1/agent/service/register : Consul registration API endpoint

Delete:

## Format
$ curl -X PUT http://192.10.192.109:18500/v1/agent/service/deregister/${id}

## Example
$ curl -X PUT http://192.10.192.109:18500/v1/agent/service/deregister/sh-middler2

Configuring Prometheus with Consul for Automatic Service Discovery

Modify Prometheus’ ConfigMap configuration file: prometheus-config.yaml

    - job_name: consul
      honor_labels: true
      metrics_path: /metrics
      scheme: http
      consul_sd_configs:    # Configuration based on Consul service discovery
        - server: 192.10.192.109:18500    # Consul listening address
          services: []                 # Match all services in Consul
      relabel_configs:             # Relabel configuration settings
      - source_labels: ['__meta_consul_tags']    # Assign __meta_consul_tags value to product
        target_label: 'servername'
      - source_labels: ['__meta_consul_dc']   # Assign __meta_consul_dc value to idc
        target_label: 'idc'
      - source_labels: ['__meta_consul_service']   
        regex: "consul"  # Match services named "consul"
        action: drop       # Drop/remove matching entries

Reload Prometheus using the above method. Open the Prometheus Target page to see the defined mysql-exporter job:

`1`	`curl -XPOST http://prometheus.kubernets.cn/-/reload`

Summary

Dynamic Service Discovery and Monitoring: By integrating with Consul, Prometheus dynamically maintains its target list, ensuring prompt discovery and monitoring as new services are deployed.
Scalability: Automated service discovery simplifies infrastructure scaling while preserving reliable monitoring availability and performance.
Seamless Integration: As the service registry, Consul enables Prometheus to integrate seamlessly with other tools in the Consul ecosystem, delivering a comprehensive solution for service infrastructure monitoring and management.
Self-Healing Capability: Automatic service discovery allows Prometheus to detect infrastructure changes in real time, continuously updating its target list to ensure uninterrupted monitoring data and high performance.

Dynamic Monitoring on Kang's Blog