Docker Compose 部署监控系统 Prometheus + Grafana + Node Exporter + Cadvisor - 思有云

前言

Prometheus + Grafana + Node Exporter + Cadvisor是一款开源的服务器系统状态监控平台系统,用于管理和存储各服务器的实时信息数据,方便监控和排查服务器故障.博主经过一段时间的安装,配置,测试也基本满足了个人以及中小企业团队对于日志管理的需求.整合了相对详细的部署和配置教程分享给大家.

本教程使用docker-compose整合部署,参考 stefanprodan/dockprom.

本文为 Stille 原创文章.经实践,测试,整理发布.如需转载请联系作者获得授权,并注明转载地址.

项目流程

服务端:部署Prometheus + Grafana到主监控服务器
客户端:部署Node Exporter + Cadvisor到需要被监控的服务器

常规流程

客户端通过Node Exporter采集系统状态信息,Cadvisor采集docker容器信息,并开放端口,由服务端Prometheus进行抓取,并由Grafana提供前端展示.

服务器部署 Prometheus + Grafana

拉取代码

git clone -b master https://github.com/stefanprodan/dockprom prom

创建 cadday 密码

docker run --rm caddy caddy hash-password --plaintext 'youtpasswd'
# 获取 hash 密码
JDJhJDE0JHJmeldKeDB0NjJnY0tjQk5wZXVYaC50QjdSbi9aVlVZeFYyV012UkI2Y

docker-compose.yml

volumes:

创建数据目录

cd prom
mkdir prometheus_data grafana_data
# 创建数据目录

修改docker-compose.yml中顶级卷配置到指定目录用于存储数据

volumes:
    prometheus_data:
      driver: local
      driver_opts:
        type: 'none'
        o: 'bind'
        device: '/root/prom/prometheus_data'
    grafana_data:
      driver: local
      driver_opts:
        type: 'none'
        o: 'bind'
        device: '/root/prom/grafana_data'

配置 grafana 密码

yourpasswd修改为你的grafana登陆密码

    environment:
      - GF_SECURITY_ADMIN_USER=${ADMIN_USER:-admin}
      - GF_SECURITY_ADMIN_PASSWORD=${ADMIN_PASSWORD:-yourpasswd}
      - GF_USERS_ALLOW_SIGN_UP=false

配置 caddy 密码

修改上文生成的caddy hash密码

    environment:
      - ADMIN_USER=${ADMIN_USER:-admin}
      - ADMIN_PASSWORD=${ADMIN_PASSWORD:-admin}
      - ADMIN_PASSWORD_HASH=${ADMIN_PASSWORD_HASH:-JDJhJDE0JHJmeldKeDB0NjJnY0tjQk5wZXVYaC50QjdSbi9aVlVZeFYyV012UkI2Y

配置 prometheus.yml

使用 file_sd_configs 配置独立配置文件路径,方便随时修改配置而不需要重启容器,并且方便批量分类管理.

创建独立配置目录

在./prometheus创建jobs目录,创建独立配置文件:

nodeexporter.yml
cadvisor.yml
prometheus.yml
pushgateway.yml

独立配置文件

参考示例./prometheus/jobs/nodeexporter.yml

- targets: 
  - "127.127.127.127:9100"
  labels: 
    hostname: Server
- targets: 
  - "127.127.127.128:9100"
  labels: 
    hostname: Server1
- targets: 
  - "127.127.127.129:9100"
  labels: 
    hostname: Server2

参考示例./prometheus/jobs/cadvisor.yml

- targets: 
  - "127.127.127.127:8080"
  labels: 
    hostname: Server
- targets: 
  - "127.127.127.128:8080"
  labels: 
    hostname: Server1
- targets: 
  - "127.127.127.129:8080"
  labels: 
    hostname: Server2

配置 prometheus.yml

参考修改独立配置文件路径./prometheus/prometheus.yml

scrape_configs:
  - job_name: 'nodeexporter'
    file_sd_configs:
      - files:
        - "/etc/prometheus/jobs/nodeexporter.yml"
        refresh_interval: 5s

  - job_name: 'cadvisor'
    file_sd_configs:
      - files:
        - "/etc/prometheus/jobs/cadvisor.yml"
        refresh_interval: 5s

  - job_name: 'prometheus'
    file_sd_configs:
      - files:
        - "/etc/prometheus/jobs/prometheus.yml"
        refresh_interval: 5s

  - job_name: 'pushgateway'
    file_sd_configs:
      - files:
        - "/etc/prometheus/jobs/pushgateway.yml"
        refresh_interval: 5s

客户端配置

docker-compose.yml

防火墙需开放以下相关端口,注意 nodeexporter 使用 network_mode: host 否则流量监控可能不准.

version: '3.2'
services:
  nodeexporter:
    image: prom/node-exporter:v1.1.2
    container_name: nodeexporter
    volumes:
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /:/rootfs:ro
    # ports:
      # - 9100:9100
    environment:
      TZ: Asia/Shanghai
    command:
      - '--path.procfs=/host/proc'
      - '--path.rootfs=/rootfs'
      - '--path.sysfs=/host/sys'
      - '--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)'
    labels:
      org.label-schema.group: "monitoring"
    restart: always
    network_mode: host


  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.39.0
    container_name: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:rw
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
      #- /cgroup:/cgroup:ro #doesn't work on MacOS only for Linux
    ports:
      - 8080:8080
    environment:
      TZ: Asia/Shanghai
    labels:
      org.label-schema.group: "monitoring"
    restart: always