![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202205041524836.png#alt=%E5%9B%BE%E7%89%87) ### 1.Prometheus简介 Prometheus 是一款基于时序数据库的开源监控告警系统，非常适合Kubernetes集群的监控。Prometheus的基本原理是通过HTTP协议周期性抓取被监控组件的状态，任意组件只要提供对应的HTTP接口就可以接入监控。不需要任何SDK或者其他的集成过程。这样做非常适合做虚拟化环境监控系统，比如VM、Docker、Kubernetes等。输出被监控组件信息的HTTP接口被叫做exporter 。目前互联网公司常用的组件大部分都有exporter可以直接使用，比如Varnish、Haproxy、Nginx、MySQL、Linux系统信息(包括磁盘、内存、CPU、网络等等)。Promethus有以下特点： - 支持多维数据模型：由度量名和键值对组成的时间序列数据 - 内置时间序列数据库TSDB - 支持PromQL查询语言，可以完成非常复杂的查询和分析，对图表展示和告警非常有意义 - 支持HTTP的Pull方式采集时间序列数据 - 支持PushGateway采集瞬时任务的数据 - 支持服务发现和静态配置两种方式发现目标 - 支持接入Grafana 官网：[https://prometheus.io/](https://prometheus.io/) ### 2.架构说明 ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202205041531946.jpg#alt=img) #### 2.1组件说明 - prometheus server是Prometheus组件中的核心部分，负责实现对监控数据的获取，存储以及查询。 - exporter简单说是采集端，通过http服务的形式保留一个url地址，prometheu server通过访问该exporter提供的endpoint端点，即可获取到需要采集的监控数据。 - AlertManager在prometheus中，支持基于PromQL创建告警规则，如果满足定义的规则，则会产生一条告警信息，进入AlertManager进行处理。可以集成邮件，微信或者通过webhook自定义报警。 - Pushgateway由于Prometheus数据采集采用pull方式进行设置的，内置必须保证prometheusserver和对应的exporter必须通信，当网络情况无法直接满足时，可以使用pushgateway来进行中转，可以通过pushgateway将内部网络数据主动push到gateway里面去，而prometheus采用pull方式拉取pushgateway中数据。 #### 2.2总结 prometheus负责从pushgateway和job中采集数据，存储到后端Storatge中，可以通过PromQL进行查询，推送alerts信息到AlertManager。AlertManager根据不同的路由规则进行报警通知。 **图二** ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202205041534475.png#alt=%E5%9B%BE%E7%89%87) **Prometheus Server: 收集指标和存储时间序列数据，并提供查询接口**
**ClientLibrary:客户端库**
**Push Gateway: 短期存储指标数据。主要用于临时性的任务**
**Exporters:采集已有的第三方服务监控指标并暴露metrics**
**Alertmanager:告警**
**Web UI :简单的web控制台** ### 3.对比zabbix | Zabbix | Prometheus | | :---: | :---: | | 后端用 C 开发，界面用 PHP 开发，定制化难度很高 | 后端用 golang 开发，前端是 Grafana，JSON 编辑即可解决。定制化难度较低 | | 集群规模上限为 10000 个节点 | 支持更大的集群规模，速度也更快 | | 更适合监控物理机环境，以IP地址为监控标识 | 更适合云环境的监控，对 OpenStack，Kubernetes 有更好的集成 | | 监控数据存储在关系型数据库内，如 MySQL，很难从现有数据中扩展维度 | 监控数据存储在基于时间序列的数据库内，便于对已有数据进行新的聚合 | | 安装简单，zabbix-server 一个软件包中包括了所有的服务端功能 | 安装相对复杂，监控、告警和界面都分属于不同的组件 | | 图形化界面比较成熟，界面上基本上能完成全部的配置操作 | 界面相对较弱，很多配置需要修改配置文件 | 主要使用场景区别是，Zabbix适合用于虚拟机、物理机的监控，因为每个监控指标是以 IP 地址作为标识进行区分的。而Prometheus的监控指标是由多个 label 组成，IP地址并不是唯一的区分指标，Prometheus 强大在可以支持自动发现规则，因此适合于容器环境。 从自定义监控项角度而言，Prometheus 开发难度较大，zabbix配合shell脚本更加方便。Prometheus在监控虚拟机上业务时，可能需要安装多个 exporter，而zabbix只需要安装一个 Agent。 Prometheus 采用拉数据方式，即使采用的是push-gateway，prometheus也是从push-gateway拉取数据。而Zabbix可以推可以拉。 ### 实验环境 | 主机名 | ip | 备注 | | :---: | :---: | :---: | | pr1 | 192.168.245.215 | server | | pr2 | 192.168.245.216 | node | | pr3 | 192.168.245.217 | node | ### Prometheus部署 在生产上启动命令应该做成软链接到/usr/bin下 #### Prometheus安装 ```shell # 9090端口 [root@pr1 ~]# cd /opt/ [root@pr1 opt]# yum -y install wget [root@pr1 opt]# wget https://github.com/prometheus/prometheus/releases/download/v2.35.0/prometheus-2.35.0.linux-amd64.tar.gz [root@pr1 opt]# ls prometheus-2.35.0.linux-amd64.tar.gz yum [root@pr1 opt]# tar -xvf prometheus-2.35.0.linux-amd64.tar.gz [root@pr1 opt]# cd prometheus-2.35.0.linux-amd64/ && ls console_libraries consoles LICENSE NOTICE prometheus prometheus.yml promtool [root@pr1 prometheus-2.35.0.linux-amd64]# ./prometheus --help #有些参数需要在启动prometheus时添加，具体参数可通过-help查看 [root@pr1 prometheus-2.35.0.linux-amd64]# ./prometheus --config.file="prometheus.yml" #指定配置文件启动 [root@pr1 prometheus-2.35.0.linux-amd64]# netstat -nultp # 检查9090端口 Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 860/sshd tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1071/master tcp6 0 0 :::22 :::* LISTEN 860/sshd tcp6 0 0 ::1:25 :::* LISTEN 1071/master tcp6 0 0 :::9090 :::* LISTEN 1201/./prometheus udp 0 0 127.0.0.1:323 0.0.0.0:* 602/chronyd udp6 0 0 ::1:323 :::* 602/chronyd ``` ```shell [root@pr1 prometheus-2.35.0.linux-amd64]# vim prometheus.yml # 全局配置 global: scrape_interval: 15s # 抓取数据间隔设置为15秒，默认为1分钟 evaluation_interval: 15s # 评估规则默认周期为15秒评估一次，默认1分钟 #scrape_timeout: 1m # 抓取超时时间默认为1分钟 # Alertmanager告警相关配置 alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # 加载告警规则，并根据全局定义的评估规则时间定期评估（是否符合告警条件） rule_files: # - "first_rules.yml" #告警规则文件所在位置 # - "second_rules.yml" # 收集数据配置列表 scrape_configs: # 作业命名 - job_name: 'prometheus' #静态配置目录列表 static_configs: #静态配置指定目标 - targets: ['localhost:9090'] ``` 浏览器打开IP地址:9090，如下图 ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271320044.png#alt=) IP地址:9090/metrics显示所有的监控项 ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271424045.png#alt=) #### 监控节点信息(node_exporter) 在Prometheus中，不仅提供了丰富的exporter，用于监控基础指标、中间件、网络设备等 在这里，node_exporter用于收集机器基础的监控指标，像cpu、内存、磁盘可用空间等，常用exporter如下： | 范围 | 常用Exporter | | :---: | :---: | | 数据库 | MySQL Exporter, Redis Exporter, MongoDB Exporter, MSSQL Exporter等 | | 硬件 | Apcupsd Exporter，IoT Edison Exporter， IPMI Exporter, Node Exporter等 | | 消息队列 | Beanstalkd Exporter, Kafka Exporter, NSQ Exporter, RabbitMQ Exporter等 | | 存储 | Ceph Exporter, Gluster Exporter, HDFS Exporter, ScaleIO Exporter等 | | HTTP服务 | Apache Exporter, HAProxy Exporter, Nginx Exporter等 | | API服务 | AWS ECS Exporter， Docker Cloud Exporter, Docker Hub Exporter, GitHub Exporter等 | | 日志 | Fluentd Exporter, Grok Exporter等 | | 监控系统 | Collectd Exporter, Graphite Exporter, InfluxDB Exporter, Nagios Exporter, SNMP Exporter等 | | 其它 | Blockbox Exporter, JIRA Exporter, Jenkins Exporter， Confluence Exporter等 | ```shell # 9100端口 [root@pr1 opt]# wget https://github.com/prometheus/node_exporter/releases/download/v1.3.1/node_exporter-1.3.1.linux-amd64.tar.gz [root@pr1 opt]# tar -xvf node_exporter-1.3.1.linux-amd64.tar.gz [root@pr1 opt]# cd node_exporter-1.3.1.linux-amd64 [root@pr1 node_exporter-1.3.1.linux-amd64]# ls LICENSE node_exporter NOTICE [root@pr1 node_exporter-1.3.1.linux-amd64]# ./node_exporter --help [root@pr1 node_exporter-1.3.1.linux-amd64]# ln -s /opt/node_exporter-1.3.1.linux-amd64/node_exporter /usr/bin/node_exporter [root@pr1 node_exporter-1.3.1.linux-amd64]# nohup ./node_exporter & [root@pr1 node_exporter-1.3.1.linux-amd64]# netstat -lntp |grep 9100 tcp6 0 0 :::9100 :::* LISTEN 10018/node_exporter ``` ```shell # 修改配置文件 [root@pr1 node_exporter-1.3.1.linux-amd64]# cd /opt/prometheus-2.35.0.linux-amd64 [root@pr1 prometheus-2.35.0.linux-amd64]# vim prometheus.yml [root@pr1 prometheus-2.35.0.linux-amd64]# grep -Ev "#|^$" prometheus.yml global: alerting: alertmanagers: - static_configs: - targets: rule_files: scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] - job_name: "node" static_configs: - targets: - 192.168.245.215:9100 # 重启 # 先kill [root@pr1 prometheus-2.35.0.linux-amd64]# nohup ./prometheus --config.file="prometheus.yml" & [1] 10329 [root@pr1 prometheus-2.35.0.linux-amd64]# nohup: 忽略输入并把输出追加到"nohup.out" # 浏览器分别打开IP地址:9090和IP地址:9100/metrics，在9090上尝试搜 ``` 搜一下node ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271504217.png#alt=) 如果直接是IP地址:9100，进去看到的是一个metrics超链接 ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271509507.png#alt=) 搜一下up可以看到node，也就是配置文件里面写的job_name ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271512860.png#alt=) 要查询cpu或磁盘空间可直接输入cpu或filesystem，在下边就会检索到，想进一步学习查询，可学习PromQL，Prometheus中查询数据、告警规则定义都要使用PromQL ```shell [root@pr1 opt]# scp -r node_exporter-1.3.1.linux-amd64 192.168.245.216:/opt/ [root@pr1 opt]# scp -r node_exporter-1.3.1.linux-amd64 192.168.245.217:/opt/ ln -s /opt/node_exporter-1.3.1.linux-amd64/node_exporter /usr/bin/node_exporter nohup node_exporter & [root@pr1 opt]# cd /opt/prometheus-2.35.0.linux-amd64 [root@pr1 prometheus-2.35.0.linux-amd64]# grep -Ev "#|^$" prometheus.yml global: alerting: alertmanagers: - static_configs: - targets: rule_files: scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] - job_name: "node" static_configs: - targets: ["192.168.245.215:9100","192.168.245.216:9100","192.168.245.217:9100"] 第二种写法：取一种即可 - 192.168.245.215:9100 - 192.168.245.216:9100 - 192.168.245.217:9100 ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271612570.png#alt=) #### 安装General [官网传送门](https://prometheus.io/docs/visualization/grafana/) ```shell # 下载，可以去清华源下载下来再传上去 [root@loki opt]# sudo /bin/systemctl daemon-reload [root@pr1 opt]# systemctl start grafana-server.service [root@pr1 opt]# systemctl status grafana-server.service [root@pr1 opt]# systemctl enable grafana-server.service # 浏览器输入IP地址:3000 # 账号密码默认都是admin，首次会强制更新密码 # 添加数据源 ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271653578.png#alt=) ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271657238.png#alt=) ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271658351.png#alt=) 导入模板，让数据显示更好看 ``` { "__inputs": [ { "name": "DS_TEST-PROMETHEUS", "label": "Prometheus", "description": "", "type": "datasource", "pluginId": "prometheus", "pluginName": "Prometheus" } ], "__elements": [], "__requires": [ { "type": "panel", "id": "bargauge", "name": "Bar gauge", "version": "" }, { "type": "grafana", "id": "grafana", "name": "Grafana", "version": "8.3.6" }, { "type": "panel", "id": "graph", "name": "Graph (old)", "version": "" }, { "type": "datasource", "id": "prometheus", "name": "Prometheus", "version": "1.0.0" }, { "type": "panel", "id": "stat", "name": "Stat", "version": "" }, { "type": "panel", "id": "table", "name": "Table", "version": "" } ], "annotations": { "list": [ { "$$hashKey": "object:2875", "builtIn": 1, "datasource": "-- Grafana --", "enable": true, "hide": true, "iconColor": "rgba(0, 211, 255, 1)", "name": "Annotations & Alerts", "target": { "limit": 100, "matchAny": false, "tags": [], "type": "dashboard" }, "type": "dashboard" } ] }, "description": "基于ConsulManager采集的ECS，可匹配自动同步方式采集ECS信息字段的展示，优化重要指标展示。使用Grafana8新表格重建，新增健康评分概念，并新增了整体资源消耗信息的一些图表。包含整体资源展示与资源明细图表：CPU 内存 磁盘 IO 网络等监控指标。https://github.com/starsliao/ConsulManager", "editable": true, "fiscalYearStartMonth": 0, "gnetId": 8919, "graphTooltip": 0, "id": null, "iteration": 1649806176515, "links": [ { "$$hashKey": "object:2300", "icon": "bolt", "tags": [], "targetBlank": true, "title": "Update", "tooltip": "更新当前仪表板", "type": "link", "url": "https://grafana.com/dashboards/8919" }, { "$$hashKey": "object:2301", "icon": "question", "tags": [], "targetBlank": true, "title": "GitHub", "tooltip": "查看更多仪表板", "type": "link", "url": "https://github.com/starsliao/ConsulManager" }, { "$$hashKey": "object:2302", "asDropdown": true, "icon": "external link", "tags": [], "targetBlank": true, "title": "", "type": "dashboards" } ], "liveNow": false, "panels": [ { "collapsed": false, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 }, "id": 187, "panels": [], "title": "资源总览：当前选中主机：$show_name，实例：$instance", "type": "row" }, { "description": "分区使用率、磁盘读取、磁盘写入、下载带宽、上传带宽，如果有多个网卡或者多个分区，是采集的使用率最高的网卡或者分区的数值。\n\n连接数：CurrEstab - 当前状态为 ESTABLISHED 或 CLOSE-WAIT 的 TCP 连接数。\n\n健康值是一个新增的指标，根据CPU，内存，IO计算出来的一个值，低于90分说明系统的资源使用情况需要注意了，这是一个正在测试的指标，参数可能需要根据实际情况再优化。", "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "custom": { "align": "center", "displayMode": "auto", "filterable": false }, "decimals": 1, "mappings": [], "max": 100, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null } ] }, "unit": "none" }, "overrides": [ { "matcher": { "id": "byName", "options": "内存" }, "properties": [ { "id": "unit", "value": "bytes" }, { "id": "decimals" }, { "id": "custom.width", "value": 67 }, { "id": "color", "value": { "fixedColor": "blue", "mode": "fixed" } }, { "id": "custom.displayMode", "value": "color-text" }, { "id": "decimals", "value": 0 } ] }, { "matcher": { "id": "byName", "options": "启动(天)" }, "properties": [ { "id": "unit", "value": "none" }, { "id": "custom.width", "value": 40 }, { "id": "decimals" } ] }, { "matcher": { "id": "byName", "options": "磁盘读取*" }, "properties": [ { "id": "unit", "value": "binBps" }, { "id": "custom.displayMode", "value": "color-background" }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "rgba(50, 172, 45, 0.97)", "value": null }, { "color": "rgba(237, 129, 40, 0.89)", "value": 10485760 }, { "color": "rgba(245, 54, 54, 0.9)", "value": 20485760 } ] } }, { "id": "custom.width", "value": 78 } ] }, { "matcher": { "id": "byName", "options": "磁盘写入*" }, "properties": [ { "id": "unit", "value": "binBps" }, { "id": "custom.displayMode", "value": "color-background" }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "rgba(50, 172, 45, 0.97)", "value": null }, { "color": "rgba(237, 129, 40, 0.89)", "value": 10485760 }, { "color": "rgba(245, 54, 54, 0.9)", "value": 20485760 } ] } }, { "id": "custom.width", "value": 81 } ] }, { "matcher": { "id": "byName", "options": "下载带宽*" }, "properties": [ { "id": "unit", "value": "binbps" }, { "id": "custom.displayMode", "value": "color-background" }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "rgba(50, 172, 45, 0.97)", "value": null }, { "color": "rgba(237, 129, 40, 0.89)", "value": 30485760 }, { "color": "rgba(245, 54, 54, 0.9)", "value": 104857600 } ] } }, { "id": "custom.width", "value": 81 }, { "id": "decimals" } ] }, { "matcher": { "id": "byName", "options": "上传带宽*" }, "properties": [ { "id": "unit", "value": "binbps" }, { "id": "custom.displayMode", "value": "color-background" }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "rgba(50, 172, 45, 0.97)", "value": null }, { "color": "rgba(237, 129, 40, 0.89)", "value": 30485760 }, { "color": "rgba(245, 54, 54, 0.9)", "value": 104857600 } ] } }, { "id": "custom.width", "value": 85 }, { "id": "decimals" } ] }, { "matcher": { "id": "byName", "options": "负载" }, "properties": [ { "id": "decimals", "value": 2 }, { "id": "custom.width", "value": 50 } ] }, { "matcher": { "id": "byName", "options": "连接数" }, "properties": [ { "id": "custom.displayMode", "value": "color-background" }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "rgba(50, 172, 45, 0.97)", "value": null }, { "color": "rgba(237, 129, 40, 0.89)", "value": 1000 }, { "color": "rgba(245, 54, 54, 0.9)", "value": 1500 } ] } }, { "id": "custom.width", "value": 54 }, { "id": "decimals" } ] }, { "matcher": { "id": "byName", "options": "TCP_tw" }, "properties": [ { "id": "custom.displayMode", "value": "color-background" }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "rgba(50, 172, 45, 0.97)", "value": null }, { "color": "rgba(237, 129, 40, 0.89)", "value": 5000 }, { "color": "rgba(245, 54, 54, 0.9)", "value": 20000 } ] } }, { "id": "custom.width", "value": 63 }, { "id": "decimals" } ] }, { "matcher": { "id": "byName", "options": "CPU" }, "properties": [ { "id": "custom.width", "value": 44 }, { "id": "decimals", "value": 0 }, { "id": "custom.displayMode", "value": "color-text" }, { "id": "color", "value": { "fixedColor": "blue", "mode": "fixed" } } ] }, { "matcher": { "id": "byName", "options": "IP" }, "properties": [ { "id": "custom.width", "value": 80 }, { "id": "custom.filterable", "value": true }, { "id": "mappings", "value": [ { "options": { "pattern": "/(.*):.*/", "result": { "index": 0, "text": "$1" } }, "type": "regex" } ] } ] }, { "matcher": { "id": "byName", "options": "名称" }, "properties": [ { "id": "custom.filterable", "value": true }, { "id": "custom.width", "value": 75 } ] }, { "matcher": { "id": "byName", "options": "健康值" }, "properties": [ { "id": "custom.width", "value": 52 }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "red", "value": null }, { "color": "orange", "value": 80 }, { "color": "green", "value": 90 } ] } }, { "id": "color", "value": { "mode": "thresholds" } }, { "id": "custom.displayMode", "value": "color-background" } ] }, { "matcher": { "id": "byRegexp", "options": "/.*使用率.*/" }, "properties": [ { "id": "unit", "value": "percent" }, { "id": "custom.displayMode", "value": "gradient-gauge" }, { "id": "color", "value": { "mode": "continuous-GrYlRd" } }, { "id": "custom.width", "value": 100 } ] }, { "matcher": { "id": "byName", "options": "IOutil使用率*" }, "properties": [ { "id": "custom.width", "value": 95 } ] }, { "matcher": { "id": "byName", "options": "分区使用率*" }, "properties": [ { "id": "custom.width", "value": 96 } ] }, { "matcher": { "id": "byName", "options": "到期日" }, "properties": [ { "id": "custom.width", "value": 86 } ] }, { "matcher": { "id": "byName", "options": "实例ID" }, "properties": [ { "id": "custom.width", "value": 62 } ] } ] }, "gridPos": { "h": 10, "w": 24, "x": 0, "y": 1 }, "id": 198, "options": { "footer": { "fields": [ "Value #B", "Value #C", "Value #L", "Value #H", "Value #I", "Value #M", "Value #N", "Value #J", "Value #K" ], "reducer": [ "sum" ], "show": false }, "showHeader": true, "sortBy": [ { "desc": false, "displayName": "健康值" } ] }, "pluginVersion": "8.3.6", "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"} - 0", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "主机名", "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "sum(time() - node_boot_time_seconds{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"})by(instance)/86400", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "运行时间", "refId": "D" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "node_memory_MemTotal_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"} - 0", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "总内存", "refId": "B" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "count(node_cpu_seconds_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",mode='system',name=~\".*$sname.*\"}) by (instance)", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "总核数", "refId": "C" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "node_load5{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}", "format": "table", "instant": true, "interval": "", "legendFormat": "5分钟负载", "refId": "L" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "(1 - avg(rate(node_cpu_seconds_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",mode=\"idle\",name=~\".*$sname.*\"}[$interval])) by (instance)) * 100", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "CPU使用率", "refId": "F" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "(1 - (node_memory_MemAvailable_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"} / (node_memory_MemTotal_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"})))* 100", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "内存使用率", "refId": "G" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "max((node_filesystem_size_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"ext.?|xfs\"}-node_filesystem_free_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"ext.?|xfs\"}) *100/(node_filesystem_avail_bytes {vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"ext.?|xfs\"}+(node_filesystem_size_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"ext.?|xfs\"}-node_filesystem_free_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"ext.?|xfs\"})))by(instance)", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "分区使用率", "refId": "E" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "max(rate(node_disk_read_bytes_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}[$interval])) by (instance)", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "最大读取", "refId": "H" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "max(rate(node_disk_written_bytes_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}[$interval])) by (instance)", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "最大写入", "refId": "I" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "node_netstat_Tcp_CurrEstab{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"} - 0", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "连接数", "refId": "M" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "node_sockstat_TCP_tw{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"} - 0", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "TIME_WAIT", "refId": "N" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "max(rate(node_network_receive_bytes_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}[$interval])*8) by (instance)", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "下载带宽", "refId": "J" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "max(rate(node_network_transmit_bytes_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}[$interval])*8) by (instance)", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "上传带宽", "refId": "K" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "((1-(1 - avg(irate(node_cpu_seconds_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",mode=\"idle\"}[$interval])) by (instance))^1.3)^(1/3)*0.5 + \r\n(1-(1 - avg(node_memory_MemAvailable_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"} / node_memory_MemTotal_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"})by (instance))^6)^(1/3)*0.3 + \r\n(1 - max(irate(node_disk_io_time_seconds_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}[$interval]))by (instance)^1.1)^(1/2)*0.2)*100", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "健康评分", "refId": "O" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "max(rate(node_disk_io_time_seconds_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}[$interval])) by (instance) *100", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "IOutil使用率", "refId": "P" } ], "title": "服务器资源总览表【分组：$group，主机总数：$total】", "transformations": [ { "id": "merge", "options": { "reducers": [] } }, { "id": "filterFieldsByName", "options": { "include": { "pattern": "/^Value #[^A]|^instance$|^name$|^iid$|^exp$/" } } }, { "id": "organize", "options": { "excludeByName": {}, "indexByName": { "Value #B": 6, "Value #C": 7, "Value #D": 3, "Value #E": 12, "Value #F": 9, "Value #G": 10, "Value #H": 13, "Value #I": 14, "Value #J": 17, "Value #K": 18, "Value #L": 8, "Value #M": 15, "Value #N": 16, "Value #O": 5, "Value #P": 11, "exp": 4, "iid": 2, "instance": 1, "name": 0 }, "renameByName": { "Value #B": "内存", "Value #C": "CPU", "Value #D": "启动(天)", "Value #E": "分区使用率*", "Value #F": "CPU使用率", "Value #G": "内存使用率", "Value #H": "磁盘读取*", "Value #I": "磁盘写入*", "Value #J": "下载带宽*", "Value #K": "上传带宽*", "Value #L": "负载", "Value #M": "连接数", "Value #N": "TCP_tw", "Value #O": "健康值", "Value #P": "IOutil使用率*", "exp": "到期日", "iid": "实例ID", "instance": "IP", "name": "名称", "nodename": "主机名" } } } ], "type": "table" }, { "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "custom": { "align": "auto", "displayMode": "auto" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "red", "value": 80 } ] } }, "overrides": [ { "matcher": { "id": "byRegexp", "options": "/.*使用率/" }, "properties": [ { "id": "unit", "value": "percent" }, { "id": "decimals", "value": 1 }, { "id": "custom.width", "value": 60 } ] }, { "matcher": { "id": "byName", "options": "IP" }, "properties": [ { "id": "custom.width", "value": 84 }, { "id": "mappings", "value": [ { "options": { "pattern": "/(.+):.+/", "result": { "index": 0, "text": "$1" } }, "type": "regex" } ] } ] }, { "matcher": { "id": "byName", "options": "P99内存使用率" }, "properties": [ { "id": "custom.width", "value": 75 } ] }, { "matcher": { "id": "byName", "options": "P99CPU使用率" }, "properties": [ { "id": "custom.width", "value": 70 } ] }, { "matcher": { "id": "byName", "options": "名称" }, "properties": [ { "id": "custom.width", "value": 115 } ] } ] }, "gridPos": { "h": 7, "w": 6, "x": 0, "y": 11 }, "id": 200, "options": { "footer": { "fields": "", "reducer": [ "sum" ], "show": false }, "showHeader": true, "sortBy": [ { "desc": false, "displayName": "P99内存使用率" } ] }, "pluginVersion": "8.3.6", "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"} - 0", "format": "table", "instant": true, "interval": "", "legendFormat": "主机名", "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "quantile_over_time(0.99, cpu:usage:rate1m{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}[7d:1h])", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "CPU使用率P99", "refId": "B" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "quantile_over_time(0.99, mem:usage:rate1m{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}[7d:1h])", "format": "table", "hide": false, "instant": true, "interval": "", "legendFormat": "内存使用率P99", "refId": "C" } ], "title": "最近7天P99资源使用率", "transformations": [ { "id": "seriesToColumns", "options": { "byField": "instance" } }, { "id": "filterFieldsByName", "options": { "include": { "pattern": "/^Value #[^A]|^instance$|^name 1$/" } } }, { "id": "organize", "options": { "excludeByName": {}, "indexByName": {}, "renameByName": { "Value #B": "P99CPU使用率", "Value #C": "P99内存使用率", "instance": "IP", "name": "名称", "name 1": "名称", "nodename": "主机名" } } } ], "type": "table" }, { "aliasColors": { "192.168.200.241:9100_Total": "dark-red", "Idle - Waiting for something to happen": "#052B51", "guest": "#9AC48A", "idle": "#052B51", "iowait": "#EAB839", "irq": "#BF1B00", "nice": "#C15C17", "sdb_每秒I/O操作%": "#d683ce", "softirq": "#E24D42", "steal": "#FCE2DE", "system": "#508642", "user": "#5195CE", "磁盘花费在I/O操作占比": "#ba43a9" }, "bars": false, "dashLength": 10, "dashes": false, "description": "", "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 0, "gridPos": { "h": 7, "w": 6, "x": 6, "y": 11 }, "hiddenSeries": false, "id": 191, "legend": { "alignAsTable": false, "avg": false, "current": false, "hideEmpty": true, "hideZero": true, "max": false, "min": false, "rightSide": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "maxDataPoints": 100, "maxPerRow": 6, "nullPointMode": "null", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:82", "alias": "总平均使用率", "yaxis": 2 }, { "$$hashKey": "object:83", "alias": "总核数", "color": "#C4162A" } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "count(node_cpu_seconds_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",mode='system'})", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "总核数", "refId": "B", "step": 240 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "sum(node_load5{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\".*$sname.*\",name=~\"$name\"})", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "总5分钟负载", "refId": "A", "step": 240 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "avg(1 - avg(rate(node_cpu_seconds_total{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",mode=\"idle\"}[$interval])) by (instance)) * 100", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "总平均使用率", "refId": "F", "step": 240 } ], "thresholds": [], "timeRegions": [], "title": "整体总负载与整体平均CPU使用率", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:8791", "format": "short", "label": "总负载", "logBase": 1, "show": true }, { "$$hashKey": "object:8792", "decimals": 1, "format": "percent", "label": "平均使用率", "logBase": 1, "show": true } ], "yaxis": { "align": false } }, { "aliasColors": { "192.168.200.241:9100_总内存": "dark-red", "内存_Avaliable": "#6ED0E0", "内存_Cached": "#EF843C", "内存_Free": "#629E51", "内存_Total": "#6d1f62", "内存_Used": "#eab839", "可用": "#9ac48a", "总内存": "#bf1b00" }, "bars": false, "dashLength": 10, "dashes": false, "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 0, "gridPos": { "h": 7, "w": 6, "x": 12, "y": 11 }, "height": "300", "hiddenSeries": false, "id": 195, "legend": { "alignAsTable": false, "avg": false, "current": false, "max": false, "min": false, "rightSide": false, "show": true, "sort": "current", "sortDesc": false, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "maxDataPoints": 100, "nullPointMode": "null", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:180", "alias": "总内存", "color": "#C4162A", "fill": 0 }, { "$$hashKey": "object:181", "alias": "总平均使用率", "yaxis": 2 } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "sum(node_memory_MemTotal_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"})", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "总内存", "refId": "A", "step": 4 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "sum(node_memory_MemTotal_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"} - node_memory_MemAvailable_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"})", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "总已用", "refId": "B", "step": 4 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "(sum(node_memory_MemTotal_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"} - node_memory_MemAvailable_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}) / sum(node_memory_MemTotal_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}))*100", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "总平均使用率", "refId": "H" } ], "thresholds": [], "timeRegions": [], "title": "整体总内存与整体平均内存使用率", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:8938", "decimals": 0, "format": "bytes", "label": "总内存量", "logBase": 1, "show": true }, { "$$hashKey": "object:8939", "decimals": 1, "format": "percent", "label": "平均使用率", "logBase": 1, "show": true } ], "yaxis": { "align": false } }, { "aliasColors": {}, "bars": false, "dashLength": 10, "dashes": false, "decimals": 1, "description": "", "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 0, "gridPos": { "h": 7, "w": 6, "x": 18, "y": 11 }, "hiddenSeries": false, "id": 197, "legend": { "alignAsTable": false, "avg": false, "current": false, "hideEmpty": false, "hideZero": false, "max": false, "min": false, "rightSide": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": false }, "lines": true, "linewidth": 2, "links": [], "maxDataPoints": 100, "nullPointMode": "null", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:280", "alias": "总平均使用率", "yaxis": 2 }, { "$$hashKey": "object:281", "alias": "总磁盘量", "color": "#C4162A" } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "sum(avg(node_filesystem_size_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"xfs|ext.*\"})by(device,instance))", "format": "time_series", "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "总磁盘量", "refId": "E" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "sum(avg(node_filesystem_size_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"xfs|ext.*\"})by(device,instance)) - sum(avg(node_filesystem_free_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"xfs|ext.*\"})by(device,instance))", "format": "time_series", "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "总使用量", "refId": "C" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "(sum(avg(node_filesystem_size_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"xfs|ext.*\"})by(device,instance)) - sum(avg(node_filesystem_free_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"xfs|ext.*\"})by(device,instance))) *100/(sum(avg(node_filesystem_avail_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"xfs|ext.*\"})by(device,instance))+(sum(avg(node_filesystem_size_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"xfs|ext.*\"})by(device,instance)) - sum(avg(node_filesystem_free_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\",fstype=~\"xfs|ext.*\"})by(device,instance))))", "format": "time_series", "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "总平均使用率", "refId": "A" } ], "thresholds": [], "timeRegions": [], "title": "整体总磁盘与整体平均磁盘使用率", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:8990", "decimals": 0, "format": "bytes", "label": "总磁盘量", "logBase": 1, "min": "0", "show": true }, { "$$hashKey": "object:8991", "decimals": 1, "format": "percent", "label": "平均使用率", "logBase": 1, "show": true } ], "yaxis": { "align": false } }, { "collapsed": false, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "gridPos": { "h": 1, "w": 24, "x": 0, "y": 18 }, "id": 189, "panels": [], "title": "资源明细：【$show_name】【$instance】【$iid】", "type": "row" }, { "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "decimals": 1, "mappings": [ { "options": { "0": { "text": "N/A" } }, "type": "value" } ], "max": 100, "min": 0.1, "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "#EAB839", "value": 70 }, { "color": "red", "value": 90 } ] }, "unit": "percent" }, "overrides": [] }, "gridPos": { "h": 6, "w": 3, "x": 0, "y": 19 }, "id": 177, "options": { "displayMode": "lcd", "orientation": "horizontal", "reduceOptions": { "calcs": [ "last" ], "fields": "", "values": false }, "showUnfilled": true }, "pluginVersion": "8.3.6", "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "100 - (avg(rate(node_cpu_seconds_total{instance=~\"$instance:.+\",mode=\"idle\"}[$interval])) * 100)", "instant": true, "interval": "", "legendFormat": "总CPU使用率", "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "avg(rate(node_cpu_seconds_total{instance=~\"$instance:.+\",mode=\"iowait\"}[$interval])) * 100", "hide": true, "instant": true, "interval": "", "legendFormat": "IOwait使用率", "refId": "C" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "(1 - (node_memory_MemAvailable_bytes{instance=~\"$instance:.+\"} / (node_memory_MemTotal_bytes{instance=~\"$instance:.+\"})))* 100", "instant": true, "interval": "", "legendFormat": "内存使用率", "refId": "B" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "(node_filesystem_size_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs\",mountpoint=\"$maxmount\"}-node_filesystem_free_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs\",mountpoint=\"$maxmount\"})*100 /(node_filesystem_avail_bytes {instance=~\"$instance:.+\",fstype=~\"ext.*|xfs\",mountpoint=\"$maxmount\"}+(node_filesystem_size_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs\",mountpoint=\"$maxmount\"}-node_filesystem_free_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs\",mountpoint=\"$maxmount\"}))", "hide": false, "instant": true, "interval": "", "legendFormat": "最大分区({{mountpoint}})使用率", "refId": "D" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "(1 - ((node_memory_SwapFree_bytes{instance=~\"$instance:.+\"} + 1)/ (node_memory_SwapTotal_bytes{instance=~\"$instance:.+\"} + 1))) * 100", "instant": true, "interval": "", "legendFormat": "交换分区使用率", "refId": "F" } ], "transformations": [], "type": "bargauge" }, { "description": "本看板中的：磁盘总量、使用量、可用量、使用率保持和df命令的Size、Used、Avail、Use% 列的值一致，并且Use%的值会四舍五入保留一位小数，会更加准确。\n\n注：df中Use%算法为：(size - free) * 100 / (avail + (size - free))，结果是整除则为该值，非整除则为该值+1，结果的单位是%。\n参考df命令源码：", "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "custom": { "align": "center", "displayMode": "auto" }, "displayName": "", "mappings": [], "thresholds": { "mode": "percentage", "steps": [ { "color": "green", "value": null } ] }, "unit": "none" }, "overrides": [ { "matcher": { "id": "byName", "options": "分区" }, "properties": [ { "id": "custom.width", "value": 81 } ] }, { "matcher": { "id": "byName", "options": "剩余空间" }, "properties": [ { "id": "unit", "value": "bytes" }, { "id": "decimals", "value": 0 }, { "id": "custom.displayMode", "value": "color-text" }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "red", "value": null }, { "color": "orange", "value": 10000000000 }, { "color": "green", "value": 20000000000 } ] } }, { "id": "custom.width", "value": 72 } ] }, { "matcher": { "id": "byName", "options": "使用率" }, "properties": [ { "id": "unit", "value": "percent" }, { "id": "decimals", "value": 0 }, { "id": "custom.displayMode", "value": "gradient-gauge" }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 70 }, { "color": "red", "value": 85 } ] } }, { "id": "custom.width", "value": 110 }, { "id": "min", "value": 0 }, { "id": "max", "value": 100 } ] }, { "matcher": { "id": "byName", "options": "总空间" }, "properties": [ { "id": "unit", "value": "bytes" }, { "id": "custom.width", "value": 87 } ] }, { "matcher": { "id": "byName", "options": "文件系统" }, "properties": [ { "id": "custom.width", "value": 46 } ] }, { "matcher": { "id": "byName", "options": "设备名" }, "properties": [ { "id": "custom.width" } ] } ] }, "gridPos": { "h": 6, "w": 10, "x": 3, "y": 19 }, "id": 181, "links": [ { "targetBlank": true, "title": "https://github.com/coreutils/coreutils/blob/master/src/df.c", "url": "https://github.com/coreutils/coreutils/blob/master/src/df.c" } ], "options": { "footer": { "fields": "", "reducer": [ "sum" ], "show": false }, "showHeader": true, "sortBy": [ { "desc": true, "displayName": "使用率" } ] }, "pluginVersion": "8.3.6", "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "node_filesystem_size_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}-0", "format": "table", "hide": false, "instant": true, "interval": "", "intervalFactor": 1, "legendFormat": "总量", "refId": "C" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "node_filesystem_avail_bytes {instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}-0", "format": "table", "hide": false, "instant": true, "interval": "10s", "intervalFactor": 1, "legendFormat": "", "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "(node_filesystem_size_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}-node_filesystem_free_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}) *100/(node_filesystem_avail_bytes {instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}+(node_filesystem_size_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}-node_filesystem_free_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}))", "format": "table", "hide": false, "instant": true, "interval": "", "intervalFactor": 1, "legendFormat": "", "refId": "B" } ], "title": "【$show_name】：各分区可用空间(EXT.*/XFS)", "transformations": [ { "id": "merge", "options": { "reducers": [] } }, { "id": "filterFieldsByName", "options": { "include": { "names": [ "device", "fstype", "mountpoint", "Value #C", "Value #A", "Value #B" ] } } }, { "id": "organize", "options": { "excludeByName": {}, "indexByName": {}, "renameByName": { "Value #A": "剩余空间", "Value #B": "使用率", "Value #C": "总空间", "device": "设备名", "fstype": "文件系统", "mountpoint": "分区" } } } ], "type": "table" }, { "fieldConfig": { "defaults": { "color": { "mode": "thresholds" }, "mappings": [], "thresholds": { "mode": "absolute", "steps": [ { "color": "green", "value": null } ] }, "unit": "none" }, "overrides": [ { "matcher": { "id": "byName", "options": "运行时间" }, "properties": [ { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "red", "value": null }, { "color": "orange", "value": 3600 }, { "color": "green", "value": 7200 } ] } }, { "id": "unit", "value": "s" } ] }, { "matcher": { "id": "byName", "options": "总内存" }, "properties": [ { "id": "unit", "value": "bytes" }, { "id": "decimals", "value": 0 } ] }, { "matcher": { "id": "byName", "options": "CPU iowait" }, "properties": [ { "id": "unit", "value": "percent" }, { "id": "thresholds", "value": { "mode": "percentage", "steps": [ { "color": "green", "value": null }, { "color": "orange", "value": 40 }, { "color": "red", "value": 60 } ] } }, { "id": "decimals", "value": 2 } ] }, { "matcher": { "id": "byName", "options": "总文件描述符" }, "properties": [ { "id": "unit", "value": "short" }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "red", "value": null }, { "color": "orange", "value": 50000 }, { "color": "green", "value": 200000 } ] } } ] }, { "matcher": { "id": "byName", "options": "最大进程打开文件" }, "properties": [ { "id": "unit", "value": "none" }, { "id": "thresholds", "value": { "mode": "absolute", "steps": [ { "color": "red", "value": null }, { "color": "orange", "value": 10000 }, { "color": "green", "value": 50000 } ] } } ] } ] }, "gridPos": { "h": 6, "w": 3, "x": 13, "y": 19 }, "id": 205, "interval": "15s", "links": [], "options": { "colorMode": "background", "graphMode": "none", "justifyMode": "center", "orientation": "auto", "reduceOptions": { "calcs": [ "lastNotNull" ], "fields": "", "values": false }, "text": {}, "textMode": "auto" }, "pluginVersion": "8.3.6", "targets": [ { "alias": "", "bucketAggs": [ { "id": "2", "settings": { "interval": "auto" }, "type": "date_histogram" } ], "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "avg(time() - node_boot_time_seconds{instance=~\"$instance:.+\"})", "hide": false, "instant": true, "interval": "", "legendFormat": "运行时间", "metrics": [ { "id": "1", "type": "count" } ], "query": "", "refId": "C", "timeField": "@timestamp" }, { "alias": "", "bucketAggs": [ { "id": "2", "settings": { "interval": "auto" }, "type": "date_histogram" } ], "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "count(node_cpu_seconds_total{instance=~\"$instance:.+\", mode='system'})", "instant": true, "interval": "", "legendFormat": "CPU 核数", "metrics": [ { "id": "1", "type": "count" } ], "query": "", "refId": "A", "timeField": "@timestamp" }, { "alias": "", "bucketAggs": [ { "id": "2", "settings": { "interval": "auto" }, "type": "date_histogram" } ], "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "sum(node_memory_MemTotal_bytes{instance=~\"$instance:.+\"})", "hide": false, "instant": true, "interval": "", "legendFormat": "总内存", "metrics": [ { "id": "1", "type": "count" } ], "query": "", "refId": "B", "timeField": "@timestamp" }, { "alias": "", "bucketAggs": [ { "id": "2", "settings": { "interval": "auto" }, "type": "date_histogram" } ], "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "avg(rate(node_cpu_seconds_total{instance=~\"$instance:.+\",mode=\"iowait\"}[$interval])) * 100", "hide": false, "instant": true, "interval": "", "legendFormat": "CPU iowait", "metrics": [ { "id": "1", "type": "count" } ], "query": "", "refId": "D", "timeField": "@timestamp" }, { "alias": "", "bucketAggs": [ { "id": "2", "settings": { "interval": "auto" }, "type": "date_histogram" } ], "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "node_filefd_maximum{instance=~\"$instance:.+\"}", "hide": false, "instant": true, "interval": "", "legendFormat": "总文件描述符", "metrics": [ { "id": "1", "type": "count" } ], "query": "", "refId": "E", "timeField": "@timestamp" }, { "alias": "", "bucketAggs": [ { "id": "2", "settings": { "interval": "auto" }, "type": "date_histogram" } ], "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": false, "expr": "process_max_fds{instance=~\"$instance:.+\"}", "hide": false, "instant": true, "interval": "", "legendFormat": "最大进程打开文件", "metrics": [ { "id": "1", "type": "count" } ], "query": "", "refId": "F", "timeField": "@timestamp" } ], "type": "stat" }, { "aliasColors": { "filefd_192.168.200.241:9100": "super-light-green", "switches_192.168.200.241:9100": "semi-dark-red", "使用的文件描述符_10.118.72.128:9100": "red", "总使用的文件描述符": "red", "总使用的文件描述符占比": "yellow", "每秒上下文切换次数": "red", "每秒上下文切换次数_10.118.71.245:9100": "yellow", "每秒上下文切换次数_10.118.72.128:9100": "yellow" }, "bars": false, "dashLength": 10, "dashes": false, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "description": "", "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 1, "gridPos": { "h": 6, "w": 8, "x": 16, "y": 19 }, "hiddenSeries": false, "hideTimeOverride": false, "id": 16, "legend": { "alignAsTable": false, "avg": false, "current": false, "max": false, "min": false, "rightSide": false, "show": true, "total": false, "values": false }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "options": { "alertThreshold": false }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 1, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:1456", "alias": "/.*占比/", "color": "#FADE2A", "linewidth": 1, "yaxis": 2 } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "node_filefd_allocated{instance=~\"$instance:.+\"}", "format": "time_series", "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "总使用的文件描述符", "refId": "B" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "rate(node_context_switches_total{instance=~\"$instance:.+\"}[$interval])", "interval": "", "intervalFactor": 1, "legendFormat": "每秒上下文切换次数", "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "(node_filefd_allocated{instance=~\"$instance:.+\"}/node_filefd_maximum{instance=~\"$instance:.+\"}) *100", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "总使用的文件描述符占比", "refId": "C" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "(process_open_fds{instance=~\"$instance:.+\"}/process_max_fds{instance=~\"$instance:.+\"}) *100", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "进程使用的文件描述符占比", "refId": "D" } ], "thresholds": [], "timeRegions": [], "title": "文件描述符/每秒上下文切换", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:1478", "format": "none", "label": "", "logBase": 1, "min": "0", "show": true }, { "$$hashKey": "object:1479", "decimals": 0, "format": "percent", "label": "", "logBase": 1, "min": "0", "show": true } ], "yaxis": { "align": false } }, { "aliasColors": { "192.168.200.241:9100_Total": "dark-red", "Idle - Waiting for something to happen": "#052B51", "guest": "#9AC48A", "idle": "#052B51", "iowait": "#EAB839", "irq": "#BF1B00", "nice": "#C15C17", "sdb_每秒I/O操作%": "#d683ce", "softirq": "#E24D42", "steal": "#FCE2DE", "system": "#508642", "user": "#5195CE", "磁盘花费在I/O操作占比": "#ba43a9" }, "bars": false, "dashLength": 10, "dashes": false, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "decimals": 2, "description": "", "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 0, "gridPos": { "h": 8, "w": 8, "x": 0, "y": 25 }, "hiddenSeries": false, "id": 7, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": true, "rightSide": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "maxPerRow": 6, "nullPointMode": "null", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:785", "alias": "/.*总使用率/", "color": "#C4162A", "fill": 0, "linewidth": 2 } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "avg(rate(node_cpu_seconds_total{instance=~\"$instance:.+\",mode=\"system\"}[$interval])) by (instance) *100", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "系统使用率", "refId": "A", "step": 20 }, { "expr": "avg(rate(node_cpu_seconds_total{instance=~\"$instance:.+\",mode=\"user\"}[$interval])) by (instance) *100", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "用户使用率", "refId": "B", "step": 240 }, { "expr": "avg(rate(node_cpu_seconds_total{instance=~\"$instance:.+\",mode=\"iowait\"}[$interval])) by (instance) *100", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "磁盘IO使用率", "refId": "D", "step": 240 }, { "expr": "(1 - avg(rate(node_cpu_seconds_total{instance=~\"$instance:.+\",mode=\"idle\"}[$interval])) by (instance))*100", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "总使用率", "refId": "F", "step": 240 } ], "thresholds": [], "timeRegions": [], "title": "CPU使用率", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:11294", "decimals": 0, "format": "percent", "label": "", "logBase": 1, "show": true }, { "$$hashKey": "object:11295", "format": "short", "logBase": 1, "show": false } ], "yaxis": { "align": false } }, { "aliasColors": { "192.168.200.241:9100_总内存": "dark-red", "使用率": "yellow", "内存_Avaliable": "#6ED0E0", "内存_Cached": "#EF843C", "内存_Free": "#629E51", "内存_Total": "#6d1f62", "内存_Used": "#eab839", "可用": "#9ac48a", "总内存": "#bf1b00" }, "bars": false, "dashLength": 10, "dashes": false, "decimals": 2, "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 0, "gridPos": { "h": 8, "w": 8, "x": 8, "y": 25 }, "height": "300", "hiddenSeries": false, "id": 156, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": true, "rightSide": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "options": { "alertThreshold": false }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:494", "alias": "总内存", "color": "#C4162A", "lines": false, "pointradius": 1, "points": true }, { "$$hashKey": "object:495", "alias": "使用率", "color": "rgb(0, 209, 255)", "linewidth": 2, "yaxis": 2 } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "node_memory_MemTotal_bytes{instance=~\"$instance:.+\"}", "format": "time_series", "hide": false, "instant": false, "interval": "2m", "intervalFactor": 1, "legendFormat": "总内存", "refId": "A", "step": 4 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "node_memory_MemTotal_bytes{instance=~\"$instance:.+\"} - node_memory_MemAvailable_bytes{instance=~\"$instance:.+\"}", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "已用", "refId": "B", "step": 4 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "node_memory_MemAvailable_bytes{instance=~\"$instance:.+\"}", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "可用", "refId": "F", "step": 4 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "(1 - (node_memory_MemAvailable_bytes{instance=~\"$instance:.+\"} / (node_memory_MemTotal_bytes{instance=~\"$instance:.+\"})))* 100", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "使用率", "refId": "H" } ], "thresholds": [], "timeRegions": [], "title": "内存信息", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:518", "format": "bytes", "logBase": 1, "min": "0", "show": true }, { "$$hashKey": "object:519", "format": "percent", "label": "内存使用率", "logBase": 1, "max": "100", "min": "0", "show": true } ], "yaxis": { "align": false } }, { "aliasColors": { "192.168.10.227:9100_em1_in下载": "super-light-green", "192.168.10.227:9100_em1_out上传": "dark-blue" }, "bars": false, "dashLength": 10, "dashes": false, "decimals": 2, "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 0, "gridPos": { "h": 8, "w": 8, "x": 16, "y": 25 }, "height": "300", "hiddenSeries": false, "id": 157, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": true, "rightSide": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 2, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:115", "alias": "/.*_out上传$/", "transform": "negative-Y" } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "rate(node_network_receive_bytes_total{instance=~\"$instance:.+\",device=~\"$device\"}[$interval])*8", "format": "time_series", "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}_in下载", "refId": "A", "step": 4 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "rate(node_network_transmit_bytes_total{instance=~\"$instance:.+\",device=~\"$device\"}[$interval])*8", "format": "time_series", "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}_out上传", "refId": "B", "step": 4 } ], "thresholds": [], "timeRegions": [], "title": "每秒网络带宽使用$device", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:122", "format": "bps", "label": "上传（-）/下载（+）", "logBase": 1, "show": true }, { "$$hashKey": "object:123", "format": "short", "logBase": 1, "show": false } ], "yaxis": { "align": false } }, { "aliasColors": { "15分钟": "#6ED0E0", "1分钟": "#BF1B00", "5分钟": "#CCA300" }, "bars": false, "dashLength": 10, "dashes": false, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "editable": true, "error": false, "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 1, "grid": {}, "gridPos": { "h": 8, "w": 8, "x": 0, "y": 33 }, "height": "300", "hiddenSeries": false, "id": 13, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": true, "rightSide": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "maxPerRow": 6, "nullPointMode": "null", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:873", "alias": "/.*总核数/", "color": "#C4162A", "lines": false, "pointradius": 1, "points": true } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "node_load1{instance=~\"$instance:.+\"}", "format": "time_series", "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "1分钟负载", "metric": "", "refId": "A", "step": 20, "target": "" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "node_load5{instance=~\"$instance:.+\"}", "format": "time_series", "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "5分钟负载", "refId": "B", "step": 20 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "node_load15{instance=~\"$instance:.+\"}", "format": "time_series", "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "15分钟负载", "refId": "C", "step": 20 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": " sum(count(node_cpu_seconds_total{instance=~\"$instance:.+\", mode='system'}) by (cpu,instance)) by(instance)", "format": "time_series", "instant": false, "interval": "2m", "intervalFactor": 1, "legendFormat": "CPU总核数", "refId": "D", "step": 20 } ], "thresholds": [], "timeRegions": [], "title": "系统平均负载", "tooltip": { "msResolution": false, "shared": true, "sort": 2, "value_type": "cumulative" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:880", "decimals": 0, "format": "short", "logBase": 1, "show": true }, { "$$hashKey": "object:881", "format": "short", "logBase": 1, "show": false } ], "yaxis": { "align": false } }, { "aliasColors": { "等待IO完成阻塞的进程": "red", "运行态的进程": "green" }, "bars": true, "dashLength": 10, "dashes": false, "editable": true, "error": false, "fill": 0, "fillGradient": 0, "grid": {}, "gridPos": { "h": 8, "w": 8, "x": 8, "y": 33 }, "hiddenSeries": false, "id": 202, "instanceColors": {}, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": false, "show": true, "total": false, "values": true }, "lines": false, "linewidth": 1, "links": [], "nullPointMode": "null as zero", "options": { "alertThreshold": false }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": true, "steppedLine": false, "targets": [ { "calculatedInterval": "2m", "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "datasourceErrors": {}, "errors": {}, "exemplar": true, "expr": "node_procs_running{instance=~\"$instance:.+\"}", "format": "time_series", "interval": "1m", "intervalFactor": 1, "legendFormat": "运行态的进程", "metric": "", "prometheusLink": "/api/datasources/proxy/1/graph#%5B%7B%22expr%22%3A%22node_procs_running%7Binstance%3D%5C%22%24host%5C%22%7D%22%2C%22range_input%22%3A%2243200s%22%2C%22end_input%22%3A%222015-9-18%2013%3A46%22%2C%22step_input%22%3A%22%22%2C%22stacked%22%3Atrue%2C%22tab%22%3A0%7D%5D", "refId": "A", "step": 5, "target": "" }, { "calculatedInterval": "2m", "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "datasourceErrors": {}, "errors": {}, "exemplar": true, "expr": "node_procs_blocked{instance=~\"$instance:.+\"}", "format": "time_series", "interval": "1m", "intervalFactor": 1, "legendFormat": "等待IO完成阻塞的进程", "metric": "", "prometheusLink": "/api/datasources/proxy/1/graph#%5B%7B%22expr%22%3A%22node_procs_blocked%7Binstance%3D%5C%22%24host%5C%22%7D%22%2C%22range_input%22%3A%2243200s%22%2C%22end_input%22%3A%222015-9-18%2013%3A46%22%2C%22step_input%22%3A%22%22%2C%22stacked%22%3Atrue%2C%22tab%22%3A0%7D%5D", "refId": "B", "step": 5, "target": "" } ], "thresholds": [], "timeRegions": [], "title": "进程运行状态", "tooltip": { "msResolution": false, "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:1845", "format": "none", "label": "", "logBase": 1, "min": 0, "show": true }, { "$$hashKey": "object:1846", "format": "none", "logBase": 1, "min": 0, "show": false } ], "yaxis": { "align": false } }, { "aliasColors": { "容量%：/": "red" }, "bars": false, "dashLength": 10, "dashes": false, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "decimals": 1, "description": "", "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 0, "gridPos": { "h": 8, "w": 8, "x": 16, "y": 33 }, "hiddenSeries": false, "id": 174, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": true, "rightSide": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 2, "links": [], "nullPointMode": "null", "options": { "alertThreshold": false }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:4248", "alias": "/Inodes.*/", "color": "#5794F2", "linewidth": 1, "yaxis": 2 } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "(node_filesystem_size_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}-node_filesystem_free_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}) *100/(node_filesystem_avail_bytes {instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}+(node_filesystem_size_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}-node_filesystem_free_bytes{instance=~\"$instance:.+\",fstype=~\"ext.*|xfs|nfs\",mountpoint !~\".*pod.*\"}))", "format": "time_series", "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "容量%：{{mountpoint}}", "refId": "A" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "1 - node_filesystem_files_free{instance=~\"$instance:.+\",fstype=~\"ext.?|xfs\"} / node_filesystem_files{instance=~\"$instance:.+\",fstype=~\"ext.?|xfs\"}", "hide": false, "interval": "", "legendFormat": "Inodes%：{{mountpoint}}", "refId": "B" } ], "thresholds": [], "timeRegions": [], "title": "磁盘使用率", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:4255", "format": "percent", "label": "", "logBase": 1, "show": true }, { "$$hashKey": "object:4256", "format": "percentunit", "logBase": 1, "show": true } ], "yaxis": { "align": false } }, { "aliasColors": { "vda_write": "#6ED0E0" }, "bars": false, "dashLength": 10, "dashes": false, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "decimals": 2, "description": "Read bytes 每个磁盘分区每秒读取的比特数\nWritten bytes 每个磁盘分区每秒写入的比特数", "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 0, "gridPos": { "h": 8, "w": 6, "x": 0, "y": 41 }, "height": "300", "hiddenSeries": false, "id": 168, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:704", "alias": "/.*_读取$/", "transform": "negative-Y" } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "rate(node_disk_read_bytes_total{instance=~\"$instance:.+\"}[$interval])", "format": "time_series", "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}_读取", "refId": "A", "step": 10 }, { "expr": "rate(node_disk_written_bytes_total{instance=~\"$instance:.+\"}[$interval])", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}_写入", "refId": "B", "step": 10 } ], "thresholds": [], "timeRegions": [], "title": "每秒磁盘读写容量", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:711", "format": "Bps", "label": "读取（-）/写入（+）", "logBase": 1, "show": true }, { "$$hashKey": "object:712", "format": "short", "logBase": 1, "show": false } ], "yaxis": { "align": false } }, { "aliasColors": { "vda_write": "#6ED0E0" }, "bars": false, "dashLength": 10, "dashes": false, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "decimals": 0, "description": "Reads completed: 每个磁盘分区每秒读完成次数\n\nWrites completed: 每个磁盘分区每秒写完成次数\n\nIO now 每个磁盘分区每秒正在处理的输入/输出请求数", "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 0, "gridPos": { "h": 8, "w": 6, "x": 6, "y": 41 }, "height": "300", "hiddenSeries": false, "id": 161, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:5507", "alias": "/.*_读取$/", "transform": "negative-Y" } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "rate(node_disk_reads_completed_total{instance=~\"$instance:.+\"}[$interval])", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}_读取", "refId": "A", "step": 10 }, { "expr": "rate(node_disk_writes_completed_total{instance=~\"$instance:.+\"}[$interval])", "format": "time_series", "hide": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}_写入", "refId": "B", "step": 10 }, { "expr": "node_disk_io_now{instance=~\"$instance:.+\"}", "format": "time_series", "hide": true, "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}", "refId": "C" } ], "thresholds": [], "timeRegions": [], "title": "磁盘读写速率（IOPS）", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:5514", "format": "none", "label": "读取（-）/写入（+）I/O ops/sec", "logBase": 1, "show": true }, { "$$hashKey": "object:5515", "format": "short", "logBase": 1, "show": true } ], "yaxis": { "align": false } }, { "aliasColors": { "vda": "#6ED0E0" }, "bars": false, "dashLength": 10, "dashes": false, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "decimals": 2, "description": "Read time seconds 每个磁盘分区读操作花费的秒数\n\nWrite time seconds 每个磁盘分区写操作花费的秒数\n\nIO time seconds 每个磁盘分区输入/输出操作花费的秒数\n\nIO time weighted seconds每个磁盘分区输入/输出操作花费的加权秒数", "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 1, "gridPos": { "h": 8, "w": 6, "x": 12, "y": 41 }, "height": "300", "hiddenSeries": false, "id": 160, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null as zero", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:1194", "alias": "/,*_读取$/", "transform": "negative-Y" } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "expr": "rate(node_disk_read_time_seconds_total{instance=~\"$instance:.+\"}[$interval]) / rate(node_disk_reads_completed_total{instance=~\"$instance:.+\"}[$interval])", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}_读取", "refId": "B" }, { "expr": "rate(node_disk_write_time_seconds_total{instance=~\"$instance:.+\"}[$interval]) / rate(node_disk_writes_completed_total{instance=~\"$instance:.+\"}[$interval])", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}_写入", "refId": "C" }, { "expr": "rate(node_disk_io_time_seconds_total{instance=~\"$instance:.+\"}[$interval])", "format": "time_series", "hide": true, "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}", "refId": "A", "step": 10 }, { "expr": "rate(node_disk_io_time_weighted_seconds_total{instance=~\"$instance:.+\"}[$interval])", "format": "time_series", "hide": true, "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}_加权", "refId": "D" } ], "thresholds": [], "timeRegions": [], "title": "每次IO读写的耗时（参考：小于100ms）", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:1201", "format": "s", "label": "读取（-）/写入（+）", "logBase": 1, "show": true }, { "$$hashKey": "object:1202", "format": "short", "logBase": 1, "show": false } ], "yaxis": { "align": false } }, { "aliasColors": { "Idle - Waiting for something to happen": "#052B51", "guest": "#9AC48A", "idle": "#052B51", "iowait": "#EAB839", "irq": "#BF1B00", "nice": "#C15C17", "sdb_每秒I/O操作%": "#d683ce", "softirq": "#E24D42", "steal": "#FCE2DE", "system": "#508642", "user": "#5195CE", "磁盘花费在I/O操作占比": "#ba43a9" }, "bars": false, "dashLength": 10, "dashes": false, "description": "每一秒钟的自然时间内，花费在I/O上的耗时。（wall-clock time）\n\nnode_disk_io_time_seconds_total：\n磁盘花费在输入/输出操作上的秒数。该值为累加值。（Milliseconds Spent Doing I/Os）\n\nrate(node_disk_io_time_seconds_total[1m])：\n计算每秒的速率：(last值-last前一个值)/时间戳差值，即：1秒钟内磁盘花费在I/O操作的时间占比。", "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 1, "gridPos": { "h": 8, "w": 6, "x": 18, "y": 41 }, "hiddenSeries": false, "id": 175, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": false, "show": true, "sort": "current", "sortDesc": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "maxPerRow": 6, "nullPointMode": "null", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "rate(node_disk_io_time_seconds_total{instance=~\"$instance:.+\"}[$interval])", "format": "time_series", "interval": "", "intervalFactor": 1, "legendFormat": "{{device}}_每秒I/O操作%", "refId": "C" } ], "thresholds": [], "timeRegions": [], "title": "每1秒内I/O操作耗时占比（I/O Util）", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:177", "format": "percentunit", "label": "", "logBase": 1, "show": true }, { "$$hashKey": "object:178", "format": "short", "logBase": 1, "show": false } ], "yaxis": { "align": false } }, { "aliasColors": { "192.168.200.241:9100_TCP_alloc": "semi-dark-blue", "TCP": "#6ED0E0", "TCP_alloc": "blue" }, "bars": false, "dashLength": 10, "dashes": false, "description": "Sockets_used - 已使用的所有协议套接字总量\n\nCurrEstab - 当前状态为 ESTABLISHED 或 CLOSE-WAIT 的 TCP 连接数\n\nTCP_alloc - 已分配（已建立、已申请到sk_buff）的TCP套接字数量\n\nTCP_tw - 等待关闭的TCP连接数\n\nUDP_inuse - 正在使用的 UDP 套接字数量\n\nRetransSegs - TCP 重传报文数\n\nOutSegs - TCP 发送的报文数\n\nInSegs - TCP 接收的报文数", "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 0, "fillGradient": 0, "gridPos": { "h": 8, "w": 16, "x": 0, "y": 49 }, "height": "300", "hiddenSeries": false, "id": 158, "interval": "", "legend": { "alignAsTable": true, "avg": false, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": false, "rightSide": true, "show": true, "total": false, "values": true }, "lines": true, "linewidth": 1, "links": [], "nullPointMode": "null", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 5, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:450", "alias": "/.*Sockets_used/", "color": "#E02F44", "lines": true, "linewidth": 2, "yaxis": 2 } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "node_netstat_Tcp_CurrEstab{instance=~\"$instance:.+\"}", "format": "time_series", "hide": false, "instant": false, "interval": "", "intervalFactor": 1, "legendFormat": "CurrEstab", "refId": "A", "step": 20 }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "node_sockstat_TCP_tw{instance=~\"$instance:.+\"}", "format": "time_series", "interval": "", "intervalFactor": 1, "legendFormat": "TCP_tw", "refId": "D" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "node_sockstat_sockets_used{instance=~\"$instance:.+\"}", "hide": false, "interval": "2m", "intervalFactor": 1, "legendFormat": "Sockets_used", "refId": "B" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "node_sockstat_UDP_inuse{instance=~\"$instance:.+\"}", "interval": "", "legendFormat": "UDP_inuse", "refId": "C" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "node_sockstat_TCP_alloc{instance=~\"$instance:.+\"}", "interval": "", "legendFormat": "TCP_alloc", "refId": "E" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "rate(node_netstat_Tcp_PassiveOpens{instance=~\"$instance:.+\"}[$interval])", "hide": true, "interval": "", "legendFormat": "{{instance}}_Tcp_PassiveOpens", "refId": "G" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "rate(node_netstat_Tcp_ActiveOpens{instance=~\"$instance:.+\"}[$interval])", "hide": true, "interval": "", "legendFormat": "{{instance}}_Tcp_ActiveOpens", "refId": "F" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "rate(node_netstat_Tcp_InSegs{instance=~\"$instance:.+\"}[$interval])", "interval": "", "legendFormat": "Tcp_InSegs", "refId": "H" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "rate(node_netstat_Tcp_OutSegs{instance=~\"$instance:.+\"}[$interval])", "interval": "", "legendFormat": "Tcp_OutSegs", "refId": "I" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "rate(node_netstat_Tcp_RetransSegs{instance=~\"$instance:.+\"}[$interval])", "hide": false, "interval": "", "legendFormat": "Tcp_RetransSegs", "refId": "J" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "expr": "rate(node_netstat_TcpExt_ListenDrops{instance=~\"$instance:.+\"}[$interval])", "hide": true, "interval": "", "legendFormat": "", "refId": "K" } ], "thresholds": [], "timeRegions": [], "title": "网络Socket连接信息", "tooltip": { "shared": true, "sort": 2, "value_type": "individual" }, "transformations": [], "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:465", "format": "none", "logBase": 1, "show": true }, { "$$hashKey": "object:466", "format": "none", "label": "已使用的所有协议套接字总量", "logBase": 1, "show": true } ], "yaxis": { "align": false } }, { "aliasColors": { "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_cni0_in": "light-red", "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_cni0_in下载": "green", "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_cni0_out上传": "yellow", "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_eth0_in下载": "purple", "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_eth0_out": "purple", "cn-shenzhen.i-wz9cq1dcb6zwc39ehw59_eth0_out上传": "blue" }, "bars": true, "dashLength": 10, "dashes": false, "editable": true, "error": false, "fieldConfig": { "defaults": { "links": [] }, "overrides": [] }, "fill": 1, "fillGradient": 0, "grid": {}, "gridPos": { "h": 8, "w": 8, "x": 16, "y": 49 }, "hiddenSeries": false, "id": 183, "legend": { "alignAsTable": true, "avg": true, "current": true, "hideEmpty": true, "hideZero": true, "max": true, "min": false, "show": false, "sort": "current", "sortDesc": true, "total": true, "values": true }, "lines": false, "linewidth": 2, "links": [], "nullPointMode": "null as zero", "options": { "alertThreshold": true }, "percentage": false, "pluginVersion": "8.3.6", "pointradius": 1, "points": false, "renderer": "flot", "seriesOverrides": [ { "$$hashKey": "object:172", "alias": "/.*_out上传$/", "transform": "negative-Y" } ], "spaceLength": 10, "stack": false, "steppedLine": false, "targets": [ { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "increase(node_network_receive_bytes_total{instance=~\"$instance:.+\",device=~\"$device\"}[1m])", "interval": "1m", "intervalFactor": 2, "legendFormat": "{{device}}_in下载", "metric": "", "refId": "A", "step": 600, "target": "" }, { "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "exemplar": true, "expr": "increase(node_network_transmit_bytes_total{instance=~\"$instance:.+\",device=~\"$device\"}[1m])", "hide": false, "interval": "1m", "intervalFactor": 2, "legendFormat": "{{device}}_out上传", "refId": "B", "step": 600 } ], "thresholds": [], "timeRegions": [], "title": "每分钟流量$device", "tooltip": { "msResolution": false, "shared": true, "sort": 0, "value_type": "cumulative" }, "type": "graph", "xaxis": { "mode": "time", "show": true, "values": [] }, "yaxes": [ { "$$hashKey": "object:267", "format": "bytes", "label": "上传（-）/下载（+）", "logBase": 1, "show": true }, { "$$hashKey": "object:268", "format": "short", "logBase": 1, "show": false } ], "yaxis": { "align": false } } ], "refresh": false, "schemaVersion": 34, "style": "dark", "tags": [ "Prometheus", "node_exporter", "StarsL.cn" ], "templating": { "list": [ { "current": {}, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "definition": "label_values(node_uname_info, vendor)", "hide": 0, "includeAll": false, "label": "云厂商", "multi": false, "name": "vendor", "options": [], "query": { "query": "label_values(node_uname_info, vendor)", "refId": "StandardVariableQuery" }, "refresh": 2, "regex": "", "skipUrlSync": false, "sort": 5, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "current": {}, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "definition": "label_values(node_uname_info{vendor=~\"$vendor\"}, account)", "hide": 0, "includeAll": false, "label": "账户", "multi": false, "name": "account", "options": [], "query": { "query": "label_values(node_uname_info{vendor=~\"$vendor\"}, account)", "refId": "StandardVariableQuery" }, "refresh": 2, "regex": "", "skipUrlSync": false, "sort": 5, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "current": {}, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "definition": "label_values(node_uname_info{vendor=~\"$vendor\",account=~\"$account\"}, group)", "hide": 0, "includeAll": true, "label": "分组", "multi": false, "name": "group", "options": [], "query": { "query": "label_values(node_uname_info{vendor=~\"$vendor\",account=~\"$account\"}, group)", "refId": "StandardVariableQuery" }, "refresh": 2, "regex": "", "skipUrlSync": false, "sort": 5, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "current": {}, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "definition": "label_values(node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\"}, name)", "hide": 0, "includeAll": true, "label": "名称", "multi": false, "name": "name", "options": [], "query": { "query": "label_values(node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\"}, name)", "refId": "StandardVariableQuery" }, "refresh": 2, "regex": "", "skipUrlSync": false, "sort": 5, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "allFormat": "glob", "current": {}, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "definition": "label_values(node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\"},instance)", "hide": 0, "includeAll": false, "label": "IP", "multi": false, "multiFormat": "regex values", "name": "instance", "options": [], "query": { "query": "label_values(node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\"},instance)", "refId": "StandardVariableQuery" }, "refresh": 2, "regex": "/(.*):.*/", "skipUrlSync": false, "sort": 5, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "auto": false, "auto_count": 100, "auto_min": "10s", "current": { "selected": false, "text": "2m", "value": "2m" }, "hide": 0, "label": "间隔", "name": "interval", "options": [ { "selected": false, "text": "30s", "value": "30s" }, { "selected": false, "text": "1m", "value": "1m" }, { "selected": true, "text": "2m", "value": "2m" }, { "selected": false, "text": "3m", "value": "3m" }, { "selected": false, "text": "5m", "value": "5m" }, { "selected": false, "text": "10m", "value": "10m" }, { "selected": false, "text": "30m", "value": "30m" } ], "query": "30s,1m,2m,3m,5m,10m,30m", "queryValue": "", "refresh": 2, "skipUrlSync": false, "type": "interval" }, { "current": {}, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "definition": "query_result(count(node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}))", "hide": 2, "includeAll": false, "label": "主机数", "multi": false, "name": "total", "options": [], "query": { "query": "query_result(count(node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",name=~\".*$sname.*\"}))", "refId": "StandardVariableQuery" }, "refresh": 2, "regex": "/{} (.*) .*/", "skipUrlSync": false, "sort": 0, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "allFormat": "glob", "current": {}, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "definition": "label_values(node_network_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",instance=~\"$instance:.+\",device!~'tap.*|veth.*|br.*|docker.*|virbr.*|lo.*|cni.*'},device)", "hide": 0, "includeAll": true, "label": "网卡", "multi": true, "multiFormat": "regex values", "name": "device", "options": [], "query": { "query": "label_values(node_network_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",instance=~\"$instance:.+\",device!~'tap.*|veth.*|br.*|docker.*|virbr.*|lo.*|cni.*'},device)", "refId": "StandardVariableQuery" }, "refresh": 2, "regex": "", "skipUrlSync": false, "sort": 1, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "current": {}, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "definition": "query_result(topk(1,sort_desc (max(node_filesystem_size_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",instance=~\"$instance:.+\",fstype=~\"ext.?|xfs\",mountpoint!~\".*pods.*\"}) by (mountpoint))))", "hide": 2, "includeAll": false, "label": "最大挂载目录", "multi": false, "name": "maxmount", "options": [], "query": { "query": "query_result(topk(1,sort_desc (max(node_filesystem_size_bytes{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",instance=~\"$instance:.+\",fstype=~\"ext.?|xfs\",mountpoint!~\".*pods.*\"}) by (mountpoint))))", "refId": "StandardVariableQuery" }, "refresh": 2, "regex": "/.*\\\"(.*)\\\".*/", "skipUrlSync": false, "sort": 5, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "current": {}, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "definition": "label_values(node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",instance=~\"$instance:.+\"}, name)", "hide": 2, "includeAll": false, "label": "展示使用的名称", "multi": false, "name": "show_name", "options": [], "query": { "query": "label_values(node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\",instance=~\"$instance:.+\"}, name)", "refId": "StandardVariableQuery" }, "refresh": 2, "regex": "", "skipUrlSync": false, "sort": 5, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "allFormat": "glob", "current": {}, "datasource": { "type": "prometheus", "uid": "${DS_TEST-PROMETHEUS}" }, "definition": "label_values(node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\"},iid)", "hide": 2, "includeAll": false, "label": "实例ID", "multi": false, "multiFormat": "regex values", "name": "iid", "options": [], "query": { "query": "label_values(node_uname_info{vendor=~\"$vendor\",account=~\"$account\",group=~\"$group\",name=~\"$name\"},iid)", "refId": "StandardVariableQuery" }, "refresh": 2, "regex": "", "skipUrlSync": false, "sort": 5, "tagValuesQuery": "", "tagsQuery": "", "type": "query", "useTags": false }, { "current": { "selected": true, "text": "", "value": "" }, "description": "总览表名称字段支持筛选，可以使用正则，如：.*aa.*bb.*", "hide": 0, "label": "查询", "name": "sname", "options": [ { "selected": true, "text": "", "value": "" } ], "query": "", "skipUrlSync": false, "type": "textbox" } ] }, "time": { "from": "now-1h", "to": "now" }, "timepicker": { "hidden": false, "now": true, "refresh_intervals": [ "15s", "30s", "1m", "5m", "15m", "30m" ], "time_options": [ "5m", "15m", "1h", "6h", "12h", "24h", "2d", "7d", "30d" ] }, "timezone": "browser", "title": "1 Node Exporter for Prometheus Dashboard CN 0413 ConsulManager自动同步版", "uid": "aka", "version": 15, "weekStart": "" } ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271700992.png#alt=) ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271701796.png#alt=) ### 自定义标签 ```shell [root@pr1 prometheus-2.35.0.linux-amd64]# grep -Ev "#|^$" prometheus.yml global: alerting: alertmanagers: - static_configs: - targets: rule_files: scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] - job_name: "node" static_configs: - targets: - 192.168.245.215:9100 - 192.168.245.216:9100 - 192.168.245.217:9100 labels: student: ljr # 浏览器打开9090，条件搜索node_disk_info{student="ljr"}，应该是有6条记录 ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204271829897.png#alt=) ### 服务发现 #### 基于文件的服务发现 ```shell [root@pr1 ~]# cd /opt/prometheus-2.35.0.linux-amd64 [root@pr1 targets]# grep -Ev "#|^$" /opt/prometheus-2.35.0.linux-amd64/prometheus.yml global: alerting: alertmanagers: - static_configs: - targets: rule_files: scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] - job_name: "node" file_sd_configs: - files: - targets/node*.yaml refresh_interval: 1m #每分钟检测一次 [root@pr1 prometheus-2.35.0.linux-amd64]# mkdir targets [root@pr1 prometheus-2.35.0.linux-amd64]# cd targets/ [root@pr1 targets]# grep -Ev "#|^$" /opt/prometheus-2.35.0.linux-amd64/targets/node.yaml - targets: - 192.168.245.215:9100 - 192.168.245.216:9100 - 192.168.245.217:9100 labels: student: ljr app: uplooking # 重启普罗米修斯服务 [root@pr1 targets]# cd .. [root@pr1 prometheus-2.35.0.linux-amd64]# kill -9 pid [root@pr1 prometheus-2.35.0.linux-amd64]# nohup ./prometheus --config.file="prometheus.yml" & # 浏览器打开IP地址:9090，等待扫描机子 ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204281140421.png#alt=) ### 与mysql监控起来 安装mysql 略 ```shell [root@pr1 opt]# wget https://github.com/prometheus/mysqld_exporter/releases/download/v0.14.0/mysqld_exporter-0.14.0.linux-amd64.tar.gz mysql> grant process,replication client,select on *.* to 'exporter'@'%' IDENTIFIED BY '123456'; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> FLUSH PRIVILEGES; Query OK, 0 rows affected (0.00 sec) ``` ```shell # 官网下载上传 [root@pr1 opt]# tar xvf mysqld_exporter-0.14.0.linux-amd64.tar.gz [root@pr1 opt]# cd mysqld_exporter-0.14.0.linux-amd64 [root@pr1 mysqld_exporter-0.14.0.linux-amd64]# ls LICENSE mysqld_exporter NOTICE [root@pr1 mysqld_exporter-0.14.0.linux-amd64]# grep -Ev "#|^$" my.cnf [client] host=192.168.245.215 user=exporter password=123456 ``` ```shell [root@pr1 mysqld_exporter-0.14.0.linux-amd64]# grep -Ev "#|^$" /opt/prometheus-2.35.0.linux-amd64/prometheus.yml global: alerting: alertmanagers: - static_configs: - targets: rule_files: scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] - job_name: "node" file_sd_configs: - files: - targets/node*.yaml refresh_interval: 1m - job_name: "mysql" file_sd_configs: - files: - targets/mysql*.yaml refresh_interval: 1m [root@pr1 mysqld_exporter-0.14.0.linux-amd64]# cd /opt/prometheus-2.35.0.linux-amd64 [root@pr1 prometheus-2.35.0.linux-amd64]# cd targets/ [root@pr1 targets]# cp node.yaml mysql.yaml [root@pr1 targets]# grep -Ev -Ev "#|^$" mysql.yaml - targets: - 192.168.245.215:9104 labels: databases: mysql version: 5.7 # 重启普罗米修斯服务 [root@pr1 targets]# cd .. [root@pr1 prometheus-2.35.0.linux-amd64]# kill -9 pid [root@pr1 prometheus-2.35.0.linux-amd64]# nohup ./prometheus --config.file="prometheus.yml" & [root@pr1 prometheus-2.35.0.linux-amd64]# cd /opt/mysqld_exporter-0.14.0.linux-amd64 [root@pr1 mysqld_exporter-0.14.0.linux-amd64]# kill -9 pid [root@pr1 mysqld_exporter-0.14.0.linux-amd64]# nohup ./mysqld_exporter --config.my-cnf="my.cnf" & ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204281444987.png#alt=) ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204281445422.png#alt=) ### 对target重新打标 ```shell [root@pr1 prometheus-2.35.0.linux-amd64]# grep -Ev -Ev "#|^$" prometheus.yml global: alerting: alertmanagers: - static_configs: - targets: rule_files: scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] - job_name: "node" file_sd_configs: - files: - targets/node*.yaml refresh_interval: 1m relabel_configs: #对页面的标签重新显示 - source_labels: - __scheme__ - __address__ - __metrics_path__ regex: "(http|https)(.*)" separator: "" target_label: "zero" replacement: "${1}://${2}" action: replace - job_name: "mysql" file_sd_configs: - files: - targets/mysql*.yaml refresh_interval: 1m # 重启普罗米修斯服务 [root@pr1 targets]# cd .. [root@pr1 prometheus-2.35.0.linux-amd64]# kill -9 pid [root@pr1 prometheus-2.35.0.linux-amd64]# nohup ./prometheus --config.file="prometheus.yml" & ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204281525945.png#alt=) ### consul 官网https://www.consul.io/ ```shell agent -dev:运行开发模式 agent -server：运行server模式 -ui：ui界面 -data-dir：数据位置 /etc/consul:可以以文件形式定义各个services的配置，也可以基于api接口直接配置 -client：监听地址 ``` ```shell # 单点模式 # 为了方便管理集群 [root@pr1 opt]# yum -y install unzip [root@pr1 opt]# unzip consul_1.12.0_linux_amd64.zip Archive: consul_1.12.0_linux_amd64.zip inflating: consul [root@pr1 opt]# mkdir /etc/consul [root@pr1 opt]# ./consul agent -dev -config-dir=/etc/consul -client=0.0.0.0 # 以开发者模式启动 # 其他模式参考这篇文章https://blog.csdn.net/sanxiaxugang/article/details/54576845 # 浏览器输入IP地址:8500，打开如下 ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204281718602.png#alt=) #### 单点 ```shell # 管理一个机子 [root@pr1 opt]# cd /etc/consul/ [root@pr1 consul]# grep -Ev "#|^$" node.json { "service": { "id": "node-prom", "name": "prometheus", "address": "192.168.245.215", "port": 9100, "tags": ["prometheus"], "checks": [{ "http": "http://192.168.245.215:9100/metrics", "interval": "5s" }] } } ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204281901887.png#alt=) #### 多点 ```shell # 管理多个机子 [root@pr1 ~]# cd /etc/consul [root@pr1 consul]# grep -Ev "#|^$" many.json { "services": [{ "id": "node-01", "name": "node-01", "address": "192.168.245.216", "port": 9100, "tags": ["nodes"], "checks": [{ "http": "http://192.168.245.216:9100/metrics", "interval": "5s" }] }, { "id": "node-02", "name": "node-02", "address": "192.168.245.217", "port": 9100, "tags": ["nodes"], "checks": [{ "http": "http://192.168.245.217:9100/metrics", "interval": "5s" }] }] } # 重启consul，浏览器查看是否成功 ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204282025566.png#alt=) #### 与prometheus相结合 ```shell [root@pr1 prometheus-2.35.0.linux-amd64]# grep -Ev "#|^$" prometheus.yml global: alerting: alertmanagers: - static_configs: - targets: rule_files: scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] - job_name: "mysql" file_sd_configs: - files: - targets/mysql*.yaml refresh_interval: 1m - job_name: "node" consul_sd_configs: - server: "192.168.75.30:8500" # 重启普罗米修斯服务 [root@pr1 targets]# cd .. [root@pr1 prometheus-2.35.0.linux-amd64]# kill -9 pid [root@pr1 prometheus-2.35.0.linux-amd64]# nohup ./prometheus --config.file="prometheus.yml" & # 浏览器打开192.168.245.215:9090查看机子，如下图 # 再打开192.168.245.215:9107查看，如下图 ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204282100338.png#alt=) ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204282116692.png#alt=) ```shell # 解决问题 [root@pr1 ~]# cd /opt/ [root@pr1 opt]# wget https://github.com/prometheus/consul_exporter/releases/download/v0.8.0/consul_exporter-0.8.0.linux-amd64.tar.gz [root@pr1 opt]# tar xvf consul_exporter-0.8.0.linux-amd64.tar.gz [root@pr1 opt]# cd consul_exporter-0.8.0.linux-amd64 [root@pr1 consul_exporter-0.8.0.linux-amd64]# ./consul_exporter # 修改prometheus配置文件 [root@pr1 prometheus-2.35.0.linux-amd64]# grep -Ev "#|^$" prometheus.yml global: alerting: alertmanagers: - static_configs: - targets: rule_files: scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] - job_name: "mysql" file_sd_configs: - files: - targets/mysql*.yaml refresh_interval: 1m - job_name: "node" consul_sd_configs: - server: "192.168.245.215:8500" - job_name: "consul-exporter" static_configs: - targets: ["localhost:9107"] # 重启 [root@pr1 targets]# cd .. [root@pr1 prometheus-2.35.0.linux-amd64]# nohup ./prometheus --config.file="prometheus.yml" & # 浏览器打开192.168.245.215:9090查看机子，如下图 # 过滤掉那个down的，留下先监控的consul # 过滤教程(drop) # https://mutoulazy.github.io/2019/06/28/kubernetes/prometheus/prometheus-drop-metrics/#%E8%BF%87%E6%BB%A4prometheus-opretor%E4%B8%ADServicemonit%E9%85%8D%E7%BD%AE%E7%9A%84job%E6%8C%87%E6%A0%87 ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204290903541.png#alt=) ```shell { "service": { "id": "node-prom", "name": "prometheus", "address": "192.168.245.215", "port": 8300, "tags": ["prometheus"], "checks": [{ "http": "http://192.168.245.215:8/metrics", "interval": "5s" }] } } ``` #### 三种发现方式 静态发现、服务发现、文件发现 ### 通知媒介 时间散列，与zabbix不一样，不是一到那个值就报警；而是分析那个时间趋势是否会达到那个值 ```shell # 下载alertmanager [root@pr1 opt]# tar xvf alertmanager-0.24.0.linux-amd64.tar.gz [root@pr1 opt]# cd alertmanager-0.24.0.linux-amd64 [root@pr1 alertmanager-0.24.0.linux-amd64]# ls alertmanager alertmanager.yml amtool LICENSE NOTICE [root@pr1 alertmanager-0.24.0.linux-amd64]# cp alertmanager.yml alertmanager.yml.bak [root@pr1 alertmanager-0.24.0.linux-amd64]# >alertmanager.yml [root@pr1 alertmanager-0.24.0.linux-amd64]# vim alertmanager.yml [root@pr1 alertmanager-0.24.0.linux-amd64]# grep -Ev "#|^$" alertmanager.yml global: resolve_timeout: 5m route: group_by: ['altername'] group_wait: 10s group_interval: 10s repeat_interval: 10s receiver: 'emial-me' receivers: - name: 'emial-me' email_configs: - to: '429496374@qq.com' from: '429496374@qq.com' smarthost: 'smtp.qq.com:465' auth_username: '429496374@qq.com' auth_identity: '429496374@qq.com' auth_password: 'xxxxxxxxx' # 授权码 require_tls: false ``` ```shell [root@pr1 alertmanager-0.24.0.linux-amd64]# cd /opt/prometheus-2.35.0.linux-amd64 [root@pr1 prometheus-2.35.0.linux-amd64]# grep -Ev "#|^$" prometheus.yml global: alerting: alertmanagers: - static_configs: - targets: - 192.168.245.215:9093 rule_files: - "rules/*.yaml" scrape_configs: - job_name: "prometheus" static_configs: - targets: ["localhost:9090"] - job_name: "node" file_sd_configs: - files: - targets/node*.yaml refresh_interval: 1m relabel_configs: - source_labels: - __scheme__ - __address__ - __metrics_path__ regex: "(http|https)(.*)" separator: "" target_label: "zero" replacement: "${1}://${2}" action: replace [root@pr1 prometheus-2.35.0.linux-amd64]# mkdir rules [root@pr1 prometheus-2.35.0.linux-amd64]# cd rules/ [root@pr1 rules]# grep -Ev "#|^$" node.yaml groups: - name: AllInstances rules: - alert: InstanceDown expr: up == 0 for: 20s annotations: title: 'Instance down' description: 'Instance has been down for more than 20s.' labels: severity: 'critical' # 启动alertmanager [root@pr1 rules]# cd /opt/alertmanager-0.24.0.linux-amd64 [root@pr1 alertmanager-0.24.0.linux-amd64]# nohup ./alertmanager & [2] 1718 [root@pr1 alertmanager-0.24.0.linux-amd64]# nohup: 忽略输入并把输出追加到"nohup.out" [root@pr1 alertmanager-0.24.0.linux-amd64]# # 浏览器打开IP地址:9093，如下图 # 重启prometheus [root@pr1 prometheus-2.35.0.linux-amd64]# kill -9 pid [root@pr1 prometheus-2.35.0.linux-amd64]# nohup ./prometheus --config.file="prometheus.yml" & [3] 1730 [root@pr1 prometheus-2.35.0.linux-amd64]# nohup: 忽略输入并把输出追加到"nohup.out" # 回到浏览器IP地址:9090，查看是否有警告监控，如下图 ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204291110732.png#alt=) ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204291110408.png#alt=) ```shell # 模拟故障去对应页面可以看到都判断红色，邮箱也收到邮件，每20s发送一次邮件 ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202204291141213.png#alt=) ### PromQL介绍 ```tex Prometheus提供了一种名为PromQL(Prometheus查询语言)的函数式查询语言，允许用户实时选择和聚合时间序列数据。表达式的结果既可以显示为图形，也可以在Prometheus的表达式浏览器中作为表格数据查看，或者通过HTTPAPI由外部系统使用。 ``` ```tex 运算： 乘：* 除：/ 加：+ 减：- 常用函数： sum()函数：求出找到所有value的值irate()函数：统计平均速率 by(标签名) 范围匹配 #5分钟之内 [5m] ``` #### 查询指定mertic_name ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202205041623804.png#alt=image-20220504162315713) ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202205041624025.png#alt=) #### 带标签的查询 ```shell node_cpu_seconds_total{instance="192.168.153.144:9100"} ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202205041625657.png#alt=) #### 多标签查询 ```shell node_cpu_seconds_total{instance="192.168.153.144:9100", mode="system"} ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202205041625989.png#alt=img) #### 计算CPU使用率 ```shell # 从里看到外，一个括号一个括号的看 100 - (avg(irate(node_cpu_seconds_total{mode="idle"}[5m])) by (instance) * 100) ``` ![](https://figure-bed-1304788733.cos.ap-guangzhou.myqcloud.com/typora/202205041627202.png#alt=)