Monitoring platform
Monitoring instructions
In the past deployment and implementation of enterprise projects, we found that most customers have their own independent monitoring platform, so we did not integrate the monitoring platform into our environment. However, some users lack this environment,If you For monitoring requirements, we provide you with a simple and visual monitoring platform. You can refer to the following methods for installation
Monitoring indicators
Hardware monitoring
| name | index | normal | warning | abnormal |
|---|---|---|---|---|
| CPU | Utilization rate | < 80% | 80%-90% | > 90% |
| Memory | Utilization rate | < 80% | 80%-90% | > 90% |
| disk | Utilization rate | < 70% | 70%-80% | > 80% |
| network | Bandwidth (KB) | > 512 | >= 100 | < 50 |
| Time error | second | <= 15 | <= 30 | >30 |
Middleware monitoring
| name | index | normal | warning | abnormal |
|---|---|---|---|---|
| mysql | node | MySQL Cluster is normal | MySQL exception 2 nodes remaining | MySQL Cluster failure |
| redis | node | Redis cluster is normal | Redis exception 2 nodes remaining | Redis cluster failure |
| rabbitmq | node | Rabbitmq cluster is normal | Rabbitmq exception 2 remaining nodes | Rabbitmq cluster failure |
| minio | node | Minio cluster is normal | Minio exception 2 nodes remaining | Minio cluster failure |
| elasticsearch | node | Elasticsearch cluster is normal | Elasticsearch exception 2 nodes remaining | Elasticsearch cluster failure |
K8s node
| name | index | normal | warning | abnormal |
|---|---|---|---|---|
| node | Number of failed nodes | 0% | 33% | 66% |
| pod | Pod status | normal | restart | Non running |
Monitoring installation
tip
Since the monitoring files are stored in the module deployment package, you need to install the monitoring service on the machine that performs the deployment operation at that time
Input: /usr/local/python-3.7.6/bin/laipvt deploy --monitor
Output:
[INFO] 2022-05-05 14:29:38,056 util.py [line:345] modify_umask: 主机: 172.18.242.18, 执行命令: sed -i '$aumask 0022 #laiye' /etc/bashrc
[INFO] 2022-05-05 14:29:38,465 util.py [line:287] wrapper: 即将运行middleware下的deploy_monitor方法
[DEBUG] 2022-05-05 14:29:38,467 template.py [line:19] _fill: 渲染/root/checker-XerbzmNpb/XerbzmNpb/middleware/monitor/config/dashboards.yaml文件到/tmp/dashboards.yaml
[DEBUG] 2022-05-05 14:29:38,469 template.py [line:19] _fill: 渲染/root/checker-XerbzmNpb/XerbzmNpb/middleware/monitor/config/datasources.yaml文件到/tmp/datasources.yaml
[DEBUG] 2022-05-05 14:29:38,470 template.py [line:19] _fill: 渲染/root/checker-XerbzmNpb/XerbzmNpb/middleware/monitor/config/prometheus.yaml文件到/tmp/prometheus.yaml
[DEBUG] 2022-05-05 14:29:38,476 template.py [line:19] _fill: 渲染/root/checker-XerbzmNpb/XerbzmNpb/middleware/monitor/config/mysql_datasources.yaml文件到/tmp/mysql_datasources.yaml
[DEBUG] 2022-05-05 14:29:38,477 template.py [line:19] _fill: 渲染/root/checker-XerbzmNpb/XerbzmNpb/middleware/monitor/config/k8s_datasources.yaml文件到/tmp/k8s_datasources.yaml
[DEBUG] 2022-05-05 14:29:38,479 template.py [line:19] _fill: 渲染/root/checker-XerbzmNpb/XerbzmNpb/middleware/monitor/prometheus_istio.yaml文件到/tmp/prometheus_istio.yaml
[INFO] 2022-05-05 14:29:38,483 middlewareinterface.py [line:82] send_config_file: 分发文件/tmp/dashboards.yaml到172.18.242.18:/home/rpa/monitor/conf/dashboards.yaml
[INFO] 2022-05-05 14:29:39,132 middlewareinterface.py [line:82] send_config_file: 分发文件/tmp/prometheus.yaml到172.18.242.18:/home/rpa/monitor/conf/prometheus.yaml
[INFO] 2022-05-05 14:29:39,618 middlewareinterface.py [line:82] send_config_file: 分发文件/tmp/datasources.yaml到172.18.242.18:/home/rpa/monitor/conf/datasources.yaml
[INFO] 2022-05-05 14:29:40,118 middlewareinterface.py [line:82] send_config_file: 分发文件/tmp/k8s_datasources.yaml到172.18.242.18:/home/rpa/monitor/conf/k8s_datasources.yaml
[INFO] 2022-05-05 14:29:40,567 middlewareinterface.py [line:82] send_config_file: 分发文件/tmp/mysql_datasources.yaml到172.18.242.18:/home/rpa/monitor/conf/mysql_datasources.yaml
[INFO] 2022-05-05 14:29:41,064 middlewareinterface.py [line:82] send_config_file: 分发文件/root/checker-XerbzmNpb/XerbzmNpb/middleware/monitor/json到172.18.242.18:/home/rpa/monitor/json
[INFO] 2022-05-05 14:29:41,635 middlewareinterface.py [line:82] send_config_file: 分发文件/tmp/prometheus_istio.yaml到172.18.242.18:/home/rpa/monitor/prometheus_istio.yaml
[DEBUG] 2022-05-05 14:29:42,132 template.py [line:19] _fill: 渲染/root/checker-XerbzmNpb/XerbzmNpb/middleware/monitor/monitor.tmpl文件到/tmp/docker-compose-monitor.yaml
[INFO] 2022-05-05 14:29:42,139 middlewareinterface.py [line:71] send_docker_compose_file_hosts: 分发文件/tmp/docker-compose-monitor.yaml到172.18.242.18:/home/rpa/monitor/docker-compose.yml
[INFO] 2022-05-05 14:29:42,637 middlewareinterface.py [line:111] start_all_node: 启动172.18.242.18 monitor服务
[INFO] 2022-05-05 14:29:54,114 monitor.py [line:71] _install_prometheus: ['kubectl apply -f /home/rpa/monitor/prometheus_istio.yaml', 'helm --host=localhost:44134 install --name=k8s-metrics --set replicaCount=1 --set image.repository=172.18.242.18:8889/middleware/kube-state-metrics --set image.tag=latest /root/checker-XerbzmNpb/XerbzmNpb/middleware/monitor/kube-state-metrics']
After installation, it will be enabled by default3000The port is used as the login address of the monitoring page. The default user name and password areadmin/admin
Single node： Follow theSingle node global monitoring panel
Highly available： Follow theHighly available global monitoring panel
If you are not a technician, you only need to pay attention to the colors in the page
| green | normal |
|---|---|
| yellow | warning |
| gules | abnormal |
