PromQL

空闲的cpu指标

node_cpu_seconds_total{instance="192.168.153.21:9100",mode="idle"}

除空闲之外的其他指标

node_cpu_seconds_total{instance="192.168.153.21:9100",mode!="idle"}

192.168.153.21 的cpu使用率

100-(avg(irate(node_cpu_seconds_total{instance="192.168.153.21:9100",mode="idle"}[5m]))*100)

标签管理

自定义标签

  - job_name: 'Linux Server'
    static_configs:
    - targets: ['192.168.153.21:9100']
      labels:
        project: k8s
        type: monitor

1635734900648.png

重命名标签

  - job_name: 'BJLinux Server'
    static_configs:
    - targets: ['192.168.153.22:9100']
      labels:
        add: bj
    metric_relabel_configs:
    - action: replace                                    
      source_labels: ['instance']
      regex: (.*):([0-9]+)
      replacement: $1
      target_label: "ip"

1635733160764.png

过滤Target

 

  - job_name: 'Linux Server'
    static_configs:
    - targets: ['192.168.153.21:9100','192.168.153.22:9100']
      labels:
        project: k8s
        type: monitor
    relabel_configs:
      - action: drop
        regex: "192.168.153.22:9100"
        source_labels: ["__address__"]
        
-------------------------------------------------------------------
#查询验证，192.168.153.22:9100信息将不再显示
node_cpu_seconds_total{project="k8s"}

1635734900648.png

删除标签

  - job_name: 'SHLinux Server'
    static_configs:
    - targets: ['192.168.153.21:9100']
      labels:
        add: sh
    relabel_configs:
    - action: labeldrop
      regex: "job"

-------------------------------------------------------------------
#查询验证，job信息信息将不再显示
node_cpu_seconds_total{add="sh"}

1635735782180.png

1635735834447.png

Grafana

仪表盘

1635821014977.png

导航栏（job）

1635821510036.png

label_values(up,job)

Preview of values：
BJLinux Server
kubernetes-service-endpoints
kubernetes-nodes-cadvisor
Linux Server
prometheus

导航栏（节点）

1635821894512.png

1635822164638.png

(up{job="$job"},instance)
#每个job有不同节点

cpu核数

1635822689574.png

count(node_cpu_seconds_total{instance="$node",job="$job",mode="idle"})
组件：stat

内存总量

1635823252179.png

node_memory_MemTotal_bytes{instance="$node",job="$job"}
组件：stat

内存使用率

1635823654918.png

1635838490518.png

100 - (avg(irate(node_cpu_seconds_total{instance="$node",mode="idle"}[5m])) by (instance) * 100)

内存使用率

1635838734374.png

100 - (node_memory_MemFree_bytes+node_memory_Cached_bytes+node_memory_Buffers_bytes) / node_memory_MemTotal_bytes * 100 

100 - (node_memory_MemFree_bytes{instance="$node",job="$job"}+node_memory_Cached_bytes{instance="$node",job="$job"}+node_memory_Buffers_bytes{instance="$node",job="$job"}) / node_memory_MemTotal_bytes{instance="$node",job="$job"} * 100 

根分区使用率

1635839249336.png

100 - (node_filesystem_free_bytes{fstype=~"ext4|xfs",mountpoint="/"} / node_filesystem_size_bytes{fstype=~"ext4|xfs",mountpoint="/"} * 100)


100 - (node_filesystem_free_bytes{fstype=~"ext4|xfs",mountpoint="/",instance="$node",job="$job"} / node_filesystem_size_bytes{fstype=~"ext4|xfs",mountpoint="/",instance="$node",job="$job"} * 100)