dashboard
监控项：
- 各个k8s集群所有pod not running的状态，监控pod的CrashLoopBackOff及一直处于ContainerCreating的状态，可通过grafana做告警
-
监控pod的重启次数
k8s-pod-status.png
TechStack
promtheus+grafana
PromQL
datasource接入prometheus，PromQL语句对应上图各项title
- Container Waiting Reason
(sum(kube_pod_container_status_waiting_reason{reason!="ContainerCreating",namespace=~"$namespace",pod=~"$pod"} ) by (reason,namespace,pod) >0)
*on(pod) group_right(reason) sum(kube_pod_info) by (pod,node,host_ip,pod_ip,namespace)
or
(sum(kube_pod_container_status_waiting_reason{reason="ContainerCreating",namespace=~"$namespace",pod=~"$pod"} ) by (reason,namespace,pod) >0)
-on(pod) group_right(reason) sum(kube_pod_info) by (pod,node,host_ip,pod_ip,namespace)
- pod重启次数(Last 5m)
(sum(kube_pod_container_status_restarts_total{namespace=~"$namespace",pod=~"$pod"}) by(namespace,pod) *on(pod) group_right() sum(kube_pod_info) by (pod,node,host_ip,pod_ip,namespace)
-sum(kube_pod_container_status_restarts_total{namespace=~"$namespace",pod=~"$pod"} offset 5m) by(namespace,pod) *on(pod) group_right() sum(kube_pod_info) by (pod,node,host_ip,pod_ip,namespace))
