1 Prometheus是什么

摘自Prometheus官网：

From metrics to insight

Power your metrics and alerting with a leading open-source monitoring solution.

翻译过来就是，Prometheus是一个领先的开源监控解决方案

2 Prometheus特点

2.1 高维数据模型（Highly Dimensional Data Model）及时序列（Time Series）

Prometheus把所有的数据按时序列进行存储，所有采集上来的数据在都被打了时间戳并按时间先后顺序进行流化，这些数据属于相同的指标名以及一组标签维度（labeled dimensions）

2.1.1 时序列

A time series is a series of data points indexed (or listed or graphed) in time order.

时序列是按照时间顺序排列的一系列数据点

2.1.2 指标名（Metric names）和标签（Labels）

每个时序列被唯一识别为它的一个指标名（metric name）以及一组键值对（key-value pairs），键值对就是我们所说的标签（labels）

指标名指定了被测量系统的一个常规特征，比如：http_requests_total - 收到的HTTP请求总数。指标名的命名必须符合正则表达式规则[a-zA-Z_:][a-zA-Z0-9_:]*

标签使Prometheus的维度数据模型成为可能，对于一个指标名给定任意一组标签的组合都能标识一个特定的指标维度实例（particular dimensional instantiation of that metric），例如：通过POST方法访问 /api/tracks 的HTTP请求。标签名的命名必须符合正则表达式规则[a-zA-Z_][a-zA-Z0-9_]*

2.1.3 记号（Notation）

给定一个指标名和一组标签，时序列通过Notation被标记：

 <metric name>{<label name>=<label value>, ...}

例如，一个指标名为 api_http_requests_total 且标签为 method="POST" 和 handler="/messages" 的时序列可以写成：

api_http_requests_total{method="POST", handler="/messages"}

2.2 PromQL

建立在高维数据模型上的查询语言，这里暂不展开

2.3 高效存储

Prometheus存储时序列数据于内存和本地磁盘中，不依赖分布式存储，单节点工作。扩展通过功能分片和联邦来实现

2.4 可视化效果出众

通过与Grafana集成，能够为使用者提供非常直观且漂亮的可视化效果

2.5 通过拉取方式采集数据，或者通过中间网关推送方式采集数据

2.6 通过服务发现或者静态配置来发现监控目标

3 架构及组件说明

3.1 架构图

image1

3.2 组件说明

3.2.1 Prometheus Server

负责数据采集和存储，提供PromQL查询语言的支持

3.2.2 Push gateway

负责支持short-lived jobs，push gateway能够让临时（ephemeral）或批处理job暴露他们的指标給Prometheus。因为临时和批处理job很可能并不长期存在，所以Prometheus无法抓到相应的数据，取而代之，我们让这样的job把指标数据主动推送给push gateway，之后push gateway再把这些指标数据暴露给Prometheus。push gateway就像一个指标缓存，并不负责计算。

3.2.3 Exporters

Exporters帮助把第三方系统的既有指标输出为Prometheus指标。
我们可以在 exporter default port wiki上查看Exporters的列表，也可以在
EXPORTERS AND INTEGRATIONS上查看列表

3.2.4 Alertmanager

Alertmanager负责处理从客户端应用（例如：Prometheus）发送过来的警报，把接受到的信息去重、分组，并把他们路由到正确的接收器，如PagerDuty, OpsGenie。Alertmanager还负责警报信息的消声或抑制

4 Prometheus优缺点

Prometheus对于采集纯数字值的时间序列非常在行，所以它既适合以物理机为中心的监控，也适合监控高度动态的面向服务的架构体。在微服务领域，它的多维数据采集以及查询非常独到且很有竞争力。
Prometheus最大的价值在于可靠性，用户可以在任何时候看到整个被监控系统的统计信息，即使在系统有问题的时候。但它不能做到100%的精确，比如如果你要按request数计费，那么Prometheus未必能采集到所有的请求，这个时候Prometheus就不太合适了。

5 Prometheus与Kubernetes

Prometheus是Kubernetes的近亲，Google公布的Kubernetes派生于他们的Borg集群系统，而Prometheus与Borgmon共享基础设计概念， Borgmon是与Borg的监控系统。现在Prometheus和kubernetes都被云原生计算基金会（CNCF）所掌管。技术层面上Kubernetes会把它内部的指标数据以Prometheus可以接受的格式暴露出来。

5.1 Prometheus与Kubernetes集成的方式

Prometheus Operator
kube-prometheus
kubernetes addon

5.1.1 Prometheus Operator

Operator是CoreOS引入的一种操作其他软件，并把人们收集到的操作知识集成到部署过程的软件。
Prometheus Operator可以方便的让用户安装Prometheus，并用简单的声明式配置来管理和配置Prometheus实例。其核心思想在于解耦Prometheus实例的部署与针对被监控实体的配置，使Prometheus运行在Kubernetes上的步骤尽可能的简单。

Prometheus向Kubernetes中引入了额外的资源用于声明期望Prometheus及Alertmanager集群所达到的状态，这些资源包括：

Prometheus
Alertmanager
ServiceMonitor
PrometheusRule

Altermanager不在本文范围之内，以后的文章单独陈述。

Prometheus资源声明式的描述了部署Prometheus部署时所期望达到的状态，而ServiceMonitor描述了一组被Prometheus所监控的目标。

image2

上图的Operator用来确保在任意时间对于每个处于Kubernetes集群中的Prometheus资源都有一组按照期望配置的Prometheus Server在运行。每个Prometheus实例又与各自的配置绑定在一起，这些配置指定了该监视哪些目标从而抓取指标。
用户可以手动指定这些配置，或者让Operator基于ServiceMonitor生成出来。ServiceMonitor资源指定如何从一组服务中获取指标。而Prometheus资源对象可以通过标签（labels）动态的引入ServiceMonitor对象，Operator设置Prometheus实例来监控所有被ServiceMonitor所覆盖的服务并保持配置与集群中的变化同步。

5.1.2 kube-prometheus

kube-prometheus把Prometheus Operator和一系列manifests结合起来，帮助用户从监控Kubernetes自身及跑在上面的应用开始，提供全栈的监控配置。

6 部署Prometheus

6.1 部署环境

序号	节点名	角色	内存	IP	版本
1	master.mike.com	master	2GB	192.168.56.101	v1.13.4
2	node1.mike.com	node	2GB	192.168.56.102	v1.13.4
3	node2.mike.com	node	2GB	192.168.56.103	v1.13.4

6.2 通过kube-prometheus快速部署Prometheus

6.2.1 前提

准备好一个kubernetes集群，这里参见章节6.1
确认以下flag在kubernetes集群中被设置好，目的在于告诉kubelet使用token来认证及鉴权，这可以允许更细粒度及更简单的访问控制：
- ---authentication-token-webhook=true
- --authorization-mode=Webhook
因为kube-prometheus已经包含了资源指标API服务（resource metrics API server），这与metrics-server功能相同，所以，如果kubernetes集群要是已经部署了metrics-server，则先卸载掉metrics-server，如果没有，则略过此项。

6.2.2 克隆kube-prometheus仓库

[root@master ~]# git clone https://github.com/coreos/kube-prometheus.git
Cloning into 'kube-prometheus'...
remote: Enumerating objects: 49, done.
remote: Counting objects: 100% (49/49), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 5763 (delta 19), reused 19 (delta 2), pack-reused 5714
Receiving objects: 100% (5763/5763), 3.72 MiB | 714.00 KiB/s, done.
Resolving deltas: 100% (3396/3396), done.

6.2.3 快速部署监控栈

笔者的网络环境位于围墙之外，所以当通过资源文件创建kubernetes中的各种资源时，kubernetes自动的从各种库中下载docker所需要的镜像文件而未受到任何的阻碍，如果部署的环境位于墙内，则需要预先下载好所有资源文件中涉及的镜像文件到集群中的节点上。

创建资源

[root@master kube-prometheus]# kubectl create -f manifests/
namespace/monitoring created
customresourcedefinition.apiextensions.k8s.io/alertmanagers.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheuses.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/prometheusrules.monitoring.coreos.com created
customresourcedefinition.apiextensions.k8s.io/servicemonitors.monitoring.coreos.com created
clusterrole.rbac.authorization.k8s.io/prometheus-operator created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-operator created
deployment.apps/prometheus-operator created
service/prometheus-operator created
serviceaccount/prometheus-operator created
servicemonitor.monitoring.coreos.com/prometheus-operator created
alertmanager.monitoring.coreos.com/main created
secret/alertmanager-main created
service/alertmanager-main created
serviceaccount/alertmanager-main created
servicemonitor.monitoring.coreos.com/alertmanager created
secret/grafana-datasources created
configmap/grafana-dashboard-k8s-cluster-rsrc-use created
configmap/grafana-dashboard-k8s-node-rsrc-use created
configmap/grafana-dashboard-k8s-resources-cluster created
configmap/grafana-dashboard-k8s-resources-namespace created
configmap/grafana-dashboard-k8s-resources-pod created
configmap/grafana-dashboard-k8s-resources-workload created
configmap/grafana-dashboard-k8s-resources-workloads-namespace created
configmap/grafana-dashboard-nodes created
configmap/grafana-dashboard-persistentvolumesusage created
configmap/grafana-dashboard-pods created
configmap/grafana-dashboard-statefulset created
configmap/grafana-dashboards created
deployment.apps/grafana created
service/grafana created
serviceaccount/grafana created
servicemonitor.monitoring.coreos.com/grafana created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
role.rbac.authorization.k8s.io/kube-state-metrics created
rolebinding.rbac.authorization.k8s.io/kube-state-metrics created
service/kube-state-metrics created
serviceaccount/kube-state-metrics created
servicemonitor.monitoring.coreos.com/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/node-exporter created
clusterrolebinding.rbac.authorization.k8s.io/node-exporter created
daemonset.apps/node-exporter created
service/node-exporter created
serviceaccount/node-exporter created
servicemonitor.monitoring.coreos.com/node-exporter created
apiservice.apiregistration.k8s.io/v1beta1.metrics.k8s.io created
clusterrole.rbac.authorization.k8s.io/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/system:aggregated-metrics-reader created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-adapter created
clusterrolebinding.rbac.authorization.k8s.io/resource-metrics:system:auth-delegator created
clusterrole.rbac.authorization.k8s.io/resource-metrics-server-resources created
configmap/adapter-config created
deployment.apps/prometheus-adapter created
rolebinding.rbac.authorization.k8s.io/resource-metrics-auth-reader created
service/prometheus-adapter created
serviceaccount/prometheus-adapter created
clusterrole.rbac.authorization.k8s.io/prometheus-k8s created
clusterrolebinding.rbac.authorization.k8s.io/prometheus-k8s created
prometheus.monitoring.coreos.com/k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s-config created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
rolebinding.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s-config created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
role.rbac.authorization.k8s.io/prometheus-k8s created
prometheusrule.monitoring.coreos.com/prometheus-k8s-rules created
service/prometheus-k8s created
serviceaccount/prometheus-k8s created
servicemonitor.monitoring.coreos.com/prometheus created
servicemonitor.monitoring.coreos.com/kube-apiserver created
servicemonitor.monitoring.coreos.com/coredns created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
servicemonitor.monitoring.coreos.com/kube-scheduler created
servicemonitor.monitoring.coreos.com/kubelet created

检查资源就绪状况

[root@master kube-prometheus]# until kubectl get customresourcedefinitions servicemonitors.monitoring.coreos.com ; do date; sleep 1; echo ""; done
NAME                                    CREATED AT
servicemonitors.monitoring.coreos.com   2019-05-13T05:38:41Z

[root@master kube-prometheus]# until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done
NAMESPACE    NAME                      AGE
monitoring   alertmanager              28s
monitoring   coredns                   25s
monitoring   grafana                   27s
monitoring   kube-apiserver            25s
monitoring   kube-controller-manager   25s
monitoring   kube-scheduler            25s
monitoring   kube-state-metrics        27s
monitoring   kubelet                   25s
monitoring   node-exporter             27s
monitoring   prometheus                25s
monitoring   prometheus-operator       28s

7 访问Prometheus

通过kube-Prometheus在kubernetes集群中部署的Prometheus，默认情况下，只能在集群内部访问，其建立起来的所有service的类型（type）都是ClusterIP:

[root@master kube-prometheus]# kubectl get svc -n monitoring
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
alertmanager-main       ClusterIP   10.102.26.246    <none>        9093/TCP            84m
alertmanager-operated   ClusterIP   None             <none>        9093/TCP,6783/TCP   83m
grafana                 ClusterIP   10.111.252.205   <none>        3000/TCP            84m
kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP   84m
node-exporter           ClusterIP   None             <none>        9100/TCP            84m
prometheus-adapter      ClusterIP   10.100.63.156    <none>        443/TCP             84m
prometheus-k8s          ClusterIP   10.108.202.31    <none>        9090/TCP            83m
prometheus-operated     ClusterIP   None             <none>        9090/TCP            83m
prometheus-operator     ClusterIP   None             <none>        8080/TCP            84m

如果想在集群外部访问Prometheus，就需要把服务暴露给外界，下面提供具体的方式

7.1 通过修改NodePort从外界访问Promethus

7.1.1 修改prometheus-k8s服务的类型

编辑prometheus-service.yaml文件，修改为如下内容（注意nodePort及type属性）：

apiVersion: v1
kind: Service
metadata:
  labels:
    prometheus: k8s
  name: prometheus-k8s
  namespace: monitoring
spec:
  ports:
  - name: web
    nodePort: 32090
    port: 9090
    targetPort: web
  selector:
    app: prometheus
    prometheus: k8s
  sessionAffinity: ClientIP
  type: NodePort

应用prometheus-service.yaml

[root@master manifests]# kubectl apply -f prometheus-service.yaml --force
service/prometheus-k8s created

检查prometheus-k8s service修改后的类型及对应的nodePort:

[root@master kube-prometheus]# kubectl get svc -n monitoring
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
alertmanager-main       ClusterIP   10.102.26.246    <none>        9093/TCP            139m
alertmanager-operated   ClusterIP   None             <none>        9093/TCP,6783/TCP   139m
grafana                 ClusterIP   10.111.252.205   <none>        3000/TCP            139m
kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP   139m
node-exporter           ClusterIP   None             <none>        9100/TCP            139m
prometheus-adapter      ClusterIP   10.100.63.156    <none>        443/TCP             139m
prometheus-k8s          NodePort    10.96.91.11      <none>        9090:32090/TCP      19s
prometheus-operated     ClusterIP   None             <none>        9090/TCP            138m
prometheus-operator     ClusterIP   None             <none>        8080/TCP            139m

7.1.2 通过nodePort及节点IP访问Promethus

我们已知master节点的IP为192.168.56.101，且在上面我们修改nodePort为32090，所以从集群外的节点可以访问地址：http://192.168.56.101:32090

首页如图：

Image3.png

到这里，其实我们已经可以通过Prometheus的界面以及PromQL来查询我们已经获取的所有指标信息了，当然Prometheus提供的界面相对简陋，功能比较简单，所以我们可以继续下面的操作，通过访问已经部署好的Granfana来以更加美观的各种图标来展示Prometheus中获得的数据。

7.2 通过nodePort及节点IP访问Grafana

7.2.1 修改grafana服务的类型

编辑grafana-service.yaml文件，修改为如下内容（注意nodePort及type属性）：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  ports:
  - name: http
    port: 3000
    targetPort: http
    nodePort: 32030
  selector:
    app: grafana
  type: NodePort

应用grafana-service.yaml

[root@master manifests]# kubectl apply -f grafana-service.yaml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
service/grafana configured

检查grafana service修改后的类型及对应的nodePort:

[root@master manifests]# kubectl get svc -n monitoring
NAME                    TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
alertmanager-main       ClusterIP   10.102.26.246    <none>        9093/TCP            174m
alertmanager-operated   ClusterIP   None             <none>        9093/TCP,6783/TCP   173m
grafana                 NodePort    10.111.252.205   <none>        3000:32030/TCP      174m
kube-state-metrics      ClusterIP   None             <none>        8443/TCP,9443/TCP   174m
node-exporter           ClusterIP   None             <none>        9100/TCP            174m
prometheus-adapter      ClusterIP   10.100.63.156    <none>        443/TCP             174m
prometheus-k8s          NodePort    10.96.91.11      <none>        9090:32090/TCP      35m
prometheus-operated     ClusterIP   None             <none>        9090/TCP            173m
prometheus-operator     ClusterIP   None             <none>        8080/TCP            174m

7.2.2 通过nodePort及节点IP访问grafana

我们已知master节点的IP为192.168.56.101，且在上面我们修改nodePort为32030，所以从集群外的节点可以访问地址：http://192.168.56.101:32030 (默认用户名/密码：admin:admin）