一. PV与PVC的关系
• PersistenVolume(PV): 对存储资源创建和使用的抽象，使得存储作为集群中的资源管理
• PersistentVolumeClaim(PVC): 让用户不需要关心具体的Volume实现细节
PV是提供容量，PVC是消费容量，消费这个过程称为绑定
PV分为：静态和动态
二. PV静态供给及应用案例
1. 创建PV
创建名为my-pv的PV，使用的是网络存储NFS
# cat pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: my-pv
spec:
capacity:
storage: 5Gi
accessModes:
- ReadWriteMany
nfs:
path: /data/nfs/www
server: 10.40.6.214
注意：10.40.6.214 这台nfs服务器必须存在/data/nfs/www目录，否则容器启动失败
# kubectl create -f pv.yaml
# kubectl get pv
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
my-pv 5Gi RWX Retain Available 48s
2. 创建PVC
创建PVC也就是卷需求模板, 这yaml配置建议和创建Deployment pod的yaml 文件统一到一个文件，使用"---"分割即可
# cat pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: my-pvc
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 5Gi
# kubectl create -f pvc.yaml
# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/my-pvc Bound my-pv 5Gi RWX 8m47s
3. PV与PVC绑定
启动容器应用PVC，这个消费过程称为绑定，绑定是否成功主要有三要素：名称、容量和访问模式。
# cat pod-pvc.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: nginx-deployment
spec:
replicas: 2
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
- name: nginx
image: 10.40.6.165/library/nginx:v1
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
ports:
- containerPort: 80
volumes:
- name: www
persistentVolumeClaim:
claimName: my-pvc
# kubectl create -f pod-pvc.yaml
# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/my-pv 5Gi RWX Retain Bound default/my-pvc 12m
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/my-pvc Bound my-pv 5Gi RWX 8m47s
# kubectl get pod
Kubernetes支持持久卷的存储插件:
https://kubernetes.io/docs/concepts/storage/persistent-volumes/
三. PV动态供给及应用案例
Dynamic Provisioning 机制工作的核心在于StorageClass的API对象。
StorageClass 声明存储插件，用于自动创建PVC。
1. 创建PV动态供给流程
image.png
NFS 默认是不支持PV动态供给，需要插件完成, 这插件也是一个pod
Kubernetes支持动态供给的存储插件:
https://kubernetes.io/docs/concepts/storage/storage-classes/
NFS 动态供给存储插件：
https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client
2. 创建存储类(StorageClass)
# cat storageclass-nfs.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: managed-nfs-storage
provisioner: fuseim.pri/ifs # or choose another name, must match deployment's env PROVISIONER_NAME'
parameters:
archiveOnDelete: "true"
# kubectl create -f storageclass-nfs.yaml
# kubectl get storageclass
NAME PROVISIONER AGE
managed-nfs-storage fuseim.pri/ifs 120m
3. PV动态供给apiserver授权
NFS动态供给插件动态创建PV，要连接apiserver, 所以需要ServiceAccount授权
# cat rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: nfs-client-provisioner
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: nfs-client-provisioner-runner
rules:
- apiGroups: [""]
resources: ["persistentvolumes"]
verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: [""]
resources: ["persistentvolumeclaims"]
verbs: ["get", "list", "watch", "update"]
- apiGroups: ["storage.k8s.io"]
resources: ["storageclasses"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["events"]
verbs: ["list", "watch", "create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: run-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
namespace: default
roleRef:
kind: ClusterRole
name: nfs-client-provisioner-runner
apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
rules:
- apiGroups: [""]
resources: ["endpoints"]
verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: leader-locking-nfs-client-provisioner
subjects:
- kind: ServiceAccount
name: nfs-client-provisioner
# replace with namespace where provisioner is deployed
namespace: default
roleRef:
kind: Role
name: leader-locking-nfs-client-provisioner
apiGroup: rbac.authorization.k8s.io
# kubectl create -f rbac.yaml
4. 创建NFS动态供给插件(Pod)
动态供给插件（Pod）动态创建PV，需要访问apiverser, 所以要指定刚创建的serviceAccount: nfs-client-provisioner
# cat deployment-nfs.yaml
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: nfs-client-provisioner
spec:
replicas: 1
strategy:
type: Recreate
template:
metadata:
labels:
app: nfs-client-provisioner
spec:
serviceAccount: nfs-client-provisioner
containers:
- name: nfs-client-provisioner
image: quay.io/external_storage/nfs-client-provisioner:latest
volumeMounts:
- name: nfs-client-root
mountPath: /persistentvolumes
env:
- name: PROVISIONER_NAME
value: fuseim.pri/ifs
- name: NFS_SERVER
value: 10.40.6.214
- name: NFS_PATH
value: /data/nfs
volumes:
- name: nfs-client-root
nfs:
server: 10.40.6.214
path: /data/nfs
# kubectl create -f deployment-nfs.yaml
# kubectl get deploy,pod
NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
deployment.extensions/nfs-client-provisioner 1 1 1 1 117m
NAME READY STATUS RESTARTS AGE
pod/nfs-client-provisioner-6c6f8f9bd6-nclwf 1/1 Running 0 117m
5. PV动态供给应用案例
创建一个statefulset 和 headless service 的mysql服务，在yaml文件定义使用 managed-nfs-storage StorageClass创建PVC。
# cat mysql-statefulset.yaml
apiVersion: v1
kind: Service
metadata:
name: mysql
labels:
app: mysql
spec:
ports:
- port: 3306
name: mysql
clusterIP: None
selector:
app: mysql-public
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
name: db
spec:
serviceName: mysql
template:
metadata:
labels:
app: mysql-public
spec:
containers:
- name: mysql
image: mysql:5.7
env:
- name: MYSQL_ROOT_PASSWORD
value: "123456"
- name: MYSQL_DATABASE
value: test
ports:
- containerPort: 3306
volumeMounts:
- mountPath: "/var/lib/mysql"
name: mysql-data
volumes:
- name: mysql-data
persistentVolumeClaim:
claimName: mysql-pvc
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pvc
namespace: default
spec:
accessModes:
- ReadWriteMany
storageClassName: managed-nfs-storage
resources:
requests:
storage: 8Gi
# kubectl create -f mysql-statefulset.yaml
# kubectl get svc,pod,pv,pvc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/mysql ClusterIP None <none> 3306/TCP 58s
NAME READY STATUS RESTARTS AGE
pod/db-0 1/1 Running 0 58s
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-a81a2372-9652-11e9-95a4-005056b66bc1 8Gi RWX Delete Bound default/mysql-pvc managed-nfs-storage 44s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/mysql-pvc Bound pvc-a81a2372-9652-11e9-95a4-005056b66bc1 8Gi RWX managed-nfs-storage 58s
# kubectl run -it --image mysql:5.7 mysql-client --restart Never --rm /bin/bash
root@mysql-client:/# mysql -h db-0.mysql -uroot -p123456
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
| test |
+--------------------+
5 rows in set (0.01 sec)
mysql>
五. StatefulSet 存储状态管理机制
创建一个nginx headless service 和StatefulSet pod，并创建挂载pv与pvc
https://kubernetes.io/docs/tutorials/stateful-application/basic-stateful-set/
# cat nginx-statefulset-pv.yaml
apiVersion: v1
kind: Service
metadata:
name: nginx
labels:
app: nginx
spec:
ports:
- port: 80
name: web
clusterIP: None
selector:
app: nginx
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: web
spec:
selector:
matchLabels:
app: nginx
serviceName: "nginx"
replicas: 2
template:
metadata:
labels:
app: nginx
spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: nginx
ports:
- containerPort: 80
name: web
volumeMounts:
- name: www
mountPath: /usr/share/nginx/html
volumeClaimTemplates:
- metadata:
name: www
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "managed-nfs-storage"
resources:
requests:
storage: 1Gi
# kubectl create -f nginx-statefulset-pv.yaml
# kubectl get pod
NAME READY STATUS RESTARTS AGE
web-0 1/1 Running 0 21s
web-1 1/1 Running 0 19s
查看有状态应用对存储做了哪些维护，及查看如何动态创建PV ：
# kubectl get pv,pvc
NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE
persistentvolume/pvc-96003cc3-9655-11e9-95a4-005056b66bc1 1Gi RWO Delete Bound default/www-web-0 managed-nfs-storage 2m18s
persistentvolume/pvc-977007f8-9655-11e9-95a4-005056b66bc1 1Gi RWO Delete Bound default/www-web-1 managed-nfs-storage 2m16s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
persistentvolumeclaim/www-web-0 Bound pvc-96003cc3-9655-11e9-95a4-005056b66bc1 1Gi RWO managed-nfs-storage 2m18s
persistentvolumeclaim/www-web-1 Bound pvc-977007f8-9655-11e9-95a4-005056b66bc1 1Gi RWO managed-nfs-storage 2m16s
到nfs 服务器查看 /data/nfs/目录，会自动创建pv相应的目录
# ll /data/nfs/
total 4
drwxrwxrwx 2 root root 6 Jun 24 15:56 default-www-web-0-pvc-96003cc3-9655-11e9-95a4-005056b66bc1
drwxrwxrwx 2 root root 6 Jun 24 15:56 default-www-web-1-pvc-977007f8-9655-11e9-95a4-005056b66bc1
这里的PVC有唯一标识: 0、1，www-web-（0|1），为每个pod提供不同的存储，当某个pod 宕机重新拉起会自动去找对应PVC，比如web-0 pod宕机，重新拉起时会去找到www-web-0，保持pod对应的PVC 是唯一的
