存储的高可用

官网地址
项目源码地址
上一层文章-开发环境k8s 高可用master 的搭建

申明：

这里用的loadbalance ，还是上层文章开发环境的nginx .

stream {
    server {
        listen 8443;
        proxy_pass kube_apiserver;
    }

    upstream kube_apiserver {
        server 192.168.10.133:6443 weight=50 max_fails=3 fail_timeout=5s;
        server 192.168.10.134:6443 weight=50 max_fails=3 fail_timeout=5s;
        
    }
}

说明

Etcd是一个高可用的键值存储系统，主要用于共享配置和服务发现，它通过Raft一致性算法处理日志复制以保证强一致性，我们可以理解它为一个高可用强一致性的服务发现存储仓库。

在kubernetes集群中，etcd主要用于配置共享和服务发现

Etcd主要解决的是分布式系统中数据一致性的问题，而分布式系统中的数据分为控制数据和应用数据，etcd处理的数据类型为控制数据，对于很少量的应用数据也可以进行处理。

Etcd 好处：

• 简单。
  使用Go语言编写部署简单；使用HTTP作为接口使用简单；使用Raft算法保证强一致性
  让用户易于理解。
• 数据持久化。
  etcd默认数据一更新就进行持久化。
• 安全。
  etcd支持SSL客户端安全认证。

示例部署ETCD高可用集群

上一节的问题，etcd 没有集群化

• 后果就是etcd 意外死亡，那么k8s 将不可用
• etcd 还需要做的就是，数据的备份

开始搭建etcd 集群

我们完整的结构应该是这样的：

image.png

• 我们部署的etcd 是容器化的，并没有独立成一个集群
• 参考官网 可知，在上一步的kubeadm-config.yaml 中没有配置 etcd 的地址，通过describe 命令查看etcd 也可以知道配置有问题。
  上面基于容器的做法很容易出问题，这次按照生产的标准搭建高可用k8s 集群。

开始

安装cdssl 证书生成工具

wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64

wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64

chmod +x cfssl_linux-amd64 cfssljson_linux-amd64 cfssl-certinfo_linux-amd64

mv cfssl_linux-amd64 /usr/local/bin/cfssl

mv cfssljson_linux-amd64 /usr/local/bin/cfssljson

mv cfssl-certinfo_linux-amd64 /usr/bin/cfssl-certinfo

创建 CA 配置文件（ca-config.json）

{
    "signing": {
        "default": {
            "expiry": "175200h"
        },
        "profiles": {
            "kubernetes": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
            "client auth"
                ]
            },
            "etcd": {
                "expiry": "175200h",
                "usages": [
                    "signing",
                    "key encipherment",
                    "server auth",
                    "client auth"
                ]
            }
        }
    }
}

创建 CA 证书签名请求（ca-csr.json）

{
  "CN": "kubernetes",
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shenzhen",
      "L": "shenzhen",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

生成 CA 证书和私钥
cfssl gencert -initca ca-csr.json | cfssljson -bare ca

[root@k8s-node1 bin]# cfssl gencert -initca ca-csr.json | cfssljson -bare ca
2019/10/29 21:10:59 [INFO] generating a new CA key and certificate from CSR
2019/10/29 21:10:59 [INFO] generate received request
2019/10/29 21:10:59 [INFO] received CSR
2019/10/29 21:10:59 [INFO] generating key: rsa-2048
2019/10/29 21:11:00 [INFO] encoded CSR
2019/10/29 21:11:00 [INFO] signed certificate with serial number 213974363926484894183998850838526320112981116568
[root@k8s-node1 bin]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  cfssl  cfssl-certinfo  cfssljson

创建 kubernetes证书签名请求（k8s-csr.json），这里注意hosts需要配置kubernetes 自身的域名，否则会导致coreDns 不可用（其实就是认证失败，调度不了etcd 接口）

{
  "CN": "kubernetes",
  "hosts": [
    "127.0.0.1",
    "192.168.10.133",
    "192.168.10.134",
    "192.168.10.135",
    "10.96.0.1",
      "kubernetes",
      "kubernetes.default",
      "kubernetes.default.svc",
      "kubernetes.default.svc.cluster",
      "kubernetes.default.svc.cluster.local"

  ],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "ST": "shenzhen",
      "L": "shenzhen",
      "O": "k8s",
      "OU": "System"
    }
  ]
}

执行

cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes k8s-csr.json | cfssljson -bare kubernetes


[root@k8s-node1 bin]# ls
ca-config.json  ca.csr  ca-csr.json  ca-key.pem  ca.pem  cfssl  cfssl-certinfo  cfssljson  k8s-csr.json  kubernetes.csr  kubernetes-key.pem  kubernetes.pem
[root@k8s-node1 bin]# 

复制pem后缀的证书到/etc/kubernetes/ssl下,每个节点都做
这里说明下，kubeadm reset 命令会删除 /etc/kubernetes/pki 下所有的文件。所有有些证书，比如etcd 的，我们存在另一个地方，一个是本地的master 主机上跑了etcd 服务需要，还一个就是存个备份。

在后面，可以看到我把这里的etcd 的各种证书，每次 reset 完之后，都会在 /etc/kubernetes/pki 下创建 etcd目录，然后 cp -r /etc/kubernetes/ssl/*.pem /etc/kubernetes/pki /etcd 。

可以认为。k8s 也是etcd 的一个客户端，etcd 服务做了tls ,客户端当然需要证书才能访问。

# mkdir -p /etc/kubernetes/ssl
# cp *.pem /etc/kubernetes/ssl

去https://github.com/etcd-io/etcd/releases
下载安装包，我用到最新的，解压在了 /etc/etcd 目录下即可。

配置 etcd.serice ，使用systemctl 管理 。配置127.0.0.1 不需要https.

[root@k8s-node1 bin]# cat /etc/systemd/system/etcd.service 
[Unit]
Description=Etcd Server
After=network.target
After=network-online.target
Wants=network-online.target
Documentation=https://github.com/coreos

[Service]
Type=notify
WorkingDirectory=/var/lib/etcd/
EnvironmentFile=-/etc/etcd/etcd.yaml
ExecStart=/bin/bash -c "GOMAXPROCS=$(nproc) /etc/etcd/etcd --name=etcd-0 --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls=https://192.168.10.134:2380 --listen-peer-urls=https://192.168.10.134:2380 --listen-client-urls=https://192.168.10.134:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.10.134:2379 --initial-cluster-token=etcd-cluster-0 --initial-cluster=\"etcd-0=https://192.168.10.133:2380,etcd-1=https://192.168.10.134:2380\" --initial-cluster-state=new --data-dir=/var/lib/etcd"
Restart=on-failure
RestartSec=5
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

另一个节点同样做 证书复制，把etcd 放在了 /etc/etcd 下面，配置修改操作。

启动etcd

[root@k8s-node1 bin]# systemctl daemon-reload
[root@k8s-node1 bin]# systemctl start etcd
[root@k8s-node1 bin]# ps -ef |grep etcd 
root      44905      1  2 11:37 ?        00:00:00 /etc/etcd/etcd --name=etcd-1 --cert-file=/etc/kubernetes/ssl/kubernetes.pem --key-file=/etc/kubernetes/ssl/kubernetes-key.pem --peer-cert-file=/etc/kubernetes/ssl/kubernetes.pem --peer-key-file=/etc/kubernetes/ssl/kubernetes-key.pem --trusted-ca-file=/etc/kubernetes/ssl/ca.pem --peer-trusted-ca-file=/etc/kubernetes/ssl/ca.pem --initial-advertise-peer-urls=https://192.168.10.134:2380 --listen-peer-urls=https://192.168.10.134:2380 --listen-client-urls=https://192.168.10.134:2379,http://127.0.0.1:2379 --advertise-client-urls=https://192.168.10.134:2379 --initial-cluster-token=etcd-cluster-0 --initial-cluster=etcd-0=https://192.168.10.133:2380,etcd-1=https://192.168.10.134:2380 --initial-cluster-state=new --data-dir=/var/lib/etcd
root      44918  16456  0 11:38 pts/1    00:00:00 grep --color=auto etcd

查看集群结果

[root@k8s-master k8s]# cd /etc/etcd
[root@k8s-master etcd]# ls
etcd  etcd-0.etcd  etcdctl  etcd.yml  install  nohup.out
[root@k8s-master etcd]# 
[root@k8s-master etcd]# ./etcdctl --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/kubernetes.pem --key=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.10.133:2379,https://192.168.10.134:2379  endpoint health 
https://192.168.10.134:2379 is healthy: successfully committed proposal: took = 15.768454ms
https://192.168.10.133:2379 is healthy: successfully committed proposal: took = 17.041281ms

由此可知，etcd 集群完成。

开始配置k8s

初始化第一个节点：

• 重置一个节点

[root@k8s-master k8s]# kubeadm   reset 
[reset] WARNING: changes made to this host by 'kubeadm init' or 'kubeadm join' will be reverted.
[reset] are you sure you want to proceed? [y/N]: y
[preflight] running pre-flight checks
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
W1030 13:25:39.401263   66304 reset.go:213] [reset] Unable to fetch the kubeadm-config ConfigMap, using etcd pod spec as fallback: failed to get config map: Get https://192.168.10.133:8443/api/v1/namespaces/kube-system/configmaps/kubeadm-config: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")
[reset] no etcd config found. Assuming external etcd
[reset] please manually reset etcd to prevent further issues
[reset] stopping the kubelet service
[reset] unmounting mounted directories in "/var/lib/kubelet"
[reset] deleting contents of stateful directories: [/var/lib/kubelet /etc/cni/net.d /var/lib/dockershim /var/run/kubernetes]
[reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
[reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]

The reset process does not reset or clean up iptables rules or IPVS tables.
If you wish to reset iptables, you must do so manually.
For example: 
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X

If your cluster was setup to utilize IPVS, run ipvsadm --clear (or similar)
to reset your system's IPVS tables.

[root@k8s-master k8s]# mkdir -p  /etc/kubernetes/pki/etcd/
[root@k8s-master k8s]# cp  -r /etc/kubernetes/ssl/*.pem  /etc/kubernetes/pki/etcd/
[root@k8s-master k8s]# 

初始化：

重点

先修改kubelet 自带的etcd 配置，这里可恶的竟然是cgroup-driver 设置的是systemd .
删除自带的etcd 配置文件（把所有的master 节点上的这个文件都删除了）。 这里坑了好久。。。。

[root@k8s-master kubelet.service.d]# 
[root@k8s-master kubelet.service.d]# cat 20-etcd-service-manager.conf 
[Service]
ExecStart=
#  Replace "systemd" with the cgroup driver of your container runtime. The default value in the kubelet is "cgroupfs".
ExecStart=/usr/bin/kubelet --address=127.0.0.1 --pod-manifest-path=/etc/kubernetes/manifests --cgroup-driver=systemd
Restart=always
[root@k8s-master kubelet.service.d]# 
[root@k8s-master kubelet.service.d]# 
[root@k8s-master kubelet.service.d]# rm -rf 20-etcd-service-manager.conf 

复制etcd 证书到etc/kubernetes/pki/etcd/ ，这个目录kubeadm reset 之后会被情况

[root@k8s-master k8s]# mkdir -p  /etc/kubernetes/pki/etcd/
[root@k8s-master k8s]# 
[root@k8s-master k8s]# cp  -r /etc/kubernetes/ssl/*.pem  /etc/kubernetes/pki/etcd/

上面这步骤要在所有master 上执行！上面这步骤要在所有master 上执行！上面这步骤要在所有master 上执行！

编写kubeadm 配置文件,certSANs写全自己的 master地址：

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
kubernetesVersion: stable
apiServer:
  certSANs:
  - "192.168.10.133"
  - "127.0.0.1"
  - "192.168.10.134"
controlPlaneEndpoint: "192.168.10.133:8443"
networking:
  podSubnet: 10.244.0.0/16
etcd:
    external:
        endpoints:
        - https://192.168.10.133:2379
        - https://192.168.10.134:2379
        caFile: /etc/kubernetes/pki/etcd/ca.pem
        certFile: /etc/kubernetes/pki/etcd/kubernetes.pem
        keyFile: /etc/kubernetes/pki/etcd/kubernetes-key.pem

因为前面的etcd 的cfssl 证书都拿来放在了/etc/kubernets/pki/etcd 下面了，直接进行init 操作

[root@k8s-master k8s]# mkdir -p  /etc/kubernetes/pki/etcd/
[root@k8s-master k8s]# cp  -r /etc/kubernetes/ssl/*.pem  /etc/kubernetes/pki/etcd/
[root@k8s-master k8s]# kubeadm init --config=kubeadm-config.yaml   -upload-certs --ignore-preflight-errors=all  
I1030 14:02:04.630701   70169 version.go:94] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable.txt": Get https://dl.k8s.io/release/stable.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
I1030 14:02:04.630785   70169 version.go:95] falling back to the local client version: v1.13.0
[init] Using Kubernetes version: v1.13.0
[preflight] Running pre-flight checks
    [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate authority generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate authority generation
[certs] External etcd mode: Skipping etcd/peer certificate authority generation
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 192.168.10.133 192.168.10.133 192.168.10.133]
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[apiclient] All control plane components are healthy after 19.504737 seconds
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.13" in namespace kube-system with the configuration for the kubelets in the cluster
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-master" as an annotation
[mark-control-plane] Marking the node k8s-master as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: s1ry1w.uedyn8db3jnm1tsg
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstraptoken] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstraptoken] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstraptoken] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstraptoken] creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[addons] Applied essential addon: kube-proxy

Your Kubernetes master has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of machines by running the following on each node
as root:

  kubeadm join 192.168.10.133:8443 --token s1ry1w.uedyn8db3jnm1tsg --discovery-token-ca-cert-hash sha256:fdcc767dc0aeebfadaa4faef2ee19642ce6d85462771baaba23f927696c4ef78

安装flannel 组件服务，同上一步，安装完成后，节点变为 ready 状态
见：kube-flannel.yml

[root@k8s-master k8s]# kubectl  apply -f flannel.yaml  
clusterrole.rbac.authorization.k8s.io/flannel created
clusterrolebinding.rbac.authorization.k8s.io/flannel created
serviceaccount/flannel created
configmap/kube-flannel-cfg created
daemonset.extensions/kube-flannel-ds-amd64 created
[root@k8s-master k8s]# kubectl  get no 

NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   5m12s   v1.13.0

OK ，第一个节点终于完成了。这时候，这个节点的证书就可以复制到其他的master 主机上了。
高版本的使用upload-certs 参数的话，证书也不需要每次复制了。

初始化第二个master节点：
我用的是1.13 版本的k8s , join 命令带 "--experimental-control-plane " 这个参数，代表加入的是个master 节点，不带就是node 。 注意这里也需要删除kubelete 自带的etcd 文件。

重点：
首先，就是把第一个master 上的证书拿过来，既把maseter1 上的 /etc/kubernetes/pki 文件夹全部复制到master2 上面的 /etc/kubernetes/pki 。 这个文件放的是各种证书。

开始加入主节点

[root@k8s-node1 k8s]# kubeadm join 192.168.10.133:8443 --token f2mitj.hbrghf2lpmy4n91f --discovery-token-ca-cert-hash sha256:2cbd8aa7a7822beec6fff001f8b03f54180ef3924314964e97174562d71a2873    --experimental-control-plane  
[preflight] Running pre-flight checks
    [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[discovery] Trying to connect to API Server "192.168.10.133:8443"
[discovery] Created cluster-info discovery client, requesting info from "https://192.168.10.133:8443"
[discovery] Requesting info from "https://192.168.10.133:8443" again to validate TLS against the pinned public key
[discovery] Cluster info signature and contents are valid and TLS certificate validates against pinned roots, will use API Server "192.168.10.133:8443"
[discovery] Successfully established connection with API Server "192.168.10.133:8443"
[join] Reading configuration from the cluster...
[join] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml'
[join] Running pre-flight checks before initializing the new control plane instance
    [WARNING SystemVerification]: this Docker version is not on the list of validated versions: 18.09.0. Latest validated version: 18.06
[certs] Using the existing "front-proxy-client" certificate and key
[certs] Using the existing "apiserver" certificate and key
[certs] Using the existing "apiserver-kubelet-client" certificate and key
[certs] valid certificates and keys now exist in "/etc/kubernetes/pki"
[certs] Using the existing "sa" key
[endpoint] WARNING: port specified in controlPlaneEndpoint overrides bindPort in the controlplane address
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[kubelet] Downloading configuration for the kubelet from the "kubelet-config-1.13" ConfigMap in the kube-system namespace
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Activating the kubelet service
[tlsbootstrap] Waiting for the kubelet to perform the TLS Bootstrap...
[patchnode] Uploading the CRI Socket information "/var/run/dockershim.sock" to the Node API object "k8s-node1" as an annotation
[uploadconfig] storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[mark-control-plane] Marking the node k8s-node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-node1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

This node has joined the cluster and a new control plane instance was created:

* Certificate signing request was sent to apiserver and approval was received.
* The Kubelet was informed of the new secure connection details.
* Master label and taint were applied to the new node.
* The Kubernetes control plane instances scaled up.


To start administering your cluster from this node, you need to run the following as a regular user:

    mkdir -p $HOME/.kube
    sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    sudo chown $(id -u):$(id -g) $HOME/.kube/config

Run 'kubectl get nodes' to see this node join the cluster.

[root@k8s-node1 k8s]# mkdir -p $HOME/.kube
[root@k8s-node1 k8s]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
cp：是否覆盖"/root/.kube/config"？ y
[root@k8s-node1 k8s]# sudo chown $(id -u):$(id -g) $HOME/.kube/config
[root@k8s-node1 k8s]# 
[root@k8s-node1 k8s]# 
[root@k8s-node1 k8s]# kubectl get no 
NAME         STATUS   ROLES    AGE   VERSION
k8s-master   Ready    master   28m   v1.13.0
k8s-node1    Ready    master   17m   v1.13.0

此时查看集群的组件状态：

[root@k8s-node1 pki]# kubectl  get cs 

NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok                  
scheduler            Healthy   ok                  
etcd-0               Healthy   {"health":"true"}   
etcd-1               Healthy   {"health":"true"} 

[root@k8s-node1 pki]# kubectl  get  po --all-namespaces
NAMESPACE     NAME                                 READY   STATUS    RESTARTS   AGE
kube-system   coredns-86c58d9df4-4px8n             1/1     Running   0          11m
kube-system   coredns-86c58d9df4-qwbql             1/1     Running   0          11m
kube-system   kube-apiserver-k8s-master            1/1     Running   0          10m
kube-system   kube-apiserver-k8s-node1             1/1     Running   0          6m55s
kube-system   kube-controller-manager-k8s-master   1/1     Running   0          10m
kube-system   kube-controller-manager-k8s-node1    1/1     Running   0          6m55s
kube-system   kube-flannel-ds-amd64-nntpq          1/1     Running   0          10m
kube-system   kube-flannel-ds-amd64-tj4sz          1/1     Running   0          6m55s
kube-system   kube-proxy-fm22x                     1/1     Running   0          11m
kube-system   kube-proxy-ngknh                     1/1     Running   0          6m55s
kube-system   kube-scheduler-k8s-master            1/1     Running   0          10m
kube-system   kube-scheduler-k8s-node1             1/1     Running   0          6m55s

[root@k8s-node1 pki]# kubectl  get  no
NAME         STATUS   ROLES    AGE     VERSION
k8s-master   Ready    master   11m     v1.13.0
k8s-node1    Ready    master   7m12s   v1.13.0

至此，k8s 双master 节点搭建完成，且etcd 集群外置。 当集群死亡之后，再次重启可以从etcd 恢复集群状态。

注意的是，如果想重新搭建k8s集群，需要清空etcd里存储的pod 数据。

[root@k8s-node1 etcd]# ./etcdctl --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/kubernetes/ssl/kubernetes.pem --key=/etc/kubernetes/ssl/kubernetes-key.pem --endpoints=https://192.168.10.133:2379,https://192.168.10.134:2379  del / --prefix 
447
[root@k8s-node1 etcd]# 

高版本的 master 扩展

在init 第一个master 时候的不同；

kubeadm init --config=kubeadm-config.yaml --upload-certs --ignore-preflight-errors=all

高版本其实会打出两条join 信息，一个是加入 master 节点的信息，一个是加入node 节点的信息，和低版本的join命令类似，不多在加入master 的时候又多了一个 认证的参数，如下：

kubeadm join master:6443 --token uwm1iq.bydvqhg91670w58g \
  --discovery-token-ca-cert-hash sha256:fef3eba9bc05450a0e3c705239775bf5297a82ecf4faf683a7cbb3a38b433ca0 \
  --experimental-control-plane --certificate-key cdad981734e547e60133a044ac72a60f6021038c5c3ca4d25a886e0ce79f9b57 \
 --ignore-preflight-errors=all
 # --ignore-preflight-errors=all表示忽略错误，可以不加

附 在机房花了20分钟安装k8s v1.16.2 版本安装完之后的效果：

[root@k8s-master1 master]# kubectl  get no  -o wide 
NAME          STATUS   ROLES    AGE         VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE                KERNEL-VERSION               CONTAINER-RUNTIME
k8s-master1   Ready    master   20m         v1.16.2   192.168.1.110   <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.1.el7.x86_64   docker://18.6.1
k8s-master2   Ready    master   15m         v1.16.2   192.168.1.111   <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.1.el7.x86_64   docker://18.6.1
k8s-node1     Ready    <none>   6m24s       v1.16.2   192.168.1.120   <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.1.el7.x86_64   docker://18.6.1
k8s-node2     Ready    <none>   2m55s       v1.16.2   192.168.1.121   <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.1.el7.x86_64   docker://18.6.1
k8s-node3     Ready    <none>   <invalid>   v1.16.2   192.168.1.122   <none>        CentOS Linux 7 (Core)   3.10.0-1062.4.1.el7.x86_64   docker://18.6.1

• 安装文件kubeadm-conf.yaml：

apiVersion: kubeadm.k8s.io/v1beta1
kind: ClusterConfiguration
clusterName: kubernetes
apiServer:
   certSANs:
   - "192.168.1.110"
   - "192.168.1.111"
   - "127.0.0.1"
   - "192.168.1.109"
   - "k8s-master1"
   - "k8s-master2"
    
controlPlaneEndpoint: "192.168.1.109:6443"
imageRepository: registry.aliyuncs.com/google_containers
kubernetesVersion: v1.16.0
networking:
  dnsDomain: cluster.local
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.245.0.0/16"
scheduler: {}
controllerManager: {}
etcd:
    external:
        endpoints:
        - https://192.168.1.110:2379
        - https://192.168.1.111:2379
        caFile: /etc/etcd/ssl/ca.pem
        certFile: /etc/etcd/ssl/kubernetes.pem
        keyFile: /etc/etcd/ssl/kubernetes-key.pem

• master1 初始化成功，看着命令join即可：

[root@k8s-master1 master]# kubeadm init --config=test.yaml  --upload-certs
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR ExternalEtcdVersion]: couldn't load external etcd's certificate and key pair /etc/etcd/ssl/kubernetes.pem, /etc/etcd/ssl/kubernetes-key.pe: open /etc/etcd/ssl/kubernetes-key.pe: no such file or directory
    [ERROR ExternalEtcdClientCertificates]: /etc/etcd/ssl/kubernetes-key.pe doesn't exist
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
To see the stack trace of this error execute with --v=5 or higher
[root@k8s-master1 master]# vim test.yaml 
[root@k8s-master1 master]# 
[root@k8s-master1 master]# kubeadm init --config=test.yaml  --upload-certs
[init] Using Kubernetes version: v1.16.0
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Activating the kubelet service
[certs] Using certificateDir folder "/etc/kubernetes/pki"
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [k8s-master1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local k8s-master1 k8s-master2] and IPs [10.245.0.1 192.168.1.110 192.168.1.109 192.168.1.110 192.168.1.111 127.0.0.1 192.168.1.109]
[certs] Generating "apiserver-kubelet-client" certificate and key
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
[certs] External etcd mode: Skipping etcd/ca certificate authority generation
[certs] External etcd mode: Skipping etcd/server certificate generation
[certs] External etcd mode: Skipping etcd/peer certificate generation
[certs] External etcd mode: Skipping etcd/healthcheck-client certificate generation
[certs] External etcd mode: Skipping apiserver-etcd-client certificate generation
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
[kubeconfig] Writing "admin.conf" kubeconfig file
[kubeconfig] Writing "kubelet.conf" kubeconfig file
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
[kubeconfig] Writing "scheduler.conf" kubeconfig file
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
[kubelet-check] Initial timeout of 40s passed.
[apiclient] All control plane components are healthy after 63.035131 seconds
[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace
[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
96dacef0c81bf971bcb39f347c3d3fd6e2fb2c3d253c6da01ec29f1e39eb2713
[mark-control-plane] Marking the node k8s-master1 as control-plane by adding the label "node-role.kubernetes.io/master=''"
[mark-control-plane] Marking the node k8s-master1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]
[bootstrap-token] Using token: 2sqa85.x8jxuibdr19f9wg0
[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles
[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials
[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token
[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster
[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace
[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of the control-plane node running the following command on each as root:

  kubeadm join 192.168.1.109:6443 --token 2sqa85.x8jxuibdr19f9wg0 \
    --discovery-token-ca-cert-hash sha256:f07f424c74f4871639ac3d4131db7d51c550d0965df1856e9bf51d711f79c43b \
    --control-plane --certificate-key 96dacef0c81bf971bcb39f347c3d3fd6e2fb2c3d253c6da01ec29f1e39eb2713

Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use 
"kubeadm init phase upload-certs --upload-certs" to reload certs afterward.

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 192.168.1.109:6443 --token 2sqa85.x8jxuibdr19f9wg0 \
    --discovery-token-ca-cert-hash sha256:f07f424c74f4871639ac3d4131db7d51c550d0965df1856e9bf51d711f79c43b 

默认生成的证书是临时加入使用的，失效可以用：

kubeadm init phase upload-certs --experimental-upload-certs

node 的加入就简单了，和原先的一样，不带这些主节点的参数信息如：

kubeadm join master:6443 --token 74fxrx.34uwnbxun31d9cgc \
    --discovery-token-ca-cert-hash sha256:7056fa528a5f0f962f565ee4eb7a96d7ecc5432883bd3c506ca0fa1eab09064c

例如，忘记了join 命令,如下重新生成即可：

[root@k8s-master1 ~]# kubeadm token  create --print-join-command
kubeadm join 192.168.1.109:6443 --token 779f9q.1jm3gryowdvpreff     --discovery-token-ca-cert-hash sha256:d1e5bb20d5b73dd317e5f9e7e699f5f02932b691608c138af032f45f2838e21e 
[root@k8s-master1 ~]# 
[root@k8s-master1 master]# kubeadm init phase upload-certs --upload-certs 
W1108 12:51:43.116233    8127 version.go:98] could not fetch a Kubernetes version from the internet: unable to get URL "https://dl.k8s.io/release/stable-1.txt": Get https://dl.k8s.io/release/stable-1.txt: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
W1108 12:51:43.116287    8127 version.go:99] falling back to the local client version: v1.15.0
[upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
[upload-certs] Using certificate key:
3461c4d9b5332796907d1412d9cb82a36869c1f3dd171d2d95b496643366183a

有些东西参考上一层。开发环境k8s 高可用master 的搭建

下一层 ： 锻骨境-第5层 k8s-ETCD集群备份与恢复