三主模式高可用k8s集群搭建

操作系统：CentOS Linux release 7.7.1908 (Core)

docker版本：18.09.1

kubernetes版本：v1.16.3

至少2G内存、2CPU

网络良好

特定端口是开放的(前期可以把防火墙给禁用掉)


一、配置网络

１．主机网络采用静态网址，例：

vim  /etc/sysconfig/network-scripts/ifcfg-ens33

TYPE=Ethernet

PROXY_METHOD=none

BROWSER_ONLY=no

BOOTPROTO=static    #设置为静态地址

DEFROUTE=yes

IPV4_FAILURE_FATAL=no

IPV6INIT=yes

IPV6_AUTOCONF=yes

IPV6_DEFROUTE=yes

IPV6_FAILURE_FATAL=no

IPV6_ADDR_GEN_MODE=stable-privacy

NAME=ens33

UUID=e4f81192-b2e7-407b-90ea-a36bd0cc230a

DEVICE=ens33

ONBOOT=YES

IPADDR="192.168.１.100" #本机ip地址

PREFIX="24"

GATEWAY="192.168.１.2"  #默认网关

DNS1="8.8.8.8"          #DNS

２．启动网络systemctr restart network，其他主机采用类似方法安装。

二、主机名设置

１．分别在主机上设置主机名,例如：在第一台主机上执行下列命令

hostnamectl set-hostname airport-k8s-m1

２．配置hosts

api.k8s.airport.com是我们设置的k8s集群api-server域名，也可以采用DNS域名解析实现，这里键就通过hsots模拟DNS域名解析。

    cat >>/etc/host <<EOF

    192.168.１.100 airport-k8s-m1

    192.168.１.101 airport-k8s-m2

    192.168.１.102 airport-k8s-m3

    192.168.１.100 api.k8s.airport.com

    192.168.１.101 api.k8s.airport.com

    192.168.１.102 api.k8s.airport.com

    EOF

三、生成免密登录密钥(可选)

ssh-keygen -f /root/.ssh/id_rsa -N ''

for i in airport-k8s-m1 airport-k8s-m2 airport-k8s-m3

do

ssh-copy-id $i

done

四、关闭selinux

sed -i 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config && setenforce 0

五、时间同步

安装 chrony 服务

yum install -y chrony

systemctl start chronyd

systemctl enable chronyd

六、关闭防火墙

在生产环境中，应该是配置防火墙，开放必要的端口

systemctl stop firewalld

systemctl disable firewalld

七、关闭swap分区

sed -i '11s/\/dev/# \/dev/g' /etc/fstab

swapoff -a


八、配置IPVS内核

１．默认情况下，Kube-proxy将在kubeadm部署的集群中以iptables模式运行

centos8中已经放弃iptables

yum install -y ipset ipvsadm

cat > /etc/sysconfig/modules/ipvs.modules <<EOF

#!/bin/bash

modprobe -- ip_vs

modprobe -- ip_vs_rr

modprobe -- ip_vs_wrr

modprobe -- ip_vs_sh

modprobe -- nf_conntrack

EOF

chmod +x /etc/sysconfig/modules/ipvs.modules

bash /etc/sysconfig/modules/ipvs.modules

２．配置内核参数

cat > /etc/sysctl.d/k8s.conf <<EOF

net.bridge.bridge-nf-call-ip6tables = 1

net.bridge.bridge-nf-call-iptables = 1

#net.ipv4.ip_nonlocal_bind = 1

#net.ipv4.ip_forward = 1

#vm.swappiness=0

EOF

modprobe br_netfilter

sysctl -p /etc/sysctl.d/k8s.conf

３．打开文件数

echo "* soft nofile 65536" >> /etc/security/limits.conf

echo "* hard nofile 65536" >> /etc/security/limits.conf


九、安装docker(参考官网)

sudo yum remove docker docker-client  docker-client-latest  docker-common docker-latest docker-latest-logrotate docker-logrotate docker-engine

rm -rf /var/lib/docker

sudo yum install -y yum-utils  device-mapper-persistent-data  lvm2

sudo yum-config-manager    --add-repo    https://download.docker.com/linux/centos/docker-ce.repo

sudo yum-config-manager --enable docker-ce-nightly

sudo yum-config-manager --enable docker-ce-test

sudo yum-config-manager --disable docker-ce-nightly

sudo yum install docker-ce docker-ce-cli containerd.io

如需指定版本：

sudo yum -y install docker-ce-18.09.1 docker-ce-cli-18.09.1 containerd.io

yum list docker-ce --showduplicates | sort -r

sudo systemctl start docker

sudo docker run hello-world

docker version

需要配合 kubernetes的地方有:

１．设置开机自动重启

systemctl enable docker && systemctl start docker

２．设置 cgroup驱动使用systemd .(前面有 /etc/docker目录后才会执行后面的.) 以及存储

su root

cat > /etc/docker/daemon.json <<EOF

{

  "exec-opts": ["native.cgroupdriver=systemd"],

  "log-driver": "json-file",

  "log-opts": {

    "max-size": "100m"

  },

  "storage-driver": "overlay2",

  "storage-opts": [

    "overlay2.override_kernel_check=true"

  ]

}

EOF

３．然后重启一下 docker, 因为修改了配置文件.

systemctl restart docker


十、kubeadm安装

1.安装 kubeadm, kubelet and kubectl

还是一样的所有的节点都要安装这个.

添加源

cat <<EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg

EOF

替换成阿里的源如下:

cat <<EOF > /etc/yum.repos.d/kubernetes.repo

[kubernetes]

name=Kubernetes

baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/

enabled=1

gpgcheck=1

repo_gpgcheck=1

gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg

EOF

２．安装

sudo yum install -y kubelet kubeadm kubectl --disableexcludes=kubernetes

sudo systemctl enable --now kubelet

３．kubectl 自动补全.

默认安装的有

sudo yum install bash-completion -y

echo "source <(kubectl completion bash)" >> ~/.bashrc

source ~/.bashrc


十一、安装kebenetes集群

安装airport-k8s-m1

１．准备配置文件

cat >kubeadm-config.yaml<<EOF

apiVersion: kubeadm.k8s.io/v1beta1

kind: ClusterConfiguration

kubernetesVersion: v1.16.3

apiServer:

  certSANs:

  - "api.k8s.airport.com"

controlPlaneEndpoint: "api.k8s.airport.com:6443"

imageRepository: registry.aliyuncs.com/google_containers

networking:

  dnsDomain: cluster.local

  podSubnet: "10.244.0.0/16"

  serviceSubnet: 10.96.0.0/12

EOF

２．下载所需镜像

kubeadm config images pull --config kubeadm-config.yaml

３．初始化

kubeadm init --config kubeadm-config.yaml

输出如下：

[init] Using Kubernetes version: v1.16.3

[preflight] Running pre-flight checks

[preflight] Pulling images required for setting up a Kubernetes cluster

[preflight] This might take a minute or two, depending on the speed of your internet connection

[preflight] You can also perform this action in beforehand using 'kubeadm config images pull'

[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"

[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"

[kubelet-start] Activating the kubelet service

[certs] Using certificateDir folder "/etc/kubernetes/pki"

[certs] Generating "ca" certificate and key

[certs] Generating "apiserver" certificate and key

[certs] apiserver serving cert is signed for DNS names [node1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local api.k8s.gy.com api.k8s.gy.com] and IPs [10.96.0.1 192.168.44.100]

[certs] Generating "apiserver-kubelet-client" certificate and key

[certs] Generating "front-proxy-ca" certificate and key

[certs] Generating "front-proxy-client" certificate and key

[certs] Generating "etcd/ca" certificate and key

[certs] Generating "etcd/server" certificate and key

[certs] etcd/server serving cert is signed for DNS names [node1 localhost] and IPs [192.168.44.100 127.0.0.1 ::1]

[certs] Generating "etcd/peer" certificate and key

[certs] etcd/peer serving cert is signed for DNS names [node1 localhost] and IPs [192.168.44.100 127.0.0.1 ::1]

[certs] Generating "etcd/healthcheck-client" certificate and key

[certs] Generating "apiserver-etcd-client" certificate and key

[certs] Generating "sa" key and public key

[kubeconfig] Using kubeconfig folder "/etc/kubernetes"

[kubeconfig] Writing "admin.conf" kubeconfig file

[kubeconfig] Writing "kubelet.conf" kubeconfig file

[kubeconfig] Writing "controller-manager.conf" kubeconfig file

[kubeconfig] Writing "scheduler.conf" kubeconfig file

[control-plane] Using manifest folder "/etc/kubernetes/manifests"

[control-plane] Creating static Pod manifest for "kube-apiserver"

[control-plane] Creating static Pod manifest for "kube-controller-manager"

[control-plane] Creating static Pod manifest for "kube-scheduler"

[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"

[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s

[apiclient] All control plane components are healthy after 19.007860 seconds

[upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace

[kubelet] Creating a ConfigMap "kubelet-config-1.16" in namespace kube-system with the configuration for the kubelets in the cluster

[upload-certs] Skipping phase. Please see --upload-certs

[mark-control-plane] Marking the node node1 as control-plane by adding the label "node-role.kubernetes.io/master=''"

[mark-control-plane] Marking the node node1 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule]

[bootstrap-token] Using token: ks271v.t4pp7n7k0qd5danp

[bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles

[bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials

[bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token

[bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster

[bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace

[addons] Applied essential addon: CoreDNS

[addons] Applied essential addon: kube-proxy

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube

  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

  sudo chown $(id -u):$(id -g) $HOME/.kube/config

You should now deploy a pod network to the cluster.

Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:

  https://kubernetes.io/docs/concepts/cluster-administration/addons/

You can now join any number of control-plane nodes by copying certificate authorities

and service account keys on each node and then running the following as root:

  kubeadm join api.k8s.airport.com:6443 --token ks271v.t4pp7n7k0qd5danp \

    --discovery-token-ca-cert-hash sha256:6ab48ebd0024ae79aff1a2fcfa63aa2b61e58083b4aa265ef8fc17d54e6dcca6 \

    --control-plane     

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join api.k8s.airport.com:6443 --token ks271v.t4pp7n7k0qd5danp \

    --discovery-token-ca-cert-hash sha256:6ab48ebd0024ae79aff1a2fcfa63aa2b61e58083b4aa265ef8fc17d54e6dcca6

４．在需对集群管理的节点上执行下列命令：

mkdir -p $HOME/.kube

sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

sudo chown $(id -u):$(id -g) $HOME/.kube/config

十二、安装网络插件

１．这里以 flannel 为例子.

注意这个地方下的是这个版本， 而不要去 flannel 的官网找最新的版本.

kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/2140ac876ef134e0ed5af15c65e414cf26827915/Documentation/kube-flannel.yml

２．插件安装完成后，可以检查 CoreDNS pod 是否运行正常.

kubectl get pods --all-namespace

十三、以master模式加入集群

１．当节点需要以master加入集群时，需要主master的证书和key，下面的脚本将本地的证书拷贝到其他master节点。（还有一种方法是使用kubeadm命令拷贝证书）

权限拷贝脚本sync.master.ca.sh。

#!/bin/sh

vhost="airport-k8s-m2 airport-k8s-m3"

usr=root

who=`whoami`

if [[ "$who" != "$usr" ]];then

  echo "请使用 root 用户执行或者 sudo ./sync.master.ca.sh"

  exit 1

fi

echo $who

# 需要从 node1 拷贝的 ca 文件

caFiles=(

/etc/kubernetes/pki/ca.crt

/etc/kubernetes/pki/ca.key

/etc/kubernetes/pki/sa.key

/etc/kubernetes/pki/sa.pub

/etc/kubernetes/pki/front-proxy-ca.crt

/etc/kubernetes/pki/front-proxy-ca.key

/etc/kubernetes/pki/etcd/ca.crt

/etc/kubernetes/pki/etcd/ca.key

/etc/kubernetes/admin.conf

)

pkiDir=/etc/kubernetes/pki/etcd

for h in $vhost

do

  ssh ${usr}@$h "mkdir -p $pkiDir" 

  echo "Dirs for ca scp created, start to scp..."

  # scp 文件到目标机

  scp /etc/kubernetes/pki/ca.crt /etc/kubernetes/pki/ca.key /etc/kubernetes/pki/sa.key /etc/kubernetes/pki/sa.pub /etc/kubernetes/pki/front-proxy-ca.crt /etc/kubernetes/pki/front-proxy-ca.key    ${usr}@$h:/etc/kubernetes/pki/

    scp /etc/kubernetes/pki/etcd/ca.crt /etc/kubernetes/pki/etcd/ca.key  ${usr}@$h:/etc/kubernetes/pki/etcd/

    scp /etc/kubernetes/admin.conf  ${usr}@$h:/etc/kubernetes/

  echo "Ca files transfered for $h ... ok"

done

２．在node2和node3上以下列命令加入集群：（初始化的输出里，sha256值是唯一的，请注意！）

# 下列命令以master加入集群

You can now join any number of control-plane nodes by copying certificate authorities

and service account keys on each node and then running the following as root:

kubeadm join api.k8s.airport.com:6443 --token ks271v.t4pp7n7k0qd5danp \

    --discovery-token-ca-cert-hash sha256:6ab48ebd0024ae79aff1a2fcfa63aa2b61e58083b4aa265ef8fc17d54e6dcca6 \

    --control-plane

# 下列命令可以将节点作为node加入集群   

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join api.k8s.airport.com:6443 --token ks271v.t4pp7n7k0qd5danp \

    --discovery-token-ca-cert-hash sha256:6ab48ebd0024ae79aff1a2fcfa63aa2b61e58083b4aa265ef8fc17d54e6dcca6

３．检查结果

kubectl get nodes

NAME    STATUS  ROLES    AGE    VERSION

node1  Ready    master  127m  v1.16.3

node2  Ready    master  106m  v1.16.3

node3  Ready    master  103m  v1.16.3

kubectl get pods --all-namespace #拉取镜像时间较长，耐心等待

４．将master作为node（去污）

kubectl taint nodes --all node-role.kubernetes.io/master-

最后编辑于 ：
©著作权归作者所有,转载或内容合作请联系作者
  • 人面猴
    序言：七十年代末，一起剥皮案震惊了整个滨河市，随后出现的几起案子，更是在滨河造成了极大的恐慌，老刑警刘岩，带你破解...
    沈念sama阅读 206,214评论 6赞 481
  • 死咒
    序言：滨河连续发生了三起死亡事件，死亡现场离奇诡异，居然都是意外死亡，警方通过查阅死者的电脑和手机，发现死者居然都...
    沈念sama阅读 88,307评论 2赞 382
  • 救了他两次的神仙让他今天三更去死
    文/潘晓璐 我一进店门，熙熙楼的掌柜王于贵愁眉苦脸地迎上来，“玉大人，你说我怎么就摊上这事。” “怎么了？”我有些...
    开封第一讲书人阅读 152,543评论 0赞 341
  • 道士缉凶录：失踪的卖姜人
    文/不坏的土叔 我叫张陵，是天一观的道长。 经常有香客问我，道长，这世上最难降的妖魔是什么？ 我笑而不...
    开封第一讲书人阅读 55,221评论 1赞 279
  • 港岛之恋（遗憾婚礼）
    正文 为了忘掉前任，我火速办了婚礼，结果婚礼上，老公的妹妹穿的比我还像新娘。我一直安慰自己，他们只是感情好，可当我...
    茶点故事阅读 64,224评论 5赞 371
  • 恶毒庶女顶嫁案：这布局不是一般人想出来的
    文/花漫 我一把揭开白布。 她就那样静静地躺着，像睡着了一般。 火红的嫁衣衬着肌肤如雪。 梳的纹丝不乱的头发上，一...
    开封第一讲书人阅读 49,007评论 1赞 284
  • 城市分裂传说
    那天，我揣着相机与录音，去河边找鬼。 笑死，一个胖子当着我的面吹牛，可吹牛的内容都是我干的。 我是一名探鬼主播，决...
    沈念sama阅读 38,313评论 3赞 399
  • 双鸳鸯连环套：你想象不到人心有多黑
    文/苍兰香墨 我猛地睁开眼，长吁一口气：“原来是场噩梦啊……” “哼！你这毒妇竟也来了？” 一声冷哼从身侧响起，我...
    开封第一讲书人阅读 36,956评论 0赞 259
  • 万荣杀人案实录
    序言：老挝万荣一对情侣失踪，失踪者是张志新（化名）和其女友刘颖，没想到半个月后，有当地人在树林里发现了一具尸体，经...
    沈念sama阅读 43,441评论 1赞 300
  • 护林员之死
    正文 独居荒郊野岭守林人离奇死亡，尸身上长有42处带血的脓包…… 初始之章·张勋 以下内容为张勋视角 年9月15日...
    茶点故事阅读 35,925评论 2赞 323
  • 白月光启示录
    正文 我和宋清朗相恋三年，在试婚纱的时候发现自己被绿了。 大学时的朋友给我发了我未婚夫和他白月光在一起吃饭的照片。...
    茶点故事阅读 38,018评论 1赞 333
  • 活死人
    序言：一个原本活蹦乱跳的男人离奇死亡，死状恐怖，灵堂内的尸体忽然破棺而出，到底是诈尸还是另有隐情，我是刑警宁泽，带...
    沈念sama阅读 33,685评论 4赞 322
  • 日本核电站爆炸内幕
    正文 年R本政府宣布，位于F岛的核电站，受9级特大地震影响，放射性物质发生泄漏。R本人自食恶果不足惜，却给世界环境...
    茶点故事阅读 39,234评论 3赞 307
  • 男人毒药：我在死后第九天来索命
    文/蒙蒙 一、第九天 我趴在偏房一处隐蔽的房顶上张望。 院中可真热闹，春花似锦、人声如沸。这庄子的主人今日做“春日...
    开封第一讲书人阅读 30,240评论 0赞 19
  • 一桩弑父案，背后竟有这般阴谋
    文/苍兰香墨 我抬头看了看天上的太阳。三九已至，却和暖如春，着一层夹袄步出监牢的瞬间，已是汗流浃背。 一阵脚步声响...
    开封第一讲书人阅读 31,464评论 1赞 261
  • 情欲美人皮
    我被黑心中介骗来泰国打工， 没想到刚下飞机就差点儿被人妖公主榨干…… 1. 我叫王不留，地道东北人。 一个月前我还...
    沈念sama阅读 45,467评论 2赞 352
  • 代替公主和亲
    正文 我出身青楼，却偏偏与公主长得像，于是被迫代替她去往敌国和亲。 传闻我的和亲对象是个残疾皇子，可洞房花烛夜当晚...
    茶点故事阅读 42,762评论 2赞 345

推荐阅读更多精彩内容