rke搭建k8s流程
-
工具篇
在命令行终端复制内容到配置文件时，遇到的格式错乱问题： vim 编辑 yaml 格式问题 在粘贴之前执行以下命令 ：set paste
-
部署架构
总共6台服务器： 针对每台主机设置主机名，且配置每台主机都能相互访问 每台服务器的/etc/hosts要配置正确，一定要有127.0.0.1 localhost 这一项 hostnamectl set-hostname lb-1 hostnamectl set-hostname lb-2 hostnamectl set-hostname k8s-master-1 hostnamectl set-hostname k8s-master-2 hostnamectl set-hostname k8s-master-3 hostnamectl set-hostname k8s-worker-1 cat >> /etc/hosts << EOF 192.168.0.201 lb-1 192.168.0.202 lb-2 192.168.0.211 k8s-master-1 192.168.0.212 k8s-master-2 192.168.0.213 k8s-master-3 192.168.0.221 k8s-worker-1 EOF服务器部署架构：
lb-1,lb-2 作为集群的流量入口，承担负载均衡作用， lb服务器需用keepalived配置VIP 192.168.0.200，负载均衡软件可用nginx也可用haproxy k8s-master-1,k8s-master-2,k8s-master-3 作为主节点的高可用部署 k8s-worker-1，k8s-worker-n 作为工作节点
-
系统和软件版本
系统版本：centos7.9 8.x docker版本：20.10.12 docker-compose rke版本：rke1.3.3 下载地址：https://github.com/rancher/rke rancher版本：rancher/hyperkube:v1.20.13-rancher1 keepalived nginx 或 haproxycentos修改国内源
cd /etc/yum.repos.d/ mv CentOS-Base.repo CentOS-Base.repo.bak wget -O CentOs-Base.repo http://mirrors.aliyun.com/repo/Centos-7.repo #yum源更新命令 yum clean all yum makecache yum updatedocker安装
添加阿里云的docker源 yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo yum install -y yum-utils device-mapper-persistent-data lvm2 yum -y install docker-ce修改docker/daemon.json配置文件 sudo mkdir -p /etc/docker cat <<EOF > /etc/docker/daemon.json { "exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "registry-mirrors": ["https://cenq021s.mirror.aliyuncs.com"] } EOF systemctl daemon-reload & systemctl restart docker & systemctl enable docker 对于运行 Linux 内核版本 4.0 或更高版本，或使用 3.10.0-51 及更高版本的 RHEL 或 CentOS 的系统，`overlay2`是首选的存储驱动程序。 如发现无法启动，则在配置文件中去除 ： "storage-driver": "overlay2", "log-driver": "json-file", "log-opts": { "max-size": "100m" },可能遇到的问题： yum-config-manager: command not found 需安装yum-utils yum -y install yum-utils k8s原生安装的时候，docker版本需与k8s版本一直,具体k8s的github中可以查询 如需安装指定版本docker: yum install docker-ce-19.03.* -y 已安装高版的docker降级到指定版本 yum downgrade --setopt=obsoletes=0 -y docker-ce-19.03.13 docker-ce-selinux-19.03.13docker-compose安装
yum -y install yum-utils 安装docker-compse sudo yum -y install epel-release yum install docker-compose
-
系统内核参数修改
关闭防火墙
由于有网络防火墙，系统自带的firewall防火墙可以关闭； systemctl stop firewalld systemctl disable firewalld 常用命令： #防火墙操作命令 备用 firewall-cmd --zone=public --remove-port=80/tcp --permanent 配置立即生效 firewall-cmd --reload 查看防火墙状态 systemctl status firewalld 关闭防火墙 systemctl stop firewalld 打开防火墙 systemctl start firewalld如不想关闭防火墙的，可以按照以下端口规则开放：
协议 端口 描述 TCP 32289 使用主机驱动通过 SSH 进行节点配置 TCP 2376 主机驱动与 Docker 守护进程通信的 TLS 端口 TCP 2379 etcd 客户端请求 TCP 2380 etcd 节点通信 TCP 179 Calico BGP 端口 UDP 8472 Canal/Flannel VXLAN overlay 网络 UDP 4789 Canal/Flannel VXLAN overlay 网络 TCP 9099 Canal/Flannel 健康检查 TCP 9100 Monitoring 从 Linux node-exporters 中抓取指标所需的默认端口 UDP 8443 Rancher webhook TCP 9443 Rancher webhook TCP 9796 集群监控拉取节点指标的默认端口 TCP 6783 Weave 端口 UDP 6783-6784 Weave UDP 端口 TCP 10250 Metrics server 与所有节点的通信 TCP 10254 Ingress controller 健康检查 TCP/UDP 30000-32767 NodePort 端口范围 TCP 6443 apiserver TCP 80 Ingress controller TCP 443 Ingress controller 关闭SELINUX
永久关闭： 修改/etc/selinux/config这个配置文件 sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config 查看状态 setenforce status禁用swap分区
vim /etc/fstab #/dev/mapper/cl-swap swap各服务器设置时间同步
centos7用ntp的方式 centos8用chrony yum install ntp -y 修改配置文件：time.xxx.com为你们的时间服务器地址，如果没有可以用阿里的 ntp.aliyun.com vim /etc/ntp.conf server time.xxx.com iburst 执行时间同步 ntpdate time.xxx.edu.cn 重启服务和设置开机自启动 systemctl restart ntpd & systemctl enable ntpd ---------------------------------------------------------- chrony模式 centos 时间同步 vim /etc/chrony.conf 添加时间服务器 server ntp.aliyun.com iburst 重启 systemctl restart chronyd.service 同步时间 chronyc sources -v设置内核参数
cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf br_netfilter EOF cat <<EOF > /etc/sysctl.d/k8s.conf net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-iptables = 1 net.bridge.bridge-nf-call-ip6tables = 1 fs.may_detach_mounts = 1 vm.overcommit_memory=1 vm.panic_on_oom=0 fs.inotify.max_user_watches=89100 fs.file-max=52706963 fs.nr_open=52706963 net.netfilter.nf_conntrack_max=2310720 net.ipv4.tcp_keepalive_time = 600 net.ipv4.tcp_keepalive_probes = 3 net.ipv4.tcp_keepalive_intvl =15 net.ipv4.tcp_max_tw_buckets = 36000 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_max_orphans = 327680 net.ipv4.tcp_orphan_retries = 3 net.ipv4.tcp_syncookies = 1 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.ip_conntrack_max = 65536 net.ipv4.tcp_max_syn_backlog = 16384 net.ipv4.tcp_timestamps = 0 net.core.somaxconn = 16384 EOF sudo sysctl --system安装相关命令组件
yum install ipvsadm ipset sysstat conntrack libseccomp -y
# 加入以下内容 centos7中修改ipvs.conf文件会导致模块无法启动，centos8中正常
cat <<EOF > /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
systemctl enable --now systemd-modules-load.service
如因系统版本过高,加载报错，需要将 nf_conntrack_ipv4 替换为：nf_conntrack
cat <<EOF > /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
EOF
-
rke部署
选择一台服务器作为部署节点, 并下载rke:
从rke的github中下载最新的release版， 我这边选择的是1.3.3 https://github.com/rancher/rke wget https://github.com/rancher/rke/releases/download/v1.3.3/rke_linux-amd64 mv rke_linux-amd64 /usr/local/bin/rke && chmod +x /usr/local/bin/rke为每一台服务器创建一个用户部署k8s的专用用户，该用户需要能执行docker命令的权限，以便rke程序能通过该用户自动部署。
# useradd ops # usermod -a -G docker ops配置部署服务器能免密登陆各节点：
#su – ops #ssh-keygen -t rsa -b 4096 su - ops ssh-copy-id ops@192.168.0.201 ssh-copy-id ops@192.168.0.202 ssh-copy-id ops@192.168.0.211 ssh-copy-id ops@192.168.0.212 ssh-copy-id ops@192.168.0.213 ssh-copy-id ops@192.168.0.221运行rke生成配置文件
rke config 会弹出一系列对话选项，逐个配置即可，最终会生成 cluster.yml文件 注意点： 1、id_rsa不要配错 2、ip不要填错 3、ssh用户和端口不要填错，并确保安装服务器的用户能免密登陆到各个节点 4、rke版本不要配错，rancher/hyperkube:v1.20.13-rancher1 可以从github中查找rke所支持的对应版本。 5、其他的基本上默认即可 需要ETCD定时备份的，要更改一下配置文件 services: etcd: snapshot: true creation: 6h retention: 24h 执行部署命令 rke up --config ./cluster.yml 部署成功后会生成以下文件： kube_config_cluster.yml cluster.rkestate PS： ****** kube_config_cluster.yml cluster.rkestate cluster.yml 这3份文件很重要，一定要保存好。 ****** 部署最终可能会出现以下错误,执行rke的更新命令即可： FATA[0668] Failed to get job complete status for job rke-network-plugin-deploy-job in namespace kube-system rke up --update-only --config ./cluster.yml 新增、删除节的流程： 1、修改cluster.yml的配置文件 2、执行rke命令 rke up --update-only --config ./cluster.yml安装kubectl
cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF yum install -y kubectl-1.20.0 systemctl enable kubectl mkdir $HOME/.kube && cp kube_config_cluster.yml $HOME/.kube/config kubectl --kubeconfig=$KUBECONFIG -- kube_config_rancher-cluster.yml 该文件为配置服务器上生成的文件。 即可在命令行查看k8s集群状态： kubectl get nodes kubectl get pods -A -o wide 强制删除 kubectl delete pods httpd-app-6df58645c6-cxgcm --grace-period=0 --force
-
安装rancher
生成rancher所需要的证书：可以是自己的证书文件，也可用脚本生成的
#!/bin/bash -e help () { echo ' ================================================================ ' echo ' --ssl-domain: 生成ssl证书需要的主域名，如不指定则默认为www.rancher.local，如果是ip访问服务，则可忽略；' echo ' --ssl-trusted-ip: 一般ssl证书只信任域名的访问请求，有时候需要使用ip去访问server，那么需要给ssl证书添加扩展IP，多个IP用逗号隔开；' echo ' --ssl-trusted-domain: 如果想多个域名访问，则添加扩展域名（SSL_TRUSTED_DOMAIN）,多个扩展域名用逗号隔开；' echo ' --ssl-size: ssl加密位数，默认2048；' echo ' --ssl-date: ssl有效期，默认10年；' echo ' --ca-date: ca有效期，默认10年；' echo ' --ssl-cn: 国家代码(2个字母的代号),默认CN;' echo ' 使用示例:' echo ' ./create_self-signed-cert.sh --ssl-domain=www.test.com --ssl-trusted-domain=www.test2.com \ ' echo ' --ssl-trusted-ip=1.1.1.1,2.2.2.2,3.3.3.3 --ssl-size=2048 --ssl-date=3650' echo ' ================================================================' } case "$1" in -h|--help) help; exit;; esac if [[ $1 == '' ]];then help; exit; fi CMDOPTS="$*" for OPTS in $CMDOPTS; do key=$(echo ${OPTS} | awk -F"=" '{print $1}' ) value=$(echo ${OPTS} | awk -F"=" '{print $2}' ) case "$key" in --ssl-domain) SSL_DOMAIN=$value ;; --ssl-trusted-ip) SSL_TRUSTED_IP=$value ;; --ssl-trusted-domain) SSL_TRUSTED_DOMAIN=$value ;; --ssl-size) SSL_SIZE=$value ;; --ssl-date) SSL_DATE=$value ;; --ca-date) CA_DATE=$value ;; --ssl-cn) CN=$value ;; esac done # CA相关配置 CA_DATE=${CA_DATE:-3650} CA_KEY=${CA_KEY:-cakey.pem} CA_CERT=${CA_CERT:-cacerts.pem} CA_DOMAIN=cattle-ca # ssl相关配置 SSL_CONFIG=${SSL_CONFIG:-$PWD/openssl.cnf} SSL_DOMAIN=${SSL_DOMAIN:-'www.rancher.local'} SSL_DATE=${SSL_DATE:-3650} SSL_SIZE=${SSL_SIZE:-2048} ## 国家代码(2个字母的代号),默认CN; CN=${CN:-CN} SSL_KEY=$SSL_DOMAIN.key SSL_CSR=$SSL_DOMAIN.csr SSL_CERT=$SSL_DOMAIN.crt echo -e "\033[32m ---------------------------- \033[0m" echo -e "\033[32m | 生成 SSL Cert | \033[0m" echo -e "\033[32m ---------------------------- \033[0m" if [[ -e ./${CA_KEY} ]]; then echo -e "\033[32m ====> 1. 发现已存在CA私钥，备份"${CA_KEY}"为"${CA_KEY}"-bak，然后重新创建 \033[0m" mv ${CA_KEY} "${CA_KEY}"-bak openssl genrsa -out ${CA_KEY} ${SSL_SIZE} else echo -e "\033[32m ====> 1. 生成新的CA私钥 ${CA_KEY} \033[0m" openssl genrsa -out ${CA_KEY} ${SSL_SIZE} fi if [[ -e ./${CA_CERT} ]]; then echo -e "\033[32m ====> 2. 发现已存在CA证书，先备份"${CA_CERT}"为"${CA_CERT}"-bak，然后重新创建 \033[0m" mv ${CA_CERT} "${CA_CERT}"-bak openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}" else echo -e "\033[32m ====> 2. 生成新的CA证书 ${CA_CERT} \033[0m" openssl req -x509 -sha256 -new -nodes -key ${CA_KEY} -days ${CA_DATE} -out ${CA_CERT} -subj "/C=${CN}/CN=${CA_DOMAIN}" fi echo -e "\033[32m ====> 3. 生成Openssl配置文件 ${SSL_CONFIG} \033[0m" cat > ${SSL_CONFIG} <<EOM [req] req_extensions = v3_req distinguished_name = req_distinguished_name [req_distinguished_name] [ v3_req ] basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, serverAuth EOM if [[ -n ${SSL_TRUSTED_IP} || -n ${SSL_TRUSTED_DOMAIN} ]]; then cat >> ${SSL_CONFIG} <<EOM subjectAltName = @alt_names [alt_names] EOM IFS="," dns=(${SSL_TRUSTED_DOMAIN}) dns+=(${SSL_DOMAIN}) for i in "${!dns[@]}"; do echo DNS.$((i+1)) = ${dns[$i]} >> ${SSL_CONFIG} done if [[ -n ${SSL_TRUSTED_IP} ]]; then ip=(${SSL_TRUSTED_IP}) for i in "${!ip[@]}"; do echo IP.$((i+1)) = ${ip[$i]} >> ${SSL_CONFIG} done fi fi echo -e "\033[32m ====> 4. 生成服务SSL KEY ${SSL_KEY} \033[0m" openssl genrsa -out ${SSL_KEY} ${SSL_SIZE} echo -e "\033[32m ====> 5. 生成服务SSL CSR ${SSL_CSR} \033[0m" openssl req -sha256 -new -key ${SSL_KEY} -out ${SSL_CSR} -subj "/C=${CN}/CN=${SSL_DOMAIN}" -config ${SSL_CONFIG} echo -e "\033[32m ====> 6. 生成服务SSL CERT ${SSL_CERT} \033[0m" openssl x509 -sha256 -req -in ${SSL_CSR} -CA ${CA_CERT} \ -CAkey ${CA_KEY} -CAcreateserial -out ${SSL_CERT} \ -days ${SSL_DATE} -extensions v3_req \ -extfile ${SSL_CONFIG} echo -e "\033[32m ====> 7. 证书制作完成 \033[0m" echo echo -e "\033[32m ====> 8. 以YAML格式输出结果 \033[0m" echo "----------------------------------------------------------" echo "ca_key: |" cat $CA_KEY | sed 's/^/ /' echo echo "ca_cert: |" cat $CA_CERT | sed 's/^/ /' echo echo "ssl_key: |" cat $SSL_KEY | sed 's/^/ /' echo echo "ssl_csr: |" cat $SSL_CSR | sed 's/^/ /' echo echo "ssl_cert: |" cat $SSL_CERT | sed 's/^/ /' echo echo -e "\033[32m ====> 9. 附加CA证书到Cert文件 \033[0m" cat ${CA_CERT} >> ${SSL_CERT} echo "ssl_cert: |" cat $SSL_CERT | sed 's/^/ /' echo echo -e "\033[32m ====> 10. 重命名服务证书 \033[0m" echo "cp ${SSL_DOMAIN}.key tls.key" cp ${SSL_DOMAIN}.key tls.key echo "cp ${SSL_DOMAIN}.crt tls.crt" cp ${SSL_DOMAIN}.crt tls.crt生成证书：
把上面的执行脚本保存到key.sh文件，且赋予chmod +x kye.sh 执行权限
mkdir ./rancher-ssl vim ./key.sh #加入上述脚本 chmod +x kye.sh ./key.sh --ssl-domain=rancher.xxx.com --ssl-trusted-domain=rancher2.xxx.com --ssl-trusted-ip=192.168.0.211,192.168.0.212,192.168.0.213,192.168.0.221 --ssl-size=2048 --ssl-date=36500 会生成一堆证书文件，需要保存好 注意： --ssl-domain 可信任的域名 --ssl-trusted-ip 可信任的节点IPk8s环境中配置证书：
#创建命名空间 kubectl create namespace cattle-system #设置证书 kubectl -n cattle-system create secret generic tls-ca --from-file=cacerts.pem cp cacerts.pem ca-additional.pem kubectl -n cattle-system create secret generic tls-ca-additional --from-file=ca-additional.pem kubectl -n cattle-system create secret tls tls-rancher-ingress --cert=tls.crt --key=tls.key 如出现证书已存在的情况，需先删除证书： kubecelt -n cattle-system delete secret tls-ca kubecelt -n cattle-system delete secret tls-ca-additional kubecelt -n cattle-system delete secret tls-rancher-ingress安装helm：这里选择用helm生成rancher的安装yaml文件
从github上下载helm的2进制文件 https://github.com/helm/helm tar -zxvf helm-v3.3.0-linux-amd64.tar.gz cd linux-amd64 mv helm /usr/local/bin/helm && chmod +x /usr/local/bin/helm #添加rancher helm 仓库 helm repo add rancher-stable https://releases.rancher.com/server-charts/stable #查看rancher所有版本 helm search repo rancher -l helm fetch rancher-stable/rancher --version 2.5.11 当前目录会多一个rancher-2.5.11.tgz 使用以下命令渲染模板： helm template rancher ./rancher-2.5.11.tgz \ --namespace cattle-system --output-dir . \ --set privateCA=true \ --set additionalTrustedCAs=true \ --set ingress.tls.source=secret \ --set hostname=rancher.toowe.com \ --set useBundledSystemChart=true 渲染后会生成一个rancher目录，其中ingress的配置文件需要修改一下 ingress.yaml文件修改后如下： apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: rancher labels: app: rancher chart: rancher-2.5.11 heritage: Helm release: rancher annotations: nginx.ingress.kubernetes.io/proxy-connect-timeout: "30" nginx.ingress.kubernetes.io/proxy-read-timeout: "1800" nginx.ingress.kubernetes.io/proxy-send-timeout: "1800" spec: rules: - host: rancher.toowe.com # hostname to access rancher server http: paths: - path: / pathType: Prefix backend: service: name: rancher port: number: 80 # - backend: # serviceName: rancher # servicePort: 80 tls: - hosts: - rancher.toowe.com secretName: tls-rancher-ingress 使用kubectl安装rancher kubectl -n cattle-system apply -R -f ./rancher/templates/ 报错： Warning: networking.k8s.io/v1beta1 Ingress is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress kubectl -n cattle-system delete -R -f ./rancher/templates/ingress.yaml kubectl -n cattle-system get all 安装完成
-
安装lb负载均衡
安装keepalived
yum install keepalived -y 修改keepalived 配置文件，每个lb节点上都需要修改，注意配置文件中带#部分 global_defs { notification_email { user@example.com } notification_email_from mail@example.org smtp_server 192.168.x.x smtp_connect_timeout 30 router_id LVS_MASTER # 每个节点名称要唯一 } #监测haproxy进程状态，每2秒执行一次 如果nginx则是监听nginx vrrp_script chk_haproxy { script "/bin/bash -c 'if [[ $(netstat -nlp | grep 16443) ]]; then exit 0; else exit 1; fi'" interval 2 weight 2 } vrrp_instance VI_1 { state MASTER #标示状态为MASTER interface enp0s3 virtual_router_id 51 priority 101 #MASTER权重要高于BACKUP advert_int 1 unicast_src_ip 192.168.0.183 #当前机器地址 unicast_peer { 192.168.0.201 #peer中其它地址 192.168.0.202 #peer中其它地址 } authentication { auth_type PASS #主从服务器验证方式 auth_pass 1111 } track_script { chk_haproxy #监测haproxy进程状态 } #VIP virtual_ipaddress { 192.168.0.200 #虚拟IP } } systemctl daemon-reload systemctl enable keepalived systemctl start keepalived安装haproxy 这里也可用nginx替代
yum install haproxy -y 修改haproxy配置 #--------------------------------------------------------------------- # Example configuration for a possible web application. See the # full configuration options online. # # https://www.haproxy.org/download/1.8/doc/configuration.txt # #--------------------------------------------------------------------- #--------------------------------------------------------------------- # Global settings #--------------------------------------------------------------------- global # to have these messages end up in /var/log/haproxy.log you will # need to: # # 1) configure syslog to accept network log events. This is done # by adding the '-r' option to the SYSLOGD_OPTIONS in # /etc/sysconfig/syslog # # 2) configure local2 events to go to the /var/log/haproxy.log # file. A line like the following can be added to # /etc/sysconfig/syslog # # local2.* /var/log/haproxy.log # log 127.0.0.1 local2 chroot /var/lib/haproxy pidfile /var/run/haproxy.pid maxconn 40000 user haproxy group haproxy daemon # turn on stats unix socket stats socket /var/lib/haproxy/stats # utilize system-wide crypto-policies ssl-default-bind-ciphers PROFILE=SYSTEM ssl-default-server-ciphers PROFILE=SYSTEM #--------------------------------------------------------------------- # common defaults that all the 'listen' and 'backend' sections will # use if not designated in their block #--------------------------------------------------------------------- defaults mode http log global option httplog option dontlognull option http-server-close option forwardfor except 127.0.0.0/8 option redispatch retries 3 timeout http-request 10s timeout queue 1m timeout connect 10s timeout client 1m timeout server 1m timeout http-keep-alive 10s timeout check 10s maxconn 3000 #--------------------------------------------------------------------- # kubernetes apiserver frontend which proxys to the backends #--------------------------------------------------------------------- frontend rancher-forntend mode tcp bind *:443 option tcplog default_backend rancher-backend #--------------------------------------------------------------------- # round robin balancing between the various backends #--------------------------------------------------------------------- backend rancher-backend mode tcp balance roundrobin server node-0 192.168.0.211:443 check server node-1 192.168.0.212:443 check server node-2 192.168.0.222:443 check listen admin_stats bind 0.0.0.0:19198 mode http log 127.0.0.1 local3 err #HAProxy监控页面统计自动刷新时间。 stats refresh 30s #设置监控页面URL路径。 http：//IP:19198/haproxy-status可查看 stats uri /haproxy-status #统计页面密码框提示信息 stats realm welcome login\ Haproxy #登录统计页面用户和密码 stats auth toowe:toowe #隐藏HAProxy版本信息 stats hide-version #设置TURE后可在监控页面手工启动关闭后端真实服务器 stats admin if TRUE
-
卸载k8s，可忽略
cat > clear.sh << EOF df -h|grep kubelet |awk -F % '{print $2}'|xargs umount rm /var/lib/kubelet/* -rf rm /etc/kubernetes/* -rf rm /var/lib/rancher/* -rf rm /var/lib/etcd/* -rf rm /var/lib/cni/* -rf rm -rf /var/run/calico iptables -F && iptables -t nat -F ip link del flannel.1 docker ps -a|awk '{print $1}'|xargs docker rm -f docker volume ls|awk '{print $2}'|xargs docker volume rm rm -rf /var/etcd/ rm -rf /run/kubernetes/ docker rm -fv $(docker ps -aq) docker volume rm $(docker volume ls) rm -rf /etc/cni rm -rf /opt/cni systemctl restart docker EOF #删除容器 sudo docker stop `sudo docker ps -aq` sudo docker rm -f `sudo docker ps -aq` #删除挂载卷 sudo docker volume rm $(sudo docker volume ls -q) for mount in $(mount tmpfs |grep '/vsr/lib/kubelet' |awk '{print $3}') ; do sudo umount $mount; done sudo mount |grep tmpfs |grep '/var/lib/kubelet' |awk '{print $3}' sudo umount /var/run/docker/netns/default #删除相关文件 sudo rm -rf /etc/cni sudo rm -rf /etc/kubernetes sudo rm -rf /opt/cni sudo rm -rf /opt/rke sudo rm -rf /run/secrets/kubernetes.io sudo rm -rf /run/calico sudo rm -rf /var/lib/etcd sudo rm -rf /var/lib/cni sudo rm -rf /var/lib/kubelet sudo rm -rf /var/log/containers sudo rm -rf /var/log/pods sudo rm -rf /var/lib/rancher sudo rm -rf /var/run/calico sudo rm -rf /var/run/docker sudo rm -rf /var/lib/docker sudo rm -rf /app/docker
-
自建ETCD集群，可忽略
wget https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64 #注意，以上链接若打不开，直接使用我提供的软件即可！ 设置cfssl执行权限 chmod +x cfssl* for x in cfssl*; do mv $x ${x%*_linux-amd64}; done mv cfssl* /usr/bin 创建生成证书目录 mkdir -p ~/etcd_tls cd ~/etcd_tls etcd证书json cat > ca-config.json << EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "www": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json << EOF { "CN": "etcd CA", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" } ] } EOF cfssl gencert -initca ca-csr.json | cfssljson -bare ca - cat > server-csr.json << EOF { "CN": "etcd", "hosts": [ "192.168.0.179", "192.168.0.48", "192.168.0.163" ], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=www server-csr.json | cfssljson -bare server etcd安装文件 mkdir /opt/etcd/{bin,cfg,ssl} -p tar zxvf etcd-v3.4.9-linux-amd64.tar.gz mv etcd-v3.4.9-linux-amd64/{etcd,etcdctl} /opt/etcd/bin/ etcd配置文件 cat > /opt/etcd/cfg/etcd.conf << EOF #[Member] ETCD_NAME="etcd-1" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" ETCD_LISTEN_PEER_URLS="https://192.168.0.179:2380" #2380是 集群通信的端口； ETCD_LISTEN_CLIENT_URLS="https://192.168.0.179:2379" #2379是指它的数据端口，其他客户端要访问etcd数据库的读写都走的是这个端口； #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.0.179:2380" ETCD_ADVERTISE_CLIENT_URLS="https://192.168.0.179:2379" ETCD_INITIAL_CLUSTER="etcd-1=https://192.168.0.179:2380,etcd-2=https://192.168.0.48:2380,etcd-3=https://192.168.0.163:2380" ETCD_INITIAL_CLUSTER_TOKEN="etcd-cluster" #一种简单的认证机制，网络里可能配置了多套k8s集群，防止误同步； ETCD_INITIAL_CLUSTER_STATE="new" EOF etcd 执行脚本，设置证书路径 cat > /usr/lib/systemd/system/etcd.service << EOF [Unit] Description=Etcd Server After=network.target After=network-online.target Wants=network-online.target [Service] Type=notify EnvironmentFile=/opt/etcd/cfg/etcd.conf ExecStart=/opt/etcd/bin/etcd \ --cert-file=/opt/etcd/ssl/server.pem \ --key-file=/opt/etcd/ssl/server-key.pem \ --trusted-ca-file=/opt/etcd/ssl/ca.pem \ --peer-cert-file=/opt/etcd/ssl/server.pem \ --peer-key-file=/opt/etcd/ssl/server-key.pem \ --peer-trusted-ca-file=/opt/etcd/ssl/ca.pem \ --logger=zap Restart=on-failure LimitNOFILE=65536 [Install] WantedBy=multi-user.target EOF 拷贝证书 cp ~/etcd_tls/ca*pem ~/etcd_tls/server*pem /opt/etcd/ssl/ 启动 systemctl daemon-reload systemctl start etcd systemctl enable etcd scp -r /opt/etcd/ root@192.168.0.48:/opt/ scp /usr/lib/systemd/system/etcd.service root@192.168.0.48:/usr/lib/systemd/system/ scp -r /opt/etcd/ root@192.168.0.163:/opt/ scp /usr/lib/systemd/system/etcd.service root@192.168.0.163:/usr/lib/systemd/system/ 修改每个节点上的etcd.conf文件 ETCD_NAME 每个配置文件唯一 ETCD_LISTEN_PEER_URLS ETCD_LISTEN_CLIENT_URLS ETCD_INITIAL_ADVERTISE_PEER_URLS ETCD_ADVERTISE_CLIENT_URLS 都设置为本机IP etcd 集群检测 ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/opt/etcd/ssl/ca.pem --cert=/opt/etcd/ssl/server.pem --key=/opt/etcd/ssl/server-key.pem --endpoints="https://192.168.0.179:2379,https://192.168.0.48:2379,https://192.168.0.163:2379" endpoint health --write-out=table
- 待续
