公网部署K8S
场景：
- 跨云厂商的机器
- 非同一网段的机器
机器准备：
Linux 主机3台 (CETOS7.6 2CPU 4G)
节点 MAC 地址 和 product_uuid 的唯一性
端口检查:
nc 127.0.0.1 6443-
申请域名，解析指向master机器IP（可选）
master node1 node2 IP 114.132.94.160 43.138.235.139 43.139.23.242 系统 Centos7.6 Centos7.6 Centos7.6
版本说明：
- K8S @ v1.25
- docker-engine @ 20.10.18
过程中问题整理：
目前已整理的过程中问题， 更多问题欢迎大家以issue方式提交给我， 谢谢。
部署步骤
- 清除旧K8S部署痕迹
- 安装docker(每个主机)
- 配置cri-docker使kubernetes以docker作为运行时(每个主机)
- 配置基础环境(每个主机)
- 建立虚拟机网卡(每个主机)
- 安装kubernetes(每个主机)
- 修改kubelet启动参数文件(每个主机)
- 开启云服务器端口(每个主机)
- 初始化集群(Master)
- 工作节点加入集群(Node)
- 安装flannel网络插件(Master)
- 部署nginx验证安装
清除旧K8S部署痕迹
- 安装失败需要reset集群
sudo kubeadm reset --cri-socket /var/run/cri-dockerd.sock - 清除文件
rm -rf /root/.kube/ sudo rm -rf /etc/kubernetes/ sudo rm -rf /var/lib/kubelet/ sudo rm -rf /var/lib/dockershim sudo rm -rf /var/run/kubernetes sudo rm -rf /var/lib/cni sudo rm -rf /var/lib/etcd sudo rm -rf /etc/cni/net.d - 删除掉k8s对本机网卡iptables转发的配置
iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X ipvsadm -C ipvsadm --clear - 执行 ip addr 会发现还有一些虚拟veth、cni、flannel等设备
ip link delete xxx
安装dcoker
- 已安装建议删除，重新安装
# 杀死所有运行容器 docker kill $(docker ps -a -q) # 删除所有容器 docker rm $(docker ps -a -q) # 删除所有镜像 docker rmi $(docker images -q) # 停止 docker 服务 systemctl stop docker # 删除存储目录 rm -rf /etc/docker rm -rf /run/docker rm -rf /var/lib/dockershim rm -rf /var/lib/docker # 卸载 docker yum remove docker docker-engine docker-common docker-selinux - yum安装docker
# 安装部分依赖 yum install -y yum-utils device-mapper-persistent-data lvm2 # 添加docker yum源 sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo # 安装docker yum install -y docker-ce docker-ce-selinux # 启动docker并设置开机自启动 systemctl enable docker systemctl start docker
配置cri-docker使kubernetes以docker作为运行时
- 下载最新版 cri-docker
- 解压出cri-docker
tar -zxf cri-dockerd-0.2.5.amd64.tgz cp cri-dockerd/cri-dockerd /usr/bin/ - 创建cri-docker启动文件
cat > /usr/lib/systemd/system/cri-docker.service << EOF [Unit] Description=CRI Interface for Docker Application Container Engine Documentation=https://docs.mirantis.com After=network-online.target firewalld.service docker.service Wants=network-online.target Requires=cri-docker.socket [Service] Type=notify ExecStart=/usr/bin/cri-dockerd --network-plugin=cni --pod-infra-container-image=registry.aliyuncs.com/google_containers/pause:3.8 ExecReload=/bin/kill -s HUP $MAINPID TimeoutSec=0 RestartSec=2 Restart=always StartLimitBurst=3 StartLimitInterval=60s LimitNOFILE=infinity LimitNPROC=infinity LimitCORE=infinity TasksMax=infinity Delegate=yes KillMode=process [Install] WantedBy=multi-user.target EOFcat > /usr/lib/systemd/system/cri-docker.socket << EOF [Unit] Description=CRI Docker Socket for the API PartOf=cri-docker.service [Socket] ListenStream=%t/cri-dockerd.sock SocketMode=0660 SocketUser=root SocketGroup=docker [Install] WantedBy=sockets.target EOF - 启动cri-docker并设置开机自动启动
systemctl daemon-reload systemctl enable cri-docker --now systemctl status cri-docker
配置基础环境
- 准备工作
# 禁用防火墙和iptables systemctl stop firewalld systemctl disable firewalld systemctl stop iptables systemctl disable iptables # 禁用selinux sudo setenforce 0 sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config # 禁用swap分区 swapoff -a sed -i '/swap/s/^/#/' /etc/fstab #允许 iptables 检查桥接流量 cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf br_netfilter EOF cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1 EOF sudo sysctl --system
建立虚拟机网卡
- 创建一个虚拟网卡
cat > /etc/sysconfig/network-scripts/ifcfg-eth0:1 <<EOF BOOTPROTO=static DEVICE=eth0:1 IPADDR=机器公网IP PREFIX=32 TYPE=Ethernet USERCTL=no ONBOOT=yes EOF - 重启网卡及检查
systemctl restart network ip addr
安装kubernetes
- 添加阿里云k8s镜像源（国内网络）
cat > /etc/yum.repos.d/kubernetes.repo << EOF [kubernetes] name=Kubernetes baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64 enabled=1 gpgcheck=0 repo_gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg EOF - 执行yum命令
sudo yum install -y kubelet kubeadm kubectl--disableexcludes=kubernetes - 设置kubelet开机自启
systemctl enable kubelet.service
修改kubelet启动参数文件
添加 kubelet的启动参数--node-ip=公网IP， 每个主机都要添加并指定对应的公网ip
- 修改
/usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf, 此文件安装kubeadm后就存在了vim /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf - 在末尾添加参数 --node-ip=公网IP
..... ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS --node-ip=机器公网IP
开启云服务器端口
- 先默认开启所有的出入站规则
- 需要配置最小化Kubernetes出入站规则， 参考Kubernetes服务间需要用到的端口范围
初始化集群
- kubeadm init
sudo kubeadm init \ --kubernetes-version v1.25.0 \ --control-plane-endpoint=114.132.94.160 \ --apiserver-advertise-address=114.132.94.160 \ --image-repository registry.aliyuncs.com/google_containers \ --service-cidr=10.96.0.0/12 \ --pod-network-cidr=10.244.0.0/16 \ --v=5 \ --cri-socket /run/cri-dockerd.sock - 初始化kubectl
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config - 修改kube-apiserver参数
在 master 节点，kube-apiserver 添加--bind-address=0.0.0.0和修改--advertise-addres=<公网IP>
sudo vi /etc/kubernetes/manifests/kube-apiserver.yaml - 检查Nodes 及 Pods
kubectl get nodes -o wide kubectl get pods -o wide --all-namespaces
工作节点加入集群
- 使用
kubeadm join命令加入集群kubeadm join 114.132.94.160:6443 --token 0bd2ih.7afjzcq0lpcy17lt \ --discovery-token-ca-cert-hash sha256:fc83b436652b4c1501862ae971bab0fa1762de541e9115b6ecfcf1032033703b \ --cri-socket /var/run/cri-dockerd.sock - 如果token过期，可生成新token：
kubeadm token create --ttl 0 --print-join-command
安装flannel网络插件
- 下载flannel的yml文件
curl -OL https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml - 修改kube-flannel.yml文件
...略 apiVersion: apps/v1 kind: DaemonSet metadata: name: kube-flannel-ds-amd64 namespace: kube-system labels: tier: node app: flannel spec: selector: matchLabels: app: flannel template: metadata: labels: tier: node app: flannel spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: beta.kubernetes.io/os operator: In values: - linux - key: beta.kubernetes.io/arch operator: In values: - amd64 hostNetwork: true tolerations: - operator: Exists effect: NoSchedule serviceAccountName: flannel initContainers: - name: install-cni image: quay.io/coreos/flannel:v0.11.0-amd64 command: - cp args: - -f - /etc/kube-flannel/cni-conf.json - /etc/cni/net.d/10-flannel.conflist volumeMounts: - name: cni mountPath: /etc/cni/net.d - name: flannel-cfg mountPath: /etc/kube-flannel/ containers: - name: kube-flannel image: quay.io/coreos/flannel:v0.11.0-amd64 command: - /opt/bin/flanneld args: - --ip-masq - --public-ip=$(PUBLIC_IP) # 添加此参数，申明公网IP - --iface=eth0 # 添加此参数，绑定网卡 - --kube-subnet-mgr resources: requests: cpu: "100m" memory: "50Mi" limits: cpu: "100m" memory: "50Mi" securityContext: privileged: false capabilities: add: ["NET_ADMIN"] env: - name: PUBLIC_IP # 添加环境变量 valueFrom: # * fieldRef: # * fieldPath: status.podIP # * - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name ... 略 - 创建flannel
kubectl create -f kube-flannel.yml - 检查flannel Pod
kubectl logs -n kube-flannel pods/kube-flannel-ds-dj2cb kubectl logs -n kube-flannel pods/kube-flannel-ds-gxlx6 kubectl logs -n kube-flannel pods/kube-flannel-ds-rvxc4
部署nginx验证安装
- 新建
nginx-deployment.yamlapiVersion: v1 kind: Service metadata: name: nginx-service labels: app: nginx spec: ports: - protocol: TCP port: 80 targetPort: 80 selector: app: nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-deployment labels: app: nginx spec: selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: nginx:1.7.9 ports: - containerPort: 80 -
kubectl createkubectl create -f nginx-deployment.yaml - 验证
kubectl get svc -o wide --all-namespaces curl nginx-service对应的CLUSTER-IP
