使用kubeadm安装我们的kubernetes的测试环境
环境：CentOS Linux release 7.4.1708 (Core)
安装docker-Master节点
修改内核配置：
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
kernel.sysrq = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65023
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_max_orphans = 60000
net.ipv4.tcp_synack_retries = 3
net.core.somaxconn = 10000
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0
- 之后执行
sysctl -p
- 修改host配置，根据自己的IP修改，必须和hostname一致
hostnamectl set-hostname master
10.2.29.148 master
- 安装iptables
yum install iptables-devel.x86_64 iptables-services.x86_64 iptables.x86_64 -y
iptables -F
service iptables save
- 使用的是阿里云的docker镜像仓库
yum makecache
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo yum makecache fast
- 找到合适的docker版本，这里不推荐使用18.0以上的docker，原因有两点：
- 最新版的加入了太多的docker swarm的内容
- 没有和kubernetes做最新的适配
# yum list docker-ce --showduplicates
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
Available Packages
docker-ce.x86_64 17.03.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.03.3.ce-1.el7 docker-ce-stable
docker-ce.x86_64 17.06.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.06.2.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.09.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 17.12.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 18.03.0.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 18.03.1.ce-1.el7.centos docker-ce-stable
docker-ce.x86_64 18.06.0.ce-3.el7 docker-ce-stable
docker-ce.x86_64 18.06.1.ce-3.el7 docker-ce-stable
- 这里安装的是17.06.2.ce-1
- 因为阿里云kubernetes推荐使用的是17.06.2.ce-1
yum install docker-ce-17.06.2.ce-1.el7.centos
systemctl start docker.service
systemctl enable docker.service
- 配置docker的镜像加速仓库
# vim /etc/docker/daemon.json
{
"registry-mirrors": ["http://xxxxxxx.m.daocloud.io"],
"log-driver": "journald",
"log-opts": {
"tag": "{{.ImageName}}/{{.Name}}/{{.ID}}"
}
}
安装配置kubeadm
- 配置阿里云的kubernetes镜像仓库:
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum makecache fast
- 安装kubeadm cni 等工具
yum install -y kubernetes-cni kubelet kubeadm kubectl --skip-broken
systemctl start kubelet.service
systemctl enable kubelet.service
默认情况下这里的kubeadm kubectl kubelet默认安装的都是最新版本。如何查看kubernetes的最新版本是多少？
可以通过访问下面两个地址获得：
- https://dl.k8s.io/release/stable.txt
- https://storage.googleapis.com/kubernetes-release/release/stable.txt
注意：需要翻过6ZW/5Z+O6Ziy54Gr5aKZCg==的工具，记得开全局。(base64解码)
- 配置CNI网络配置
mkdir /etc/cni/net.d/ -p
cat >/etc/cni/net.d/10-mynet.conf <<-EOF
{
"cniVersion": "0.3.0",
"name": "mynet",
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.244.0.0/16",
"routes": [
{"dst": "0.0.0.0/0"}
]
}
}
EOF
cat >/etc/cni/net.d/99-loopback.conf <<-EOF
{
"cniVersion": "0.3.0",
"type": "loopback"
}
EOF
- 配置kubeadm的配置文件
kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha2
#kubernetesVersion: "stable"
kubernetesVersion: "v1.12.2"
apiServerCertSANs: []
#imageRepository: crproxy.trafficmanager.net:6000/google_containers
#imageRepository: mirrorgooglecontainers
imageRepository: registry.aliyuncs.com/google_containers
#imageRepository: ""
controllerManagerExtraArgs:
horizontal-pod-autoscaler-use-rest-clients: "true"
horizontal-pod-autoscaler-sync-period: "10s"
node-monitor-grace-period: "10s"
feature-gates: "AllAlpha=true"
enable-dynamic-provisioning: "true"
apiServerExtraArgs:
runtime-config: "api/all=true"
feature-gates: "AllAlpha=true"
#feature-gates: "CoreDNS=true"
networking:
podSubnet: "10.244.0.0/16"
说这国内的几个坑
- kubernetesVersion：如果没有指定具体的版本，kubeadm会访问网站https://dl.k8s.io/release/stable.txt 和 https://storage.googleapis.com/kubernetes-release/release/stable.txt
这两个地址都需要工具，才能拿到最新版本的kubernetes 的版本。 - 默认情况下使用的是谷歌爸爸的镜像仓库地址，这些对于5aSp5pydCg==来说都是奢望
- 还有apiVersion的版本会根据版本的变化而变化，一般改变的只是后面的数字
拉取镜像
# kubeadm --config kubeadm.yml config images pull
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-apiserver:v1.12.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-controller-manager:v1.12.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-scheduler:v1.12.2
[config/images] Pulled registry.aliyuncs.com/google_containers/kube-proxy:v1.12.2
[config/images] Pulled registry.aliyuncs.com/google_containers/pause:3.1
[config/images] Pulled registry.aliyuncs.com/google_containers/etcd:3.2.24
[config/images] Pulled registry.aliyuncs.com/google_containers/coredns:1.2.2
- 转换镜像tag，pause这个是每个POD必须的，有疑问的会在学习过程中了解到
docker tag registry.aliyuncs.com/google_containers/kube-proxy:v1.12.2 k8s.gcr.io/kube-proxy:v1.12.2
docker tag registry.aliyuncs.com/google_containers/kube-apiserver:v1.12.2 k8s.gcr.io/kube-apiserver:v1.12.2
docker tag registry.aliyuncs.com/google_containers/kube-controller-manager:v1.12.2 k8s.gcr.io/kube-controller-manager:v1.12.2
docker tag registry.aliyuncs.com/google_containers/kube-scheduler:v1.12.2 k8s.gcr.io/kube-scheduler:v1.12.2
docker tag registry.aliyuncs.com/google_containers/etcd:3.2.24 k8s.gcr.io/etcd:3.2.24
docker tag registry.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
docker tag registry.aliyuncs.com/google_containers/coredns:1.2.2 k8s.gcr.io/coredns:1.2.2
- 初始化
kubeadm init --config kubeadm.yml --ignore-preflight-errors all
执行成功后可以看到如下内容：
.....[addons] Applied essential addon: CoreDNS
[addons] Applied essential addon: kube-proxy
Your Kubernetes master has initialized successfully!
To start using your cluster, you need to run the following as a regular user:
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
https://kubernetes.io/docs/concepts/cluster-administration/addons/
You can now join any number of machines by running the following on each node
as root:
kubeadm join 10.2.29.148:6443 --token 5m131g.yz6x7217jjfz5w98 --discovery-token-ca-cert-hash sha256:2258ceec06ea8e5b855273cf4d7e45f4a83c42fbd7c3cce2331984418bf504c6
- 切换到一个普通用户执行：
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
查看节点：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 NotReady master 4m42s v1.12.2
如果重启或者kubeadm reset 将清理掉cni配置，需要重新配置
mkdir /etc/cni/net.d/ -p
cat >/etc/cni/net.d/10-mynet.conf <<-EOF
{
"cniVersion": "0.3.0",
"name": "mynet",
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.244.0.0/16",
"routes": [
{"dst": "0.0.0.0/0"}
]
}
}
EOF
cat >/etc/cni/net.d/99-loopback.conf <<-EOF
{
"cniVersion": "0.3.0",
"type": "loopback"
}
EOF
查看节点情况
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 5m43s v1.12.2
在node节点上安装
- 安装docker
yum makecache
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
sudo yum-config-manager --add-repo http://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
sudo yum makecache fast
yum install docker-ce-17.06.2.ce-1.el7.centos
systemctl start docker.service
systemctl enable docker.service
- 修改内核配置
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1
vm.swappiness = 0
net.ipv4.neigh.default.gc_stale_time = 120
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.arp_announce = 2
net.ipv4.conf.lo.arp_announce = 2
net.ipv4.conf.all.arp_announce = 2
net.ipv4.tcp_max_tw_buckets = 5000
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_synack_retries = 2
kernel.sysrq = 1
net.ipv4.tcp_tw_reuse = 1
net.ipv4.ip_local_port_range = 1024 65023
net.ipv4.tcp_max_syn_backlog = 10240
net.ipv4.tcp_max_tw_buckets = 400000
net.ipv4.tcp_max_orphans = 60000
net.ipv4.tcp_synack_retries = 3
net.core.somaxconn = 10000
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
vm.swappiness = 0
sysctl -p
- 修改docker的配置
# cat /etc/docker/daemon.json
{
"registry-mirrors": ["http://xxxxxxx.m.daocloud.io"],
"log-driver": "journald",
"log-opts": {
"tag": "{{.ImageName}}/{{.Name}}/{{.ID}}"
}
}
- 配置阿里云的kubernetes源
cat <<EOF > /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
EOF
yum makecache fast
- 安装kubeadm
yum install -y kubernetes-cni kubelet kubeadm kubectl --skip-broken
systemctl start kubelet.service
systemctl enable kubelet.service
修改host
hostnamectl set-hostname node1
vim /etc/host
10.2.29.149 node1
加入到集群中去
kubeadm join 10.2.29.148:6443 --token 5m131g.yz6x7217jjfz5w98 --discovery-token-ca-cert-hash sha256:2258ceec06ea8e5b855273cf4d7e45f4a83c42fbd7c3cce2331984418bf504c6
配置cni网络
mkdir /etc/cni/net.d/ -p
cat >/etc/cni/net.d/10-mynet.conf <<-EOF
{
"cniVersion": "0.3.0",
"name": "mynet",
"type": "bridge",
"bridge": "cni0",
"isGateway": true,
"ipMasq": true,
"ipam": {
"type": "host-local",
"subnet": "10.244.0.0/16",
"routes": [
{"dst": "0.0.0.0/0"}
]
}
}
EOF
cat >/etc/cni/net.d/99-loopback.conf <<-EOF
{
"cniVersion": "0.3.0",
"type": "loopback"
}
EOF
在master上查看节点情况：
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready master 22m v1.12.2
node1 Ready <none> 67s v1.12.2
- 在master附属flannel网络
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/v0.10.0/Documentation/kube-flannel.yml
- 查看运行的服务
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5f5b8b4fdb-s8672 0/1 CrashLoopBackOff 8 25m
coredns-5f5b8b4fdb-zpn49 0/1 CrashLoopBackOff 8 25m
etcd-master1 1/1 Running 0 24m
kube-apiserver-master1 1/1 Running 0 24m
kube-controller-manager-master1 1/1 Running 0 24m
kube-flannel-ds-jkh96 0/1 Init:0/1 0 2m19s
kube-flannel-ds-nwq7t 1/1 Running 0 2m19s
kube-proxy-4q6sv 1/1 Running 0 25m
kube-proxy-7wjvq 0/1 ContainerCreating 0 4m32s
kube-scheduler-master1 1/1 Running 0 24m
在node上拉取pause容器
# docker pull registry.aliyuncs.com/google_containers/pause:3.1
3.1: Pulling from google_containers/pause
cf9202429979: Pull complete
Digest: sha256:759c3f0f6493093a9043cc813092290af69029699ade0e3dbe024e968fcb7cca
Status: Downloaded newer image for registry.aliyuncs.com/google_containers/pause:3.1
- 转换tag
docker tag registry.aliyuncs.com/google_containers/pause:3.1 k8s.gcr.io/pause:3.1
在master上查看
$ kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-5f5b8b4fdb-s8672 1/1 Running 9 50m
coredns-5f5b8b4fdb-zpn49 1/1 Running 9 50m
etcd-master1 1/1 Running 0 49m
kube-apiserver-master1 1/1 Running 0 49m
kube-controller-manager-master1 1/1 Running 0 49m
kube-flannel-ds-jkh96 1/1 Running 0 26m
kube-flannel-ds-nwq7t 1/1 Running 0 26m
kube-proxy-4q6sv 1/1 Running 0 50m
kube-proxy-7wjvq 1/1 Running 0 29m
kube-scheduler-master1 1/1 Running 0 49m
