Kubernetes + Dashboard + Heapster (三) nginx-ingress负载均衡

kubectl 基本命令

检查节点状态
kubectl get nodes
检查namespace
kubectl get namespace
删除节点
etcdctl ls /registry/minions
或者
kubectl delete node  xxxx
更改 rc - pods 数量

kubectl scale rc rc_name --replicas=number

nginx-ingress

原理

一般来说，svc和pod拥有的IP只能在集群内部使用。集群外部请求需要通过负载均衡转发到svc在node上暴露的NodePort,然后再由kube-proxy 将其转发给相关的pod。
[图片上传失败...(image-162def-1516283321671)]

而Ingress就是为进入集群的请求提供路由规则的集合，如图所示。
[图片上传失败...(image-8fd5c7-1516283321671)]
Ingress可以给service提供集群外部访问的URL、负载均衡、SSL终止、HTTP路由等。为了配置这些Ingress规则，集群管理员需要部署一个Ingress controller，它监听Ingress和service的变化，并根据规则配置负载均衡并提供访问入口。

配置规则

每个Ingress都需要配置rules，目前Kubernetes仅支持http规则。上面的示例表示请求/testpath时转发到服务test的80端口。
根据Ingress Spec配置的不同，Ingress可以分为以下几种类型：

单服务Ingress

单服务Ingress即该Ingress仅指定一个没有任何规则的后端服务。

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test-ingress
spec:
  backend:
    serviceName: testsvc
    servicePort: 80

路由到多服务的Ingress

路由到多服务的Ingress即根据请求路径的不同转发到不同的后端服务上

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /foo
        backend:
          serviceName: s1
          servicePort: 80
      - path: /bar
        backend:
          serviceName: s2
          servicePort: 80

虚拟主机Ingress

虚拟主机Ingress即根据名字的不同转发到不同的后端服务上，而他们共用同一个的IP地址，如下所示

foo.bar.com --|                 |-> foo.bar.com s1:80
              | 178.91.123.132  |
bar.foo.com --|                 |-> bar.foo.com s2:80

下面是一个基于Host header路由请求的Ingress：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - backend:
          serviceName: s1
          servicePort: 80
  - host: bar.foo.com
    http:
      paths:
      - backend:
          serviceName: s2
          servicePort: 80

总结

对于我来说，最常用的还是虚拟主机模式。就像nginx中conf.d多个主机似的一回事。但是对于一个大型网站的话，或许使用路由到多服务模式会更好一些。

前面是对模式的整理，但还有一个组件。其中default-http-backend作为 Nginx Ingress Controller默认的后端，处理所有404请求。

安装配置

default-http-backend

首先配置 default-http-backend
当前pod作为Nginx Ingress Controller默认的后端，处理所有404请求。当前还没有做任何配置，此时请求的时候将由此pod响应。

镜像准备

如果有加速器

docker pull gcr.io/google_containers/defaultbackend:1.4

如果没有

docker pull registry.cn-beijing.aliyuncs.com/cloudexp/defaultbackend:latest

yml 文件

curl -o default-http-backend.yml https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml

或者

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: registry.cn-beijing.aliyuncs.com/cloudexp/defaultbackend:latest
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi

部署

kubectl create -f default-http-backend.yml
#返回结果
deployment "default-http-backend" created
service "default-http-backend" created

nginx-ingress-controller

镜像准备

docker pull registry.cn-hangzhou.aliyuncs.com/google-containers/nginx-ingress-controller:0.9.0

yml文件

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: nginx-ingress-controller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-controller
    spec:
      # hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration
      # however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host
      # that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used
      # like with kubeadm
      # hostNetwork: true
      terminationGracePeriodSeconds: 60
      containers:
      - image: registry.cn-hangzhou.aliyuncs.com/google-containers/nginx-ingress-controller:0.9.0
        name: nginx-ingress-controller
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        ports:
        - containerPort: 80
          hostPort: 80
        - containerPort: 443
          hostPort: 443
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        args:
        - /nginx-ingress-controller
        - --default-backend-service=kube-system/default-http-backend
        #- --publish-service=kube-system/nginx-ingress-lb
        - --apiserver-host=http://10.10.30.102:8080

注意如果启动报错找不到APIServer，就加上这一行。
如果github上的镜像无法pull，可以更换为阿里云。

部署

部署成功后创建以下两个pod：

[root@localhost kubernetes]# kubectl get pods  --namespace=kube-system -l k8s-app=nginx-ingress-controller -o wide
NAME                                        READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-ingress-controller-2867543418-f4t1t   1/1       Running   0          46s       172.30.14.4   centos-minion-1
[root@localhost kubernetes]# kubectl get pods  --namespace=kube-system -l k8s-app=nginx-ingress-controller -o wide
NAME                                        READY     STATUS    RESTARTS   AGE       IP            NODE
nginx-ingress-controller-2867543418-f4t1t   1/1       Running   0          49s       172.30.14.4   centos-minion-1

Ingress

创建服务

部署两个版本的nginx。

nginx1

apiVersion: v1
kind: Service
metadata:
  name: nginx1-8
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx1-8
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx1-8-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx1-8
    spec:
      containers:
      - name: nginx
        image: docker.io/nginx:latest
        ports:
        - containerPort: 80

nginx2

为了更好区分，还了个别的示例

apiVersion: v1
kind: Service
metadata:
  name: nginx2-8
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx2-8
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx2-8-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx2-8
    spec:
      containers:
      - name: nginx
        image: docker.io/kubeguide/guestbook-php-frontend
        ports:
        - containerPort: 80

创建完成后，检查

kubectl get pods -o wide

配置ingress

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: test
spec:
  rules:
  - host: test.xxx1.com
    http:
      paths:
      - backend:
          serviceName: nginx1-8
          servicePort: 80
  - host: test.xxx2.com
    http:
      paths:
      - backend:
          serviceName: nginx2-8
          servicePort: 80

配置hosts文件和host对应，访问即可

#主机 域名
10.10.30.102    test.xxx1.com
10.10.30.102    test.xxx2.com

测试

curl -I test.xxx1.com

curl -I test.xxx2.com

自定义上传文件大小

nginx ingress controller默认定义的上传大小为1M，因此需要更改nginx配置中client_max_body_size的大小，具体修改ingress文件，如下

cat nginx-ingress-cm.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: nginx-configuration
  namespace: kube-system
  labels:
    k8s-app: nginx-ingress-controller
data:
  proxy-body-size: "50m"

nginx-ingress-controller.yml 需要匹配官方文件
https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/without-rbac.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: kube-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ingress-nginx
  template:
    metadata:
      labels:
        app: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      nodeName: centos-minion-1
      #hostNetwork: true
      containers:
        - name: nginx-ingress-controller
          #image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.10.0
          image: registry.cn-hangzhou.aliyuncs.com/google-containers/nginx-ingress-controller:0.9.0
          args:
            - /nginx-ingress-controller
            - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --annotations-prefix=nginx.ingress.kubernetes.io
            - --apiserver-host=http://10.10.30.102:8080
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
          - name: logs
            mountPath: /var/log/nginx/
          ports:
          - name: http
            containerPort: 80
          - name: https
            containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
      volumes:
        - name: logs
          hostPath:
            path: /data/logs/nginx

更多参考：
Kubernetes + Dashboard + Heapster (一) 安装配置
 Kubernetes + Dashboard + Heapster (二) 监控部署
 Kubernetes + Dashboard + Heapster (三) ingress负载均衡
 Kubernetes + Dashboard + Heapster (四) 慢慢填坑