UK8S Ingress 配置

kubectl apply -f http://uk8s.cn-bj.ufileos.com/yaml/ingress/nginx/mandatory.yaml

在上述文章第二步的时候，参照通过外网ULB访问Service，在 yaml 中添加关于 ULB 配置的声明,
具体例如如下：

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  annotations:
    service.beta.kubernetes.io/ucloud-load-balancer-id: "ulb-xxx"
    service.beta.kubernetes.io/ucloud-load-balancer-vserver-protocol: "TCP"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

上面是TCP L4层的代理，没有搞L7，因为存在不同namespace下，需要用到不同 https 证书, 的场景，我们后续会使用 cert-manager 来解决 https 证书问题。

Nginx 配置

由于特定业务的特殊需求，需要对 Nginx 进行特殊配置，例如增大客户端允许上传数据的大小。可参看文档 Annotation-NGINX Ingress Controller

Multi Ingress

以上是外网 Ingress 配置，我们还需要个内网 ingress 配置，解决思路有两种，1种是切换另一种 ingress，还有一种是继续使用 nginx ingress。我们选择第二种方案，方便维护。

具体来讲，就是对 http://uk8s.cn-bj.ufileos.com/yaml/ingress/nginx/mandatory.yaml 进行定制。可参照 Google Multi Nginx-ingress 文档，具体步骤如下：

1. 修改外网 Nignx Controller 配置

给外网 Nignx Controller 添加 Label, 方便和内网区分开来，如: ingress-type: outer
给外网 Ingress 添加 Selector，内容为第一步多加的 Label

2. 修改 Role 配置

nginx-ingress-role 的 resourceNames 添加一行 ingress-controller-leader-internal-nginx。
使得先前创建的 nginx-ingress-serviceaccount 可以操纵将要创建的 internal-nginx。

3. 部署 internal-nginx-controller

kind: ConfigMap
apiVersion: v1
metadata:
  name: internal-nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: internal-tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: internal-udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: internal-nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
        ingress-type: inner
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      serviceAccountName: nginx-ingress-serviceaccount
      containers:
        - name: internal-nginx-ingress-controller
          image: uhub.service.ucloud.cn/library/nginx-ingress-controller:0.23.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/internal-nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/internal-tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/internal-udp-services
            - --publish-service=$(POD_NAMESPACE)/internal-ingress-nginx
            - --ingress-class=internal-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: https
              containerPort: 443
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
  name: internal-ingress-nginx
  namespace: ingress-nginx
  annotations:
    service.beta.kubernetes.io/ucloud-load-balancer-id: "ulb-xxx"
    service.beta.kubernetes.io/ucloud-load-balancer-vserver-protocol: "TCP"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  type: LoadBalancer
  ports:
    - name: http
      port: 80
      targetPort: 80
      protocol: TCP
    - name: https
      port: 443
      targetPort: 443
      protocol: TCP
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
    ingress-type: inner

这个 yaml 对比老的来说，修改点有：

nginx-ingress-controller 的 args、label 和名称，尤其是 --ingress-class=internal-nginx, Ingress 要用到，前面的 nginx-ingress-role 的修改也和它有关联。
service 对应的内网ULB ID，以及 selector 里的 ingress-type: inner。
ConfigMap 的名称，如果你想内外网都用统一的配置，也可以不改。

Ingress 区分使用

以上配置默认会使用外网Ingress Controller，
内网 Ingress 的 Annotation 里面要添加 kubernetes.io/ingress.class: internal-nginx, 才能告知内网Ingress Controller 如何 forward 流量。

获取客户端IP

参考 UCloud 文档： https://docs.ucloud.cn/compute/uk8s/service/getresourceip

UK8S Ingress 配置