一. 持续集成/部署/交付概述

持续集成(Continuous Integration，CI): 代码合并、构建、部署、测试都在一起，不断地执行这个过程，并对结果反馈。
持续部署(Continuous Deployment，CD): 部署到测试环境、预生产环境、生产环境。
持续交付(Continuous Delivery，CD): 将最终产品发布到生产环境，给用户使用。

image.png

image.png

image.png

二. 准备工作

1. 对项目的理解

• 单体架构、微服务
• 怎么部署
• 启动是否有依赖

2. 部署到k8s平台流程

(1). 制作镜像
(2). 容器放到Pod
(3). 控制器管理Pod
(4). 暴露应用
(5). 对外发布应用
(6). 日志管理/监控

3. 不同环境区分配置文件

• configmap
• entrypoint.sh
• 统一配置中心，例如 Apollo，Disconf

4. Harbor镜像仓库

部署参考: //www.greatytc.com/p/7ca6c59f9882

5. Git代码版本仓库

(1). 安装Git

# yum install git

(2). 创建Git用户密码

# useradd git
# passwd git

(3). 创建仓库

# su - git
$ mkdir java-demo.git
$ cd java-demo.git
$ git --bare init

(4). Git服务器SSH免交互认证

配置客户端(10.40.6.213)与Git服务器SSH免交互认证

# mkdir git && cd git
# git clone git@10.40.6.165:/home/git/java-demo.git
# touch 123
# git add .
# git commit -m 'test'
# git push origin master

(5). 密钥免交互测试

# ssh-keygen  
# ssh-copy-id git@10.40.6.165
# ssh git@10.40.6.165
# cd .. && rm -rf java-demo 
# git clone git@10.40.6.165:/home/git/java-demo.git

三. Kubernetes中部署Jenkins

参考:https://github.com/jenkinsci/kubernetes-plugin/tree/fc40c869edfd9e3904a9a56b0f80c5a25e988fa1/src/main/kubernetes

image.png

NFS服务部署：//www.greatytc.com/p/26003390626e
创建NFS 动态供给参考：//www.greatytc.com/p/092eb3aacefc

1. 部署有状态的jenkins Pod

# cat statefulset-jenkins.yml
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: jenkins
  labels:
    name: jenkins
spec:
  serviceName: jenkins
  replicas: 1
  updateStrategy:
    type: RollingUpdate
  template:
    metadata:
      name: jenkins
      labels:
        name: jenkins
    spec:
      terminationGracePeriodSeconds: 10
      serviceAccountName: jenkins
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts-alpine
          imagePullPolicy: Always
          ports:
            - containerPort: 8080
            - containerPort: 50000
          resources:
            limits:
              cpu: 1
              memory: 1Gi
            requests:
              cpu: 0.5
              memory: 500Mi
          env:
            - name: LIMITS_MEMORY
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
                  divisor: 1Mi
            - name: JAVA_OPTS
              value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=GMT+08
          volumeMounts:
            - name: jenkins-home
              mountPath: /var/jenkins_home
          livenessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
          readinessProbe:
            httpGet:
              path: /login
              port: 8080
            initialDelaySeconds: 60
            timeoutSeconds: 5
            failureThreshold: 12
      securityContext:
        fsGroup: 1000
  volumeClaimTemplates:
  - metadata:
      name: jenkins-home
    spec:
      storageClassName: "managed-nfs-storage"
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 1Gi

  securityContext:
    fsGroup: 1000  这个配置可以去掉，但得先给后端存储切换组，如果用这个配置，后面jenkins 使用久了，jenkins 重启后会需要很久参能启动成功，这样就会影响健康检测失败，pod一直启动失败

2. 创建jenkins service

# cat service-jenkins.yml
apiVersion: v1
kind: Service
metadata:
  name: jenkins
spec:
  selector:
    name: jenkins
  type: NodePort
  ports:
    -
      name: http
      port: 80
      targetPort: 8080
      protocol: TCP
      nodePort: 30006
    -
      name: agent
      port: 50000
      protocol: TCP

3. 创建jenkins ingress

# cat ingress-jenkins.yml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    kubernetes.io/tls-acme: "true"
    # 如果上传插件超出默认会报"413 Request Entity Too Large", 增加 client_max_body_size
    nginx.ingress.kubernetes.io/proxy-body-size: 50m
    nginx.ingress.kubernetes.io/proxy-request-buffering: "off"
    # nginx-ingress controller版本小于 0.9.0.beta-18 的配置
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/proxy-body-size: 50m
    ingress.kubernetes.io/proxy-request-buffering: "off"
spec:
  rules:
  - host: jenkins.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: jenkins
          servicePort: 80

4. jenkins认证授权

# cat service-account.yml
---
# 创建名为jenkins的ServiceAccount
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
# 创建名为jenkins的Role，授予允许管理API组的资源Pod
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
# 将名为jenkins的Role绑定到名为jenkins的ServiceAccount
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

5. 获取jenkins密码并登录

# kubectl create -f .
登录容器获取jenkins登录密码：
# kubectl exec -it jenkins-0 bash
bash-4.4$ cat /var/jenkins_home/secrets/initialAdminPassword
0f8c730f4b7a426098283cf94aa57231

jenkins地址：http://10.40.6.213:30006
jenkins 安装插件：pipeline、git、kubernetes

四. Jenkins在K8S中动态创建代理

1. 传统架构与K8S 架构的区别

(1). 传统Master/Slave架构，Master收到Job后，将请求转发到Slave节点处理。Slave节点数固定，Slave节点未能自动申缩容。

Jenkins Master/Slave架构

(2). K8S中Jenkins Master/Slave架构，Master收到Job后，会自动创建Slave节点处理此Job，根据客户端的Job自动申缩容。

K8S中Jenkins Master/Slave架构

2. jenkins插件连接K8S配置

需要先安装pipeline插件, 建议不在UI上配置pod创建模版，免得以后每来一个项目都要创建，管理不方便，建议使用pipeline统一 配置。
Kubernetes插件介绍:https://github.com/jenkinsci/kubernetes-plugin

image.png

五. 构建Jenkins Slave镜像

参考:https://github.com/jenkinsci/docker-jnlp-slave

image.png

1. 构建Jenkins Slave镜像环境准备

构建Jenkins Slave镜像环境准备：
代码拉取：git，安装git命令
单元测试：忽略，这不是我们擅长的,如果公司有可以写进来
代码编译：maven，安装maven包
构建镜像：Dockerfile文件、docker命令(通过挂载宿主机docker)
推送镜像：docker命令(通过挂载宿主机docker)
镜像启动后支持slave: 下载官方slave.jar包（获取：http://10.40.6.213:30006/jnlpJars/slave.jar）
启动 slave.ja包：jenkins-slave启动脚步(通过参考文档URL)
maven配置文件：settings.xml （这里配置阿里云的仓库源）

获取相关文件：
Dockerfile
jenkins-slave 启动脚步
settings.xml
slave.jar

创建目录并进入：
mkdir jenkins-slave && cd jenkins-slave

2. Dockerfile配置文件

# cat Dockerfile
FROM centos:7
LABEL maintainer liuzhousheng

RUN yum install -y java-1.8.0-openjdk maven curl git libtool-ltdl-devel && \
    yum clean all && \
    rm -rf /var/cache/yum/* && \
    mkdir -p /usr/share/jenkins

COPY slave.jar /usr/share/jenkins/slave.jar
COPY jenkins-slave /usr/bin/jenkins-slave
COPY settings.xml /etc/maven/settings.xml
RUN chmod +x /usr/bin/jenkins-slave

ENTRYPOINT ["jenkins-slave"]

3. jenkins-slave启动脚步

# cat jenkins-slave
#!/usr/bin/env sh

if [ $# -eq 1 ]; then

    # if `docker run` only has one arguments, we assume user is running alternate command like `bash` to inspect the image
    exec "$@"

else

    # if -tunnel is not provided try env vars
    case "$@" in
        *"-tunnel "*) ;;
        *)
        if [ ! -z "$JENKINS_TUNNEL" ]; then
            TUNNEL="-tunnel $JENKINS_TUNNEL"
        fi ;;
    esac

    # if -workDir is not provided try env vars
    if [ ! -z "$JENKINS_AGENT_WORKDIR" ]; then
        case "$@" in
            *"-workDir"*) echo "Warning: Work directory is defined twice in command-line arguments and the environment variable" ;;
            *)
            WORKDIR="-workDir $JENKINS_AGENT_WORKDIR" ;;
        esac
    fi

    if [ -n "$JENKINS_URL" ]; then
        URL="-url $JENKINS_URL"
    fi

    if [ -n "$JENKINS_NAME" ]; then
        JENKINS_AGENT_NAME="$JENKINS_NAME"
    fi

    if [ -z "$JNLP_PROTOCOL_OPTS" ]; then
        echo "Warning: JnlpProtocol3 is disabled by default, use JNLP_PROTOCOL_OPTS to alter the behavior"
        JNLP_PROTOCOL_OPTS="-Dorg.jenkinsci.remoting.engine.JnlpProtocol3.disabled=true"
    fi

    # If both required options are defined, do not pass the parameters
    OPT_JENKINS_SECRET=""
    if [ -n "$JENKINS_SECRET" ]; then
        case "$@" in
            *"${JENKINS_SECRET}"*) echo "Warning: SECRET is defined twice in command-line arguments and the environment variable" ;;
            *)
            OPT_JENKINS_SECRET="${JENKINS_SECRET}" ;;
        esac
    fi

    OPT_JENKINS_AGENT_NAME=""
    if [ -n "$JENKINS_AGENT_NAME" ]; then
        case "$@" in
            *"${JENKINS_AGENT_NAME}"*) echo "Warning: AGENT_NAME is defined twice in command-line arguments and the environment variable" ;;
            *)
            OPT_JENKINS_AGENT_NAME="${JENKINS_AGENT_NAME}" ;;
        esac
    fi

    #TODO: Handle the case when the command-line and Environment variable contain different values.
    #It is fine it blows up for now since it should lead to an error anyway.

    exec java $JAVA_OPTS $JNLP_PROTOCOL_OPTS -cp /usr/share/jenkins/slave.jar hudson.remoting.jnlp.Main -headless $TUNNEL $URL $WORKDIR $OPT_JENKINS_SECRET $OPT_JENKINS_AGENT_NAME "$@"
fi

4. 获取slave.jar包

wget http://10.40.6.213:30006/jnlpJars/slave.jar

5. maven源配置文件settings.xml

maven源配置文件settings.xml，这里配置阿里云的源。

# cat settings.xml
<?xml version="1.0" encoding="UTF-8"?>

<settings xmlns="http://maven.apache.org/SETTINGS/1.0.0"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.0.0 http://maven.apache.org/xsd/settings-1.0.0.xsd">
  <pluginGroups>
  </pluginGroups>

  <proxies>
  </proxies>

  <servers>
  </servers>

  <mirrors>
    <mirror>
      <id>central</id>
      <mirrorOf>central</mirrorOf>
      <name>aliyun maven</name>
      <url>https://maven.aliyun.com/repository/public</url>
    </mirror>
  </mirrors>

  <profiles>
  </profiles>

</settings>

6. 构建镜像, 并推送至私有镜像仓库

# docker build -t 10.40.6.165/library/jenkins-slave-jdk:1.8 .
     ....
Successfully built 3aaa12e391fc
Successfully tagged 10.40.6.165/library/jenkins-slave-jdk:1.8

# docker login 10.40.6.165   ###先登录镜像仓库再推送
Username: admin
Password:

# docker push 10.40.6.165/library/jenkins-slave-jdk:1.8

注意： 私有镜像仓库用的是http访问，kubernetes是不信任的，得每个node节点配置信任。
参考：//www.greatytc.com/p/7ca6c59f9882

六. Jenkins Pipeline构建流水线发布

1. Jenkins Pipeline 核心概念

• Jenkins Pipeline是一套插件，支持在Jenkins中实现持续集成、交付管道;
• Pipeline通过特定语法从简单到复杂的传输管道进行建模;
① 声明式:遵循与Groovy相同语法。pipeline { }
② 脚本式:支持Groovy大部分功能，也是非常表达和灵活的工具。node { }
• Jenkins Pipeline的定义被写入一个文本文件，称为Jenkinsfile。

Jenkins Pipeline 核心概念：
Node： 节点，一个 Node 就是一个 Jenkins 节点，Master 或者 Agent，是执行 Step 的具体运行环境，比如我们之前动态运行的 Jenkins Slave 就是一个 Node 节点
Stage： 阶段，一个 Pipeline 可以划分为若干个 Stage，每个 Stage 代表一组操作，比如：Build、Test、Deploy等，Stage 是一个逻辑分组的概念，可以跨多个 Node
Step： 步骤，Step 是最基本的操作单元，可以是打印一句话，也可以是构建一个 Docker 镜像，由各类 Jenkins 插件提供，比如命令：sh ‘make’，就相当于我们平时 shell 终端中执行 make 命令一样。
参考:https://jenkins.io/doc/book/pipeline/syntax/

image.png

image.png

2. 拉取代码配置

(1). Pipeline语法使用

image.png

(2). 生成Pipeline语句

image.png

(3). 拉取Git代码秘钥配置

image.png

(4). Pileline语句

image.png

然后构建拉取代码测试

七. 编写Pipeline脚本完成CI阶段

参考文档：https://github.com/jenkinsci/kubernetes-plugin
pipeline script注意：
podTemplate 中label和node括号里的字符串要一致
podTemplate 中cloud为之前配置jenkins Cloud 名称
containerTemplate中name最好保持为"jnlp"
pipeline script变量docker_registry_auth、git_auth、k8s_auth通过保存在jenkins凭据中相应的凭据ID。

Pipeline Script如下：

// 公共
def registry = "10.40.6.165"
// 项目
def project = "project"
def app_name = "java-demo"
def image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}"
def git_address = "git@10.40.6.165:/home/git/java-demo.git"
// 认证
def secret_name = "registry-pull-secret"
def docker_registry_auth = "eba0f763-747e-47e3-a7ed-20eaf5cbab31"  //Harbor login auth
def git_auth = "fec843cc-3cc4-4c97-969f-df0b0f1bcc50" //git login auth
def k8s_auth = "80e66a86-d189-4555-b1ef-054285031b7a"  // k8s auth,CI阶段不用到, CD 部署时才用到

podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
    containerTemplate(
        name: 'jnlp',
        image: "${registry}/library/jenkins-slave-jdk:1.8"
    ),
  ],
  volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
    hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker'),
    hostPathVolume(mountPath: '/etc/localtime', hostPath: '/etc/localtime')
  ],
)
{
  node("jenkins-slave"){
      // 第一步
      stage('拉取代码'){
         checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
      }
      // 第二步
      stage('代码编译'){
          sh "mvn clean package -Dmaven.test.skip=true"
      }
      // 第三步
      stage('构建镜像'){
          //Harbor镜像仓库登录验证，
          withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
            sh """
              echo '
                FROM 10.40.6.165/project/java-demo:v1
                RUN rm -rf /usr/local/tomcat/webapps/*
                ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
              ' > Dockerfile
              ls
              ls target
              docker build -t ${image_name} .
              docker login -u ${username} -p '${password}' ${registry}
              docker push ${image_name}
            """
            }
      }
  }
}

pipeline script代码可选择保存在git代码首目录的代码中
测试刚构建的镜像：
docker run -d -p 666:8080 10.40.6.165/project/java-demo:17
访问地址：http://10.40.6.213:666

八. Jenkins在K8s中持续部署

Kubernetes Continuous Deploy插件: 用于将资源配置部署到Kubernetes
插件介绍: https://plugins.jenkins.io/kubernetes-cd
支持以下资源类型:
• Deployment
• Replica Set
• Daemon Set
• Pod
• Job
• Service
• Ingress
• Secret

Jenkins将镜像部署到kubernetes需要配置:
①. 要连接kubernetes 需要用到kubeconfig配置文件，要在pipeline 中读取此配置文件，得配置到jenkins到凭据中，使用jenkins Kubernetes Continuous Deploy插件读取此凭据的ID(kubeconfigID)即可；
②. 指定部署哪个资源文件，deplayment yaml文件；
③. 资源是否使用到secret验证信息，得指定。

1. 生成kubeconfig

(1). 主体角色绑定

liuzhousheng 用户主体绑定deployment-secret角色：
mkdir deployment-client && cd deployment-client

创建角色权限
# cat rbac-role.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: deployment-secret
rules:
- apiGroups: [""]
  resources: ["pods","services"]
  verbs: ["get", "list", "watch", "create", "delete","update","patch"]
- apiGroups: ["extensions", "apps"]
  resources: ["deployments","ingresses"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

user liuzhousheng主体绑定deployment-secret角色
# cat rbac-rolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: deployment-rolebinding
subjects:
- kind: User
  name: liuzhousheng # Name is case sensitive
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role #this must be Role or ClusterRole
  name: deployment-secret # this must match the name of the Role or ClusterRole you wish to bind to
  apiGroup: rbac.authorization.k8s.io

# kubectl create -f rbac-role.yaml
# kubectl create -f rbac-rolebinding.yaml

(2). 生成kubeconfig配置文件

根据集群根证书颁发客户端证书，然后生成连接集群配置文件liuzhousheng-kubeconfig配置文件。
mkdir liuzhousheng && cd liuzhousheng

# cat rbac-user.sh
#!/bin/bash
cat > liuzhousheng-csr.json <<EOF
{
  "CN": "liuzhousheng",
  "hosts": [],
  "key": {
    "algo": "rsa",
    "size": 2048
  },
  "names": [
    {
      "C": "CN",
      "L": "BeiJing",
      "ST": "BeiJing"
    }
  ]
}
EOF
###签发一个客户端证书,注意要指定根证书
cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes liuzhousheng-csr.json | cfssljson -bare liuzhousheng

#-----------------------------------
##生成配置文件，使用配置文件连接集群

#配置集群，这里的ca.pem为集群ca证书
kubectl config set-cluster kubernetes \
  --certificate-authority=ca.pem \
  --embed-certs=true \
  --server=https://10.40.6.175:6443 \
  --kubeconfig=liuzhousheng-kubeconfig

#客户端证书配置
kubectl config set-credentials liuzhousheng \
  --client-key=liuzhousheng-key.pem \
  --client-certificate=liuzhousheng.pem \
  --embed-certs=true \
  --kubeconfig=liuzhousheng-kubeconfig

#配置上下文
kubectl config set-context default \
  --cluster=kubernetes \
  --user=liuzhousheng \
  --kubeconfig=liuzhousheng-kubeconfig

kubectl config use-context default --kubeconfig=liuzhousheng-kubeconfig

# cp  /usr/local/src/k8s/kube-apiserver/{ca.pem,ca-key.pem,ca-config.json} ./
# bash rbac-user.sh
# cat liuzhousheng-kubeconfig

2. 配置kuberconfig凭据

将liuzhousheng-kubeconfig配置文件保存到jenkins凭据中

image.png

image.png

3. 编写deploy.yaml文件

将deploy.yaml文件提交到代码仓库中：

# cat deploy.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: java-demo
  template:
    metadata:
      labels:
        app: java-demo
    spec:
      imagePullSecrets:
      - name: $SECRET_NAME
      containers:
      - name: tomcat
        image: $IMAGE_NAME
        ports:
        - containerPort: 8080
          name: web
        livenessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12
        readinessProbe:
          httpGet:
            path: /
            port: 8080
          initialDelaySeconds: 60
          timeoutSeconds: 5
          failureThreshold: 12


---
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  type: NodePort
  selector:
    app: java-demo
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: web
spec:
  rules:
  - host: java.example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: web
          servicePort: 80

# git add .
# git commit -m 'deploy.yaml'
# git push origin master

4. 编写pipeline 脚步

pipeline脚本：

// 公共
def registry = "10.40.6.165"
// 项目
def project = "project"
def app_name = "java-demo"
def image_name = "${registry}/${project}/${app_name}:${BUILD_NUMBER}"
def git_address = "git@10.40.6.165:/home/git/java-demo.git"
// 认证
def secret_name = "registry-pull-secret"
def docker_registry_auth = "eba0f763-747e-47e3-a7ed-20eaf5cbab31"  //Harbor login auth
def git_auth = "fec843cc-3cc4-4c97-969f-df0b0f1bcc50" //git login auth
def k8s_auth = "5a9f013d-b01e-43b6-b7d7-aef2b54c64c7"   //k8s login auth

podTemplate(label: 'jenkins-slave', cloud: 'kubernetes', containers: [
    containerTemplate(
        name: 'jnlp',
        image: "${registry}/library/jenkins-slave-jdk:1.8"
    ),
  ],
  volumes: [
    hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
    hostPathVolume(mountPath: '/usr/bin/docker', hostPath: '/usr/bin/docker'),
    hostPathVolume(mountPath: '/etc/localtime', hostPath: '/etc/localtime')
  ],
)
{
  node("jenkins-slave"){
      // 第一步
      stage('拉取代码'){
         checkout([$class: 'GitSCM', branches: [[name: '${Branch}']], userRemoteConfigs: [[credentialsId: "${git_auth}", url: "${git_address}"]]])
      }
      // 第二步
      stage('代码编译'){
          sh "mvn clean package -Dmaven.test.skip=true"
      }
      // 第三步
      stage('构建镜像'){
          //Harbor镜像仓库登录验证，
          withCredentials([usernamePassword(credentialsId: "${docker_registry_auth}", passwordVariable: 'password', usernameVariable: 'username')]) {
            sh """
              echo '
                FROM 10.40.6.165/project/java-demo:v1
                RUN rm -rf /usr/local/tomcat/webapps/*
                ADD target/*.war /usr/local/tomcat/webapps/ROOT.war
              ' > Dockerfile
              ls
              ls target
              docker build -t ${image_name} .
              docker login -u ${username} -p '${password}' ${registry}
              docker push ${image_name}
            """
            }
      }
    // 第四步
      stage('部署到K8S平台'){
          sh """
          sed -i 's#\$IMAGE_NAME#${image_name}#' deploy.yaml
          sed -i 's#\$SECRET_NAME#${secret_name}#' deploy.yaml
          """
          kubernetesDeploy configs: 'deploy.yaml', kubeconfigId: "${k8s_auth}"
      }
  }
}

脚本注意变量：
k8s_auth：jenkins保存kubeconfig配置凭据的ID

5. 创建harbor镜像仓库secret验证

pipeline脚步中的secret_name变量为：访问harbor镜像仓库私有库secret验证，在kubernetes master 创建，如下：

# kubectl create secret --help
# kubectl create secret docker-registry --help
# kubectl create secret docker-registry registry-pull-secret --docker-username=admin --docker-password=Harbor12345 --docker-email=888888@qq.com --docker-server=10.40.6.165

可选择通过yaml文件定义创建：

首先在其中一个node上登录私有仓库（docker 访问私库的配置，这里不做描述）
# docker login my.registry

登录成功后会在/root/.docker目录下生产config.json文件，然后执行如下命令：
# cat /root/.docker/config.json | base64 -w 0

该命令会将你的认证信息通过base64编码，生成一个编码之后的字符串。
在kubernetes中的master节点中创建registry-pull-secret-vpc.yaml：
# cat registry-pull-secret-vpc.yaml
apiVersion: v1
kind: Secret
metadata:
  name: registry-pull-secret-vpc
  namespace: test
data:
  .dockerconfigjson: xxxxxxxxxx
type: kubernetes.io/dockerconfigjson

# kubectl apply -f registry-pull-secret-vpc.yaml

6. 配置代码拉取分支变量

Branch：代码拉取分支选择

image.png

7.构建测试

jenkins构建，然后绑定hosts访问 10.40.6.213 java.example.com

九. Pipeline脚本与源代码版本管理

1. Pipeline 脚本版本管理

将Pipeline 脚本内容保存到同源代码git代码仓库中的Jenkinsfile文件中，同代码版本控制

# ll
drwxr-xr-x 2 root root    34 6月  29 18:12 db
-rw-r--r-- 1 root root  1155 7月   1 22:09 deploy.yaml
-rw-r--r-- 1 root root  2255 7月   1 23:16 Jenkinsfile
-rw-r--r-- 1 root root 11357 6月  29 18:12 LICENSE
-rw-r--r-- 1 root root  1930 6月  29 18:12 pom.xml
-rw-r--r-- 1 root root    88 6月  29 18:12 README.md
drwxr-xr-x 3 root root    18 6月  29 18:12 src

# git add .
# git commit -m 'add Jenkinsfile'
# git push origin master

2. 修改jenkins Pipeline定义

修改jenkins UI配置

image.png

回滚：
# kubectl get deployment
# kubectl rollout history deployment web

回滚到上一个版本
# kubectl rollout history deployment web

回滚到指定版本
# kubectl rollout undo deployment web --to-revision=5

扩缩容：
# kubectl scale deployment web --replicas=5

十. K8s滚动发布实现原理

1. 滚动发布原理

每次只升级一个或多个服务，升级完成后加入
生产环境，不断执行这个过程，直到集群中的
全部旧版升级新版本。
特点:
• 用户无感知，平滑过渡 缺点:
• 部署周期长
• 发布策略较复杂
• 不易回滚

image.png

2. K8s滚动发布架构

1个Deployment
2个ReplicaSet

image.png

十一. 在K8s中实现灰度发布方案

1. 灰度发布原理

只升级部分服务，即让一部分用户继续用老版本，一 部分用户开始用新版本，如果用户对新版本没有什么 意见，那么逐步扩大范围，把所有用户都迁移到新版 本上面来。
特点:
• 保证整体系统稳定性
• 用户无感知，平滑过渡 缺点:
• 自动化要求高

image.png

2. K8s灰度发布架构

结合LB，2个Deployment

image.png

十二. 小结

❖ 使用Jenkins三个插件
• Kubernetes
• Pipeline
• Kubernetes Continuous Deploy

❖ CI/CD环境特点
• Slave弹性伸缩
• 基于镜像隔离构建环境 • 流水线发布，易维护

❖ Jenkins参数化构建可帮助你完成更复杂环境CI/CD