其他分享
首页 > 其他分享> > kubesphere devops 安装以及简单使用

kubesphere devops 安装以及简单使用

作者:互联网

    简单研究下kubesphere 中devops 的使用。

1. 安装

实际上就是修改安装过程中的 cluster-configuration.yaml 文件的devops.enable 设为true 即可, kebesphere 会自己下载jenkins,我们通过kubesphere 操作流水线最终也会到jenkins 中操作对应的流水线。

参考: https://kubesphere.io/zh/docs/pluggable-components/devops/

安装过程中出现的问题: nfs服务器磁盘空间、k8snode01 磁盘空间、k8snode01 内存问题等等。建议在虚拟机初始化就设置的大一点。

1. 安装成功后查看相关pods、service 如下:

[root@k8smaster01 kubesphere]# kubectl get pods -n kubesphere-devops-system
NAME                                READY   STATUS      RESTARTS   AGE
devops-27439710-f4q5f               0/1     Completed   0          8m12s
devops-apiserver-7c6774fff5-nmj84   1/1     Running     0          21m
devops-controller-98975d478-4xhxn   1/1     Running     0          21m
devops-jenkins-64464f495f-4z8br     1/1     Running     1          21m
s2ioperator-0                       1/1     Running     0          21m
[root@k8smaster01 kubesphere]# kubectl get svc -n kubesphere-devops-system
NAME                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE
devops-apiserver              ClusterIP   10.1.248.197   <none>        9090/TCP       21m
devops-jenkins                NodePort    10.1.45.197    <none>        80:30180/TCP   21m
devops-jenkins-agent          ClusterIP   10.1.113.105   <none>        50000/TCP      21m
s2ioperator-metrics-service   ClusterIP   10.1.161.145   <none>        8080/TCP       21m
s2ioperator-trigger-service   ClusterIP   10.1.119.55    <none>        8081/TCP       21m
webhook-server-service        ClusterIP   10.1.46.76     <none>        443/TCP        21m

2. 然后登录kubesphere 控制台看到如下:

2. 登录jenkins

上面可以看到service, 30180 端口可以访问jenkins,在找密码的时候发现其没有密码文件。 最后通过官网发现: 是用kubesphere 的用户体系进行登录。

参考:  https://v2-1.docs.kubesphere.io/docs/zh-CN/devops/jenkins-setting/

接下来可以简单的看一下jenkins 控制台界面安装的一些插件等信息。

2. 使用

接下来就是使用kubesphere 的devops 发布一个简单的springboot 项目。 这里使用github作为代码仓库,阿里镜像仓库作为镜像仓库。

官网对其运行过程解释如下: 

  首先,Jenkins Master 创建一个 Pod 来运行流水线。Kubernetes 创建 Pod 作为 Jenkins Master 的 Agent,该 Pod 会在流水线完成之后销毁。主要流程包括克隆代码、构建和推送镜像以及部署工作负载。

官方提供的两个java 项目:

https://github.com/kubesphere/devops-java-sample

https://github.com/kubesphere/devops-maven-sample 本次测试使用这个项目进行测试

搭建过程参考:

https://kubesphere.io/docs/devops-user-guide/how-to-use/create-a-pipeline-using-jenkinsfile/

1.  fork项目到自己的github 仓库

  如果网络波动太大的话可以使用国内的gitee,作为代码仓库进行测试。我下面使用gitee。

2. 到kubesphere 进行操作

  这里用户都用admin 进行操作,如果想测试kubesphere 的用户体系可以自己创建多个用户进行操作。

1. 到访问控制-》企业空间创建一个企业空间 dev, 用于独立的测试。这个可以理解为大的一个资源隔离的空间。

一个空间可以有多个普通项目和devops 项目。

普通项目可以用来管理独立的kubernetes 资源,包括pod、svc 等,实际kubesphere 后台的操作就是新建了一个对应的namespace用于隔离相应资源,namespace 的名称就是项目名称;

devops 项目可以包含多个jenkins流水线, 用于devops 操作,kubesphere 后台也创建了一个对应的namespace,namespace名称为devops 项目名称加5位随机字母。

2. dev 企业空间创建一个项目,名称为 demo。 查看namespace如下

[root@k8smaster01 ~]# kubectl get ns|grep demo
demo59wjg                         Active   26s

3. 在demo 项目下面建立自己的凭证:

1》gitee 的凭证

2》镜像仓库的凭证

3》kubeconfig 用于kubernetes 使用

4. 本地复制项目Jenkinsfile-online,命名为Jenkinsfile。 原来文件用于备份,新文件用于jenkins 流水线脚本。修改Jenkinsfile 里面内容的凭证信息,最终如下:

  1 pipeline {
  2   agent {
  3     node {
  4       label 'maven'
  5     }
  6   }
  7 
  8     parameters {
  9         string(name:'TAG_NAME',defaultValue: '',description:'')
 10     }
 11 
 12     environment {
 13         DOCKER_CREDENTIAL_ID = 'ali-registery'
 14         GITHUB_CREDENTIAL_ID = 'gitee-secret'
 15         KUBECONFIG_CREDENTIAL_ID = 'demo-kubeconfig'
 16         REGISTRY = 'registry.cn-hangzhou.aliyuncs.com'
 17         DOCKERHUB_NAMESPACE = 'qlq_repository'
 18         GITHUB_ACCOUNT = 'qiao-zhi'
 19         APP_NAME = 'devops-maven-sample'
 20     }
 21 
 22     stages {
 23         stage ('checkout scm') {
 24             steps {
 25                 checkout(scm)
 26             }
 27         }
 28 
 29         stage ('unit test') {
 30             steps {
 31                 container ('maven') {
 32                     sh 'mvn clean test'
 33                 }
 34             }
 35         }
 36  
 37         stage ('build & push') {
 38             steps {
 39                 container ('maven') {
 40                     sh 'mvn clean package -DskipTests'
 41                     sh 'docker build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .'
 42                     withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) {
 43                         sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
 44                         sh 'docker push  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER'
 45                     }
 46                 }
 47             }
 48         }
 49 
 50         stage('push latest'){
 51            when{
 52              branch 'master'
 53            }
 54            steps{
 55                 container ('maven') {
 56                   sh 'docker tag  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
 57                   sh 'docker push  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
 58                 }
 59            }
 60         }
 61 
 62         stage('deploy to dev') {
 63           when{
 64             branch 'master'
 65           }
 66           steps {
 67             input(id: 'deploy-to-dev', message: 'deploy to dev?')
 68             container ('maven') {
 69                 withCredentials([
 70                     kubeconfigFile(
 71                     credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
 72                     variable: 'KUBECONFIG')
 73                     ]) {
 74                     sh 'envsubst < deploy/dev-all-in-one/devops-sample.yaml | kubectl apply -f -'
 75                 }
 76             }
 77           }
 78         }
 79         stage('push with tag'){
 80           when{
 81             expression{
 82               return params.TAG_NAME =~ /v.*/
 83             }
 84           }
 85           steps {
 86               container ('maven') {
 87                 input(id: 'release-image-with-tag', message: 'release image with tag?')
 88                   withCredentials([usernamePassword(credentialsId: "$GITHUB_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
 89                     sh 'git config --global user.email "kubesphere@yunify.com" '
 90                     sh 'git config --global user.name "kubesphere" '
 91                     sh 'git tag -a $TAG_NAME -m "$TAG_NAME" '
 92                     sh 'git push http://$GIT_USERNAME:$GIT_PASSWORD@gitee.com/$GITHUB_ACCOUNT/$APP_NAME.git --tags --ipv4'
 93                   }
 94                 sh 'docker tag  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME '
 95                 sh 'docker push  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME '
 96           }
 97           }
 98         }
 99         stage('deploy to production') {
100           when{
101             expression{
102               return params.TAG_NAME =~ /v.*/
103             }
104           }
105           steps {
106             input(id: 'deploy-to-production', message: 'deploy to production?')
107             container ('maven') {
108                 withCredentials([
109                     kubeconfigFile(
110                     credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
111                     variable: 'KUBECONFIG')
112                     ]) {
113                     sh 'envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -'
114                 }
115             }
116           }
117         }
118     }
119 }

  和原来官网项目文件相比修改的包括environment 中的凭证的id 和 自己的相关信息; 92 行推送tag 时候的相关的github 信息替换为gitee 服务器信息和用$APP_NAME 获取仓库名称。

  简单理解上面相关凭证用于拉代码、推镜像、部署到kubernetes。 REGISTRY 是docker 镜像仓库地址;DOCKERHUB_NAMESPACE 是镜像仓库命名空间;GITHUB_ACCOUNT 用于下面拼接github 代码仓库路径; APP_NAME 是打包名称和镜像名称。

5. 到阿里镜像仓库创建新的镜像仓库,名称为 devops-maven-sample(这里创建为公共仓库,不然拉取镜像会失败, 想解决参考: https://developer.aliyun.com/ask/11155?spm=a2c6h.13706215.ask-content.1.3c732a3aUTxxTA)

6. 到kubesphere demo 项目下新建流水线, 名称为 devops-maven-sample, 然后选择代码仓库,后面默认即可(Jenkinsfile 名称也匹配):

7. 点到流水线中,然后点击扫描项目,获取git 上面的Jenkinsfile 文件和代码分支,最终的扫描日志如下:

Started by user admin
[Sun Mar 06 03:47:01 UTC 2022] Starting branch indexing...
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials 
 > git ls-remote --symref -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git
 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching & pruning origin...
Listing remote references...
 > git config --get remote.origin.url # timeout=10
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials 
 > git ls-remote -h -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching upstream changes from origin
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials 
 > git fetch --tags --progress --prune -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
Checking branches...
  Checking branch master
      ‘Jenkinsfile’ found
    Met criteria
No changes detected: master (still at 4b2c785bec6d7c4d2899ec78682b431455529c16)
Processed 1 branches
[Sun Mar 06 03:47:10 UTC 2022] Finished branch indexing. Indexing took 9.2 sec
Finished: SUCCESS

8. 接下来在kubernetes 环境中创建两个namespace, 不创建namespace 在创建kubernetes 资源的时候会报错

创建dev 和 prod 环境对应的namespace

kubectl create ns kubesphere-sample-dev
kubectl create ns kubesphere-sample-prod

9. 然后选择master分支后,点击运行,输入参数v1(该参数用于jenkins 流水线脚本进行判断是否需要推送代码tag和镜像tag)运行流水线,等待其结果:(当我们不输入tag的时候相当于只是部署到开发环境,不打tag等操作)

中间build 过程中,我们全部选择处理

10. 完成后输出如下:

 查看日志如下:

Started by user admin
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git
 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials 
 > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
Obtained Jenkinsfile from 67401e9a817a371f9da32d5606d591e7b3e0598f
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
‘maven-2whbc’ is offline
Agent maven-2whbc is provisioned from template maven
---
apiVersion: "v1"
kind: "Pod"
metadata:
  annotations: {}
  labels:
    jenkins: "slave"
    jenkins/label-digest: "f02c587acd12db3d7b4a28edb5c2eae5f526ce28"
    jenkins/label: "maven"
  name: "maven-2whbc"
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: "node-role.kubernetes.io/worker"
            operator: "In"
            values:
            - "ci"
        weight: 1
  containers:
  - args:
    - "********"
    - "maven-2whbc"
    command:
    - "jenkins-slave"
    env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "devops-jenkins-agent.kubesphere-devops-system:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "maven-2whbc"
    - name: "JENKINS_NAME"
      value: "maven-2whbc"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins/agent"
    - name: "JENKINS_URL"
      value: "http://devops-jenkins.kubesphere-devops-system:80/"
    image: "jenkins/jnlp-slave:3.27-1"
    imagePullPolicy: "IfNotPresent"
    name: "jnlp"
    resources:
      limits:
        memory: "1536Mi"
        cpu: "500m"
      requests:
        memory: "400Mi"
        cpu: "50m"
    tty: false
    volumeMounts:
    - mountPath: "/root/.sonar/cache"
      name: "volume-2"
      readOnly: false
    - mountPath: "/root/.m2"
      name: "volume-1"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  - command:
    - "cat"
    image: "kubesphere/builder-maven:v3.2.0"
    imagePullPolicy: "IfNotPresent"
    name: "maven"
    resources:
      limits:
        ephemeral-storage: "10Gi"
        memory: "8192Mi"
        cpu: "4000m"
      requests:
        ephemeral-storage: "1Gi"
        memory: "100Mi"
        cpu: "100m"
    tty: true
    volumeMounts:
    - mountPath: "/opt/apache-maven-3.5.3/conf/settings.xml"
      name: "config-volume"
      subPath: "settings.xml"
    - mountPath: "/root/.sonar/cache"
      name: "volume-2"
      readOnly: false
    - mountPath: "/root/.m2"
      name: "volume-1"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins/agent"
  nodeSelector: {}
  restartPolicy: "Never"
  securityContext:
    fsGroup: 1000
  tolerations:
  - effect: "NoSchedule"
    key: "node.kubernetes.io/ci"
    operator: "Exists"
  - effect: "PreferNoSchedule"
    key: "node.kubernetes.io/ci"
    operator: "Exists"
  volumes:
  - hostPath:
      path: "/var/run/docker.sock"
    name: "volume-0"
  - hostPath:
      path: "/var/data/jenkins_sonar_cache"
    name: "volume-2"
  - hostPath:
      path: "/var/data/jenkins_maven_cache"
    name: "volume-1"
  - emptyDir:
      medium: ""
    name: "workspace-volume"
  - configMap:
      items:
      - key: "MavenSetting"
        path: "settings.xml"
      name: "ks-devops-agent"
    name: "config-volume"

Running on maven-2whbc in /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential gitee-secret
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://gitee.com/Qiao-Zhi/devops-maven-sample.git
 > git init /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master # timeout=10
Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master)
 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
Commit message: "v1"
 > git rev-list --no-walk 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout scm)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential gitee-secret
Fetching changes from the remote Git repository
Fetching without tags
 > git rev-parse --is-inside-work-tree # timeout=10
 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master)
Commit message: "v1"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (unit test)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
+ mvn clean test
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< io.kubesphere.devops:devops-sample >-----------------
[INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample ---
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.kubesphere.devops.HelloWorldControllerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.527 s - in io.kubesphere.devops.HelloWorldControllerTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.872 s
[INFO] Finished at: 2022-03-06T04:25:53Z
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (build & push)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ mvn clean package -DskipTests
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< io.kubesphere.devops:devops-sample >-----------------
[INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample ---
[INFO] Deleting /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ devops-sample ---
[INFO] Building jar: /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/devops-sample-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- spring-boot-maven-plugin:2.1.11.RELEASE:repackage (repackage) @ devops-sample ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.580 s
[INFO] Finished at: 2022-03-06T04:26:16Z
[INFO] ------------------------------------------------------------------------
[Pipeline] sh
+ docker build -f Dockerfile-online -t registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 .
Sending build context to Docker daemon  17.52MB

Step 1/4 : FROM java:openjdk-8-jre-alpine
 ---> fdc893b19a14
Step 2/4 : WORKDIR /home
 ---> Using cache
 ---> 1e0dc3b20420
Step 3/4 : COPY target/*.jar /home
 ---> fd0896260f0e
Step 4/4 : ENTRYPOINT java -jar *.jar
 ---> Running in 27d46c26a862
Removing intermediate container 27d46c26a862
 ---> 8867ca53b5ca
Successfully built 8867ca53b5ca
Successfully tagged registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3
[Pipeline] withCredentials
Masking supported pattern matches of $DOCKER_USERNAME or $DOCKER_PASSWORD
[Pipeline] {
[Pipeline] sh
+ echo ****
+ docker login registry.cn-hangzhou.aliyuncs.com -u **** --password-stdin
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
20dd87a4c2ab: Layer already exists
78075328e0da: Layer already exists
9f8566ee5135: Layer already exists
0e1bd0a16495: Pushed
SNAPSHOT-master-3: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (push latest)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
78075328e0da: Layer already exists
20dd87a4c2ab: Layer already exists
9f8566ee5135: Layer already exists
0e1bd0a16495: Layer already exists
latest: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy to dev)
[Pipeline] input
deploy to dev?
Proceed or Abort
Approved by admin
[Pipeline] container
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] sh
+ envsubst
+ kubectl apply -f -
deployment.apps/ks-sample-dev configured
service/ks-sample-dev unchanged
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (push with tag)
[Pipeline] container
[Pipeline] {
[Pipeline] input
release image with tag?
Proceed or Abort
Approved by admin
[Pipeline] withCredentials
Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[Pipeline] {
[Pipeline] sh
+ git config --global user.email kubesphere@yunify.com
[Pipeline] sh
+ git config --global user.name kubesphere
[Pipeline] sh
+ git tag -a v1 -m v1
[Pipeline] sh
+ git push http://****:****@gitee.com/****/devops-maven-sample.git --tags --ipv4
remote: Powered by GITEE.COM [GNK-6.3]        
To http://gitee.com/****/devops-maven-sample.git
 * [new tag]         v1 -> v1
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] sh
+ docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
9f8566ee5135: Layer already exists
0e1bd0a16495: Layer already exists
20dd87a4c2ab: Layer already exists
78075328e0da: Layer already exists
v1: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy to production)
[Pipeline] input
deploy to production?
Proceed or Abort
Approved by admin
[Pipeline] container
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] sh
+ envsubst
+ kubectl apply -f -
deployment.apps/ks-sample created
service/ks-sample created
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
View Code

1》jenkins 查看日志如下

 日志如下:

Started by user admin
 > git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git
 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials 
 > git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
Obtained Jenkinsfile from 67401e9a817a371f9da32d5606d591e7b3e0598f
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
‘maven-2whbc’ is offline
Agent maven-2whbc is provisioned from template maven
---
apiVersion: "v1"
kind: "Pod"
metadata:
  annotations: {}
  labels:
    jenkins: "slave"
    jenkins/label-digest: "f02c587acd12db3d7b4a28edb5c2eae5f526ce28"
    jenkins/label: "maven"
  name: "maven-2whbc"
spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - preference:
          matchExpressions:
          - key: "node-role.kubernetes.io/worker"
            operator: "In"
            values:
            - "ci"
        weight: 1
  containers:
  - args:
    - "********"
    - "maven-2whbc"
    command:
    - "jenkins-slave"
    env:
    - name: "JENKINS_SECRET"
      value: "********"
    - name: "JENKINS_TUNNEL"
      value: "devops-jenkins-agent.kubesphere-devops-system:50000"
    - name: "JENKINS_AGENT_NAME"
      value: "maven-2whbc"
    - name: "JENKINS_NAME"
      value: "maven-2whbc"
    - name: "JENKINS_AGENT_WORKDIR"
      value: "/home/jenkins/agent"
    - name: "JENKINS_URL"
      value: "http://devops-jenkins.kubesphere-devops-system:80/"
    image: "jenkins/jnlp-slave:3.27-1"
    imagePullPolicy: "IfNotPresent"
    name: "jnlp"
    resources:
      limits:
        memory: "1536Mi"
        cpu: "500m"
      requests:
        memory: "400Mi"
        cpu: "50m"
    tty: false
    volumeMounts:
    - mountPath: "/root/.sonar/cache"
      name: "volume-2"
      readOnly: false
    - mountPath: "/root/.m2"
      name: "volume-1"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
  - command:
    - "cat"
    image: "kubesphere/builder-maven:v3.2.0"
    imagePullPolicy: "IfNotPresent"
    name: "maven"
    resources:
      limits:
        ephemeral-storage: "10Gi"
        memory: "8192Mi"
        cpu: "4000m"
      requests:
        ephemeral-storage: "1Gi"
        memory: "100Mi"
        cpu: "100m"
    tty: true
    volumeMounts:
    - mountPath: "/opt/apache-maven-3.5.3/conf/settings.xml"
      name: "config-volume"
      subPath: "settings.xml"
    - mountPath: "/root/.sonar/cache"
      name: "volume-2"
      readOnly: false
    - mountPath: "/root/.m2"
      name: "volume-1"
      readOnly: false
    - mountPath: "/var/run/docker.sock"
      name: "volume-0"
      readOnly: false
    - mountPath: "/home/jenkins/agent"
      name: "workspace-volume"
      readOnly: false
    workingDir: "/home/jenkins/agent"
  nodeSelector: {}
  restartPolicy: "Never"
  securityContext:
    fsGroup: 1000
  tolerations:
  - effect: "NoSchedule"
    key: "node.kubernetes.io/ci"
    operator: "Exists"
  - effect: "PreferNoSchedule"
    key: "node.kubernetes.io/ci"
    operator: "Exists"
  volumes:
  - hostPath:
      path: "/var/run/docker.sock"
    name: "volume-0"
  - hostPath:
      path: "/var/data/jenkins_sonar_cache"
    name: "volume-2"
  - hostPath:
      path: "/var/data/jenkins_maven_cache"
    name: "volume-1"
  - emptyDir:
      medium: ""
    name: "workspace-volume"
  - configMap:
      items:
      - key: "MavenSetting"
        path: "settings.xml"
      name: "ks-devops-agent"
    name: "config-volume"

Running on maven-2whbc in /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential gitee-secret
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://gitee.com/Qiao-Zhi/devops-maven-sample.git
 > git init /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master # timeout=10
Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master)
 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
Commit message: "v1"
 > git rev-list --no-walk 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout scm)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential gitee-secret
Fetching changes from the remote Git repository
Fetching without tags
 > git rev-parse --is-inside-work-tree # timeout=10
 > git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials 
 > git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master)
Commit message: "v1"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (unit test)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
+ mvn clean test
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< io.kubesphere.devops:devops-sample >-----------------
[INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample ---
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.kubesphere.devops.HelloWorldControllerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.527 s - in io.kubesphere.devops.HelloWorldControllerTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.872 s
[INFO] Finished at: 2022-03-06T04:25:53Z
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (build & push)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ mvn clean package -DskipTests
[INFO] Scanning for projects...
[INFO] 
[INFO] -----------------< io.kubesphere.devops:devops-sample >-----------------
[INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample ---
[INFO] Deleting /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target
[INFO] 
[INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes
[INFO] 
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ devops-sample ---
[INFO] Building jar: /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/devops-sample-0.0.1-SNAPSHOT.jar
[INFO] 
[INFO] --- spring-boot-maven-plugin:2.1.11.RELEASE:repackage (repackage) @ devops-sample ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.580 s
[INFO] Finished at: 2022-03-06T04:26:16Z
[INFO] ------------------------------------------------------------------------
[Pipeline] sh
+ docker build -f Dockerfile-online -t registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 .
Sending build context to Docker daemon  17.52MB

Step 1/4 : FROM java:openjdk-8-jre-alpine
 ---> fdc893b19a14
Step 2/4 : WORKDIR /home
 ---> Using cache
 ---> 1e0dc3b20420
Step 3/4 : COPY target/*.jar /home
 ---> fd0896260f0e
Step 4/4 : ENTRYPOINT java -jar *.jar
 ---> Running in 27d46c26a862
Removing intermediate container 27d46c26a862
 ---> 8867ca53b5ca
Successfully built 8867ca53b5ca
Successfully tagged registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3
[Pipeline] withCredentials
Masking supported pattern matches of $DOCKER_USERNAME or $DOCKER_PASSWORD
[Pipeline] {
[Pipeline] sh
+ echo ****
+ docker login registry.cn-hangzhou.aliyuncs.com -u **** --password-stdin
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
20dd87a4c2ab: Layer already exists
78075328e0da: Layer already exists
9f8566ee5135: Layer already exists
0e1bd0a16495: Pushed
SNAPSHOT-master-3: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (push latest)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
78075328e0da: Layer already exists
20dd87a4c2ab: Layer already exists
9f8566ee5135: Layer already exists
0e1bd0a16495: Layer already exists
latest: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy to dev)
[Pipeline] input
deploy to dev?
Proceed or Abort
Approved by admin
[Pipeline] container
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] sh
+ envsubst
+ kubectl apply -f -
deployment.apps/ks-sample-dev configured
service/ks-sample-dev unchanged
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (push with tag)
[Pipeline] container
[Pipeline] {
[Pipeline] input
release image with tag?
Proceed or Abort
Approved by admin
[Pipeline] withCredentials
Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[Pipeline] {
[Pipeline] sh
+ git config --global user.email kubesphere@yunify.com
[Pipeline] sh
+ git config --global user.name kubesphere
[Pipeline] sh
+ git tag -a v1 -m v1
[Pipeline] sh
+ git push http://****:****@gitee.com/****/devops-maven-sample.git --tags --ipv4
remote: Powered by GITEE.COM [GNK-6.3]        
To http://gitee.com/****/devops-maven-sample.git
 * [new tag]         v1 -> v1
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] sh
+ docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
9f8566ee5135: Layer already exists
0e1bd0a16495: Layer already exists
20dd87a4c2ab: Layer already exists
78075328e0da: Layer already exists
v1: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy to production)
[Pipeline] input
deploy to production?
Proceed or Abort
Approved by admin
[Pipeline] container
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] sh
+ envsubst
+ kubectl apply -f -
deployment.apps/ks-sample created
service/ks-sample created
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
View Code

2》 kubernetes 环境查看相关的资源

[root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-devops-dev
No resources found in kubesphere-devops-dev namespace.
[root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-sample-dev
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ks-sample-dev   1/1     1            1           12h

NAME                                 READY   STATUS    RESTARTS   AGE
pod/ks-sample-dev-5dc9786dcc-6f9fx   1/1     Running   0          5m48s

NAME                    TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
service/ks-sample-dev   NodePort   10.1.66.88   <none>        8080:30861/TCP   14h
[root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-sample-prod
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/ks-sample   2/2     2            2           4m57s

NAME                             READY   STATUS    RESTARTS   AGE
pod/ks-sample-68b68dfbc6-dpkwk   1/1     Running   0          4m57s
pod/ks-sample-68b68dfbc6-qs2sw   1/1     Running   0          4m57s

NAME                TYPE       CLUSTER-IP   EXTERNAL-IP   PORT(S)          AGE
service/ks-sample   NodePort   10.1.21.72   <none>        8080:30961/TCP   4m57s

3》gitee 查看打的tag 信息

 4》阿里镜像仓库查看版本信息如下:

 5》测试访问

xx@xx MINGW64 /d/study/devops-maven-sample (master)
$ curl http://192.168.13.107:30861
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     2  100     2    0     0    166      0 --:--:-- --:--:-- --:--:--   181v1

xx@xx MINGW64 /d/study/devops-maven-sample (master)
$ curl http://192.168.13.107:30961
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100     2  100     2    0     0     68      0 --:--:-- --:--:-- --:--:--    71v1

11. 总结

(0) 前置

  其实通过kubesphere 流水线操作,最终还是都会到Jenkins中进行操作,其操作和自定义流水线一样。大致就是打代码、测试代码、打包、制作镜像并打tag、传送到镜像仓库、创建kubernetes 相关资源。在jenkins 执行过程中,会创建一个代理pod,比如:

[root@k8smaster01 ~]# kubectl get pods -A| grep maven
kubesphere-devops-worker       maven-cfqfb                                        2/2     Running     0          47s
kubesphere-devops-worker       maven-gzjdm                                        1/2     Error       0          18h

  其中git 拉取代码、maven打包都是在下面pods 操作的。进入一个pod,查看其环境如下:

(1)Jenkins file文件查看分析

pipeline {
  agent {
    node {
      label 'maven'
    }
  }

    parameters {
        string(name:'TAG_NAME',defaultValue: '',description:'')
    }

    environment {
        DOCKER_CREDENTIAL_ID = 'ali-registery'
        GITHUB_CREDENTIAL_ID = 'gitee-secret'
        KUBECONFIG_CREDENTIAL_ID = 'demo-kubeconfig'
        REGISTRY = 'registry.cn-hangzhou.aliyuncs.com'
        DOCKERHUB_NAMESPACE = 'qlq_repository'
        GITHUB_ACCOUNT = 'qiao-zhi'
        APP_NAME = 'devops-maven-sample'
    }

    stages {
        stage ('checkout scm') {
            steps {
                checkout(scm)
            }
        }

        stage ('unit test') {
            steps {
                container ('maven') {
                    sh 'mvn clean test'
                }
            }
        }
 
        stage ('build & push') {
            steps {
                container ('maven') {
                    sh 'mvn clean package -DskipTests'
                    sh 'docker build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .'
                    withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) {
                        sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
                        sh 'docker push  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER'
                    }
                }
            }
        }

        stage('push latest'){
           when{
             branch 'master'
           }
           steps{
                container ('maven') {
                  sh 'docker tag  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
                  sh 'docker push  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
                }
           }
        }

        stage('deploy to dev') {
          when{
            branch 'master'
          }
          steps {
            input(id: 'deploy-to-dev', message: 'deploy to dev?')
            container ('maven') {
                withCredentials([
                    kubeconfigFile(
                    credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
                    variable: 'KUBECONFIG')
                    ]) {
                    sh 'envsubst < deploy/dev-all-in-one/devops-sample.yaml | kubectl apply -f -'
                }
            }
          }
        }
        stage('push with tag'){
          when{
            expression{
              return params.TAG_NAME =~ /v.*/
            }
          }
          steps {
              container ('maven') {
                input(id: 'release-image-with-tag', message: 'release image with tag?')
                  withCredentials([usernamePassword(credentialsId: "$GITHUB_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
                    sh 'git config --global user.email "kubesphere@yunify.com" '
                    sh 'git config --global user.name "kubesphere" '
                    sh 'git tag -a $TAG_NAME -m "$TAG_NAME" '
                    sh 'git push http://$GIT_USERNAME:$GIT_PASSWORD@gitee.com/$GITHUB_ACCOUNT/$APP_NAME.git --tags --ipv4'
                  }
                sh 'docker tag  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME '
                sh 'docker push  $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME '
          }
          }
        }
        stage('deploy to production') {
          when{
            expression{
              return params.TAG_NAME =~ /v.*/
            }
          }
          steps {
            input(id: 'deploy-to-production', message: 'deploy to production?')
            container ('maven') {
                withCredentials([
                    kubeconfigFile(
                    credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
                    variable: 'KUBECONFIG')
                    ]) {
                    sh 'envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -'
                }
            }
          }
        }
    }
}

前面定义了一些全局环境变量,按阶段进行分析

1》checkout scm 拉取代码

2》unit test 执行单元测试

3》build & push: 执行mvn打包, 然后基于Dockerfile-online 构造本地镜像,镜像的版本用代码分支和buildnumber 做区分,然后将镜像推送代码镜像仓库

4》push latest: 当代码分支是master 的时候,将3》的镜像打包为latest版本的镜像,然后推到镜像仓库

5》deploy to dev:当代码分支是master 的时候,相当于是执行 deploy/dev-all-in-one/devops-sample.yaml文件,相当于到kubernetes 创建资源

6》push with tag:当我们传递的参数是vxxx的时候,将代码打上相应的标签并推送到代码仓库;将镜像也打上相应的标签并推送到相应的镜像仓库

7》deploy to production:当我们传递的参数是vxxx的时候, 需要我们验证下是否需要将相关资源创建到kubernetes环境的prod 仓库,使用的文件是deploy/prod-all-in-one/devops-sample.yaml。

envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -

envsubst 是用shell格式字符串中的值替换环境变量。要替换的变量应位于${var}或$var格式。例如

[root@k8smaster01 ~]# export MYKEY=123456
[root@k8smaster01 ~]# echo $MYKEY
123456
[root@k8smaster01 ~]# cat test.txt 
$MYKEY
[root@k8smaster01 ~]# envsubst < test.txt 
123456

(2) 几个重要kubernetes 文件如下:

deploy/dev-all-in-one/devops-sample.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kubesphere
    component: ks-sample-dev
    tier: backend
  name: ks-sample-dev
  namespace: kubesphere-sample-dev
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  selector:
    matchLabels:
      app: kubesphere
      component: ks-sample-dev
      tier: backend
  template:
    metadata:
      labels:
        app: kubesphere
        component: ks-sample-dev
        tier: backend
    spec:
      containers:
        - env:
            - name: CACHE_IGNORE
              value: js|html
            - name: CACHE_PUBLIC_EXPIRATION
              value: 3d
          image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            timeoutSeconds: 10
            failureThreshold: 30
            periodSeconds: 5
          imagePullPolicy: Always
          name: ks-sample
          ports:
            - containerPort: 8080
              protocol: TCP
          resources:
            limits:
              cpu: 300m
              memory: 600Mi
            requests:
              cpu: 100m
              memory: 100Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: kubesphere
    component: ks-sample-dev
  name: ks-sample-dev
  namespace: kubesphere-sample-dev
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
      nodePort: 30861
  selector:
    app: kubesphere
    component: ks-sample-dev
    tier: backend
  sessionAffinity: None
  type: NodePort
View Code 

deploy/prod-all-in-one/devops-sample.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kubesphere
    component: ks-sample
    tier: backend
  name: ks-sample
  namespace: kubesphere-sample-prod
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  selector:
    matchLabels:
      app: kubesphere
      component: ks-sample
      tier: backend
  strategy:
    rollingUpdate:
      maxSurge: 100%
      maxUnavailable: 100%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kubesphere
        component: ks-sample
        tier: backend
    spec:
      containers:
        - env:
            - name: CACHE_IGNORE
              value: js|html
            - name: CACHE_PUBLIC_EXPIRATION
              value: 3d
          image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            timeoutSeconds: 10
            failureThreshold: 30
            periodSeconds: 5
          imagePullPolicy: Always
          name: ks
          ports:
            - containerPort: 8080
              protocol: TCP
          resources:
            limits:
              cpu: 300m
              memory: 600Mi
            requests:
              cpu: 100m
              memory: 100Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: kubesphere
    component: ks-sample
  name: ks-sample
  namespace: kubesphere-sample-prod
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
      nodePort: 30961
  selector:
    app: kubesphere
    component: ks-sample
    tier: backend
  sessionAffinity: None
  type: NodePort
View Code

 

 代码仓库和修改后配置文件:https://gitee.com/Qiao-Zhi/devops-maven-sample

 

补充: 在安装过程中遇到一些磁盘不足,报错信息如下:

Warning  Evicted    49m   kubelet            The node had condition: [DiskPressure].

解决参考:

https://blog.51cto.com/riverxyz/2758421

vm中对centos 扩容: https://ld246.com/article/1566021346577

补充: 如果是阿里私有镜像仓库,需要创建secret 然后拉取,如下

1. 创建私有镜像仓库: 账号密码记得替换

kubectl create secret docker-registry registry-aliyun --namespace=kubesphere-sample-dev \
    --docker-server=registry.cn-hangzhou.aliyuncs.com --docker-username=usernamre \
--docker-password=pwd --docker-email=xxx@163.com

2. 修改yaml文件,增加镜像拉取secret

(1) deploy/dev-all-in-one/devops-sample.yaml:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kubesphere
    component: ks-sample-dev
    tier: backend
  name: ks-sample-dev
  namespace: kubesphere-sample-dev
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  selector:
    matchLabels:
      app: kubesphere
      component: ks-sample-dev
      tier: backend
  template:
    metadata:
      labels:
        app: kubesphere
        component: ks-sample-dev
        tier: backend
    spec:
      containers:
        - env:
            - name: CACHE_IGNORE
              value: js|html
            - name: CACHE_PUBLIC_EXPIRATION
              value: 3d
          image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            timeoutSeconds: 10
            failureThreshold: 30
            periodSeconds: 5
          imagePullPolicy: Always
          name: ks-sample
          ports:
            - containerPort: 8080
              protocol: TCP
          resources:
            limits:
              cpu: 300m
              memory: 600Mi
            requests:
              cpu: 100m
              memory: 100Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      imagePullSecrets:
        - name: registry-aliyun
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: kubesphere
    component: ks-sample-dev
  name: ks-sample-dev
  namespace: kubesphere-sample-dev
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
      nodePort: 30861
  selector:
    app: kubesphere
    component: ks-sample-dev
    tier: backend
  sessionAffinity: None
  type: NodePort
View Code

(2) deploy/prod-all-in-one/devops-sample.yaml

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: kubesphere
    component: ks-sample
    tier: backend
  name: ks-sample
  namespace: kubesphere-sample-prod
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  selector:
    matchLabels:
      app: kubesphere
      component: ks-sample
      tier: backend
  strategy:
    rollingUpdate:
      maxSurge: 100%
      maxUnavailable: 100%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: kubesphere
        component: ks-sample
        tier: backend
    spec:
      containers:
        - env:
            - name: CACHE_IGNORE
              value: js|html
            - name: CACHE_PUBLIC_EXPIRATION
              value: 3d
          image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME
          readinessProbe:
            httpGet:
              path: /
              port: 8080
            timeoutSeconds: 10
            failureThreshold: 30
            periodSeconds: 5
          imagePullPolicy: Always
          name: ks
          ports:
            - containerPort: 8080
              protocol: TCP
          resources:
            limits:
              cpu: 300m
              memory: 600Mi
            requests:
              cpu: 100m
              memory: 100Mi
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      imagePullSecrets:
        - name: registry-aliyun
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: kubesphere
    component: ks-sample
  name: ks-sample
  namespace: kubesphere-sample-prod
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
      nodePort: 30961
  selector:
    app: kubesphere
    component: ks-sample
    tier: backend
  sessionAffinity: None
  type: NodePort
View Code

 

标签:INFO,maven,Pipeline,git,kubesphere,devops,sample,安装
来源: https://www.cnblogs.com/qlqwjy/p/15965008.html