kubesphere devops 安装以及简单使用
作者：互联网
简单研究下kubesphere 中devops 的使用。
1. 安装
实际上就是修改安装过程中的 cluster-configuration.yaml 文件的devops.enable 设为true 即可, kebesphere 会自己下载jenkins，我们通过kubesphere 操作流水线最终也会到jenkins 中操作对应的流水线。
参考: https://kubesphere.io/zh/docs/pluggable-components/devops/
安装过程中出现的问题: nfs服务器磁盘空间、k8snode01 磁盘空间、k8snode01 内存问题等等。建议在虚拟机初始化就设置的大一点。
1. 安装成功后查看相关pods、service 如下：
[root@k8smaster01 kubesphere]# kubectl get pods -n kubesphere-devops-system NAME READY STATUS RESTARTS AGE devops-27439710-f4q5f 0/1 Completed 0 8m12s devops-apiserver-7c6774fff5-nmj84 1/1 Running 0 21m devops-controller-98975d478-4xhxn 1/1 Running 0 21m devops-jenkins-64464f495f-4z8br 1/1 Running 1 21m s2ioperator-0 1/1 Running 0 21m [root@k8smaster01 kubesphere]# kubectl get svc -n kubesphere-devops-system NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE devops-apiserver ClusterIP 10.1.248.197 <none> 9090/TCP 21m devops-jenkins NodePort 10.1.45.197 <none> 80:30180/TCP 21m devops-jenkins-agent ClusterIP 10.1.113.105 <none> 50000/TCP 21m s2ioperator-metrics-service ClusterIP 10.1.161.145 <none> 8080/TCP 21m s2ioperator-trigger-service ClusterIP 10.1.119.55 <none> 8081/TCP 21m webhook-server-service ClusterIP 10.1.46.76 <none> 443/TCP 21m
2. 然后登录kubesphere 控制台看到如下:
2. 登录jenkins
上面可以看到service， 30180 端口可以访问jenkins，在找密码的时候发现其没有密码文件。 最后通过官网发现： 是用kubesphere 的用户体系进行登录。
参考: https://v2-1.docs.kubesphere.io/docs/zh-CN/devops/jenkins-setting/
接下来可以简单的看一下jenkins 控制台界面安装的一些插件等信息。
2. 使用
接下来就是使用kubesphere 的devops 发布一个简单的springboot 项目。 这里使用github作为代码仓库，阿里镜像仓库作为镜像仓库。
官网对其运行过程解释如下：
首先，Jenkins Master 创建一个 Pod 来运行流水线。Kubernetes 创建 Pod 作为 Jenkins Master 的 Agent，该 Pod 会在流水线完成之后销毁。主要流程包括克隆代码、构建和推送镜像以及部署工作负载。
官方提供的两个java 项目：
https://github.com/kubesphere/devops-java-sample
https://github.com/kubesphere/devops-maven-sample 本次测试使用这个项目进行测试
搭建过程参考：
https://kubesphere.io/docs/devops-user-guide/how-to-use/create-a-pipeline-using-jenkinsfile/
1. fork项目到自己的github 仓库
如果网络波动太大的话可以使用国内的gitee，作为代码仓库进行测试。我下面使用gitee。
2. 到kubesphere 进行操作
这里用户都用admin 进行操作，如果想测试kubesphere 的用户体系可以自己创建多个用户进行操作。
1. 到访问控制-》企业空间创建一个企业空间 dev， 用于独立的测试。这个可以理解为大的一个资源隔离的空间。
一个空间可以有多个普通项目和devops 项目。
普通项目可以用来管理独立的kubernetes 资源，包括pod、svc 等，实际kubesphere 后台的操作就是新建了一个对应的namespace用于隔离相应资源，namespace 的名称就是项目名称；
devops 项目可以包含多个jenkins流水线， 用于devops 操作，kubesphere 后台也创建了一个对应的namespace，namespace名称为devops 项目名称加5位随机字母。
2. dev 企业空间创建一个项目，名称为 demo。 查看namespace如下
[root@k8smaster01 ~]# kubectl get ns|grep demo demo59wjg Active 26s
3. 在demo 项目下面建立自己的凭证：
1》gitee 的凭证
2》镜像仓库的凭证
3》kubeconfig 用于kubernetes 使用
4. 本地复制项目Jenkinsfile-online，命名为Jenkinsfile。 原来文件用于备份，新文件用于jenkins 流水线脚本。修改Jenkinsfile 里面内容的凭证信息，最终如下：
1 pipeline {
2 agent {
3 node {
4 label 'maven'
5 }
6 }
7
8 parameters {
9 string(name:'TAG_NAME',defaultValue: '',description:'')
10 }
11
12 environment {
13 DOCKER_CREDENTIAL_ID = 'ali-registery'
14 GITHUB_CREDENTIAL_ID = 'gitee-secret'
15 KUBECONFIG_CREDENTIAL_ID = 'demo-kubeconfig'
16 REGISTRY = 'registry.cn-hangzhou.aliyuncs.com'
17 DOCKERHUB_NAMESPACE = 'qlq_repository'
18 GITHUB_ACCOUNT = 'qiao-zhi'
19 APP_NAME = 'devops-maven-sample'
20 }
21
22 stages {
23 stage ('checkout scm') {
24 steps {
25 checkout(scm)
26 }
27 }
28
29 stage ('unit test') {
30 steps {
31 container ('maven') {
32 sh 'mvn clean test'
33 }
34 }
35 }
36
37 stage ('build & push') {
38 steps {
39 container ('maven') {
40 sh 'mvn clean package -DskipTests'
41 sh 'docker build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .'
42 withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) {
43 sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
44 sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER'
45 }
46 }
47 }
48 }
49
50 stage('push latest'){
51 when{
52 branch 'master'
53 }
54 steps{
55 container ('maven') {
56 sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
57 sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
58 }
59 }
60 }
61
62 stage('deploy to dev') {
63 when{
64 branch 'master'
65 }
66 steps {
67 input(id: 'deploy-to-dev', message: 'deploy to dev?')
68 container ('maven') {
69 withCredentials([
70 kubeconfigFile(
71 credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
72 variable: 'KUBECONFIG')
73 ]) {
74 sh 'envsubst < deploy/dev-all-in-one/devops-sample.yaml | kubectl apply -f -'
75 }
76 }
77 }
78 }
79 stage('push with tag'){
80 when{
81 expression{
82 return params.TAG_NAME =~ /v.*/
83 }
84 }
85 steps {
86 container ('maven') {
87 input(id: 'release-image-with-tag', message: 'release image with tag?')
88 withCredentials([usernamePassword(credentialsId: "$GITHUB_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
89 sh 'git config --global user.email "kubesphere@yunify.com" '
90 sh 'git config --global user.name "kubesphere" '
91 sh 'git tag -a $TAG_NAME -m "$TAG_NAME" '
92 sh 'git push http://$GIT_USERNAME:$GIT_PASSWORD@gitee.com/$GITHUB_ACCOUNT/$APP_NAME.git --tags --ipv4'
93 }
94 sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME '
95 sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME '
96 }
97 }
98 }
99 stage('deploy to production') {
100 when{
101 expression{
102 return params.TAG_NAME =~ /v.*/
103 }
104 }
105 steps {
106 input(id: 'deploy-to-production', message: 'deploy to production?')
107 container ('maven') {
108 withCredentials([
109 kubeconfigFile(
110 credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
111 variable: 'KUBECONFIG')
112 ]) {
113 sh 'envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -'
114 }
115 }
116 }
117 }
118 }
119 }
和原来官网项目文件相比修改的包括environment 中的凭证的id 和 自己的相关信息； 92 行推送tag 时候的相关的github 信息替换为gitee 服务器信息和用$APP_NAME 获取仓库名称。
简单理解上面相关凭证用于拉代码、推镜像、部署到kubernetes。 REGISTRY 是docker 镜像仓库地址；DOCKERHUB_NAMESPACE 是镜像仓库命名空间；GITHUB_ACCOUNT 用于下面拼接github 代码仓库路径； APP_NAME 是打包名称和镜像名称。
5. 到阿里镜像仓库创建新的镜像仓库，名称为 devops-maven-sample(这里创建为公共仓库，不然拉取镜像会失败， 想解决参考： https://developer.aliyun.com/ask/11155?spm=a2c6h.13706215.ask-content.1.3c732a3aUTxxTA)
6. 到kubesphere demo 项目下新建流水线， 名称为 devops-maven-sample, 然后选择代码仓库，后面默认即可(Jenkinsfile 名称也匹配)：
7. 点到流水线中，然后点击扫描项目，获取git 上面的Jenkinsfile 文件和代码分支，最终的扫描日志如下：
Started by user admin
[Sun Mar 06 03:47:01 UTC 2022] Starting branch indexing...
> git --version # timeout=10
> git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials
> git ls-remote --symref -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
> git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git
> git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching & pruning origin...
Listing remote references...
> git config --get remote.origin.url # timeout=10
> git --version # timeout=10
> git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials
> git ls-remote -h -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching upstream changes from origin
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git fetch --tags --progress --prune -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
Checking branches...
Checking branch master
‘Jenkinsfile’ found
Met criteria
No changes detected: master (still at 4b2c785bec6d7c4d2899ec78682b431455529c16)
Processed 1 branches
[Sun Mar 06 03:47:10 UTC 2022] Finished branch indexing. Indexing took 9.2 sec
Finished: SUCCESS
8. 接下来在kubernetes 环境中创建两个namespace， 不创建namespace 在创建kubernetes 资源的时候会报错
创建dev 和 prod 环境对应的namespace
kubectl create ns kubesphere-sample-dev
kubectl create ns kubesphere-sample-prod
9. 然后选择master分支后，点击运行，输入参数v1(该参数用于jenkins 流水线脚本进行判断是否需要推送代码tag和镜像tag)运行流水线，等待其结果：(当我们不输入tag的时候相当于只是部署到开发环境，不打tag等操作)
中间build 过程中，我们全部选择处理
10. 完成后输出如下：
查看日志如下：
Started by user admin
> git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git
> git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
> git --version # timeout=10
> git --version # 'git version 2.11.0'
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
Obtained Jenkinsfile from 67401e9a817a371f9da32d5606d591e7b3e0598f
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
‘maven-2whbc’ is offline
Agent maven-2whbc is provisioned from template maven
---
apiVersion: "v1"
kind: "Pod"
metadata:
annotations: {}
labels:
jenkins: "slave"
jenkins/label-digest: "f02c587acd12db3d7b4a28edb5c2eae5f526ce28"
jenkins/label: "maven"
name: "maven-2whbc"
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: "node-role.kubernetes.io/worker"
operator: "In"
values:
- "ci"
weight: 1
containers:
- args:
- "********"
- "maven-2whbc"
command:
- "jenkins-slave"
env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_TUNNEL"
value: "devops-jenkins-agent.kubesphere-devops-system:50000"
- name: "JENKINS_AGENT_NAME"
value: "maven-2whbc"
- name: "JENKINS_NAME"
value: "maven-2whbc"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://devops-jenkins.kubesphere-devops-system:80/"
image: "jenkins/jnlp-slave:3.27-1"
imagePullPolicy: "IfNotPresent"
name: "jnlp"
resources:
limits:
memory: "1536Mi"
cpu: "500m"
requests:
memory: "400Mi"
cpu: "50m"
tty: false
volumeMounts:
- mountPath: "/root/.sonar/cache"
name: "volume-2"
readOnly: false
- mountPath: "/root/.m2"
name: "volume-1"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
- command:
- "cat"
image: "kubesphere/builder-maven:v3.2.0"
imagePullPolicy: "IfNotPresent"
name: "maven"
resources:
limits:
ephemeral-storage: "10Gi"
memory: "8192Mi"
cpu: "4000m"
requests:
ephemeral-storage: "1Gi"
memory: "100Mi"
cpu: "100m"
tty: true
volumeMounts:
- mountPath: "/opt/apache-maven-3.5.3/conf/settings.xml"
name: "config-volume"
subPath: "settings.xml"
- mountPath: "/root/.sonar/cache"
name: "volume-2"
readOnly: false
- mountPath: "/root/.m2"
name: "volume-1"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
workingDir: "/home/jenkins/agent"
nodeSelector: {}
restartPolicy: "Never"
securityContext:
fsGroup: 1000
tolerations:
- effect: "NoSchedule"
key: "node.kubernetes.io/ci"
operator: "Exists"
- effect: "PreferNoSchedule"
key: "node.kubernetes.io/ci"
operator: "Exists"
volumes:
- hostPath:
path: "/var/run/docker.sock"
name: "volume-0"
- hostPath:
path: "/var/data/jenkins_sonar_cache"
name: "volume-2"
- hostPath:
path: "/var/data/jenkins_maven_cache"
name: "volume-1"
- emptyDir:
medium: ""
name: "workspace-volume"
- configMap:
items:
- key: "MavenSetting"
path: "settings.xml"
name: "ks-devops-agent"
name: "config-volume"
Running on maven-2whbc in /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential gitee-secret
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://gitee.com/Qiao-Zhi/devops-maven-sample.git
> git init /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master # timeout=10
Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git
> git --version # timeout=10
> git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master)
> git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config core.sparsecheckout # timeout=10
> git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
Commit message: "v1"
> git rev-list --no-walk 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout scm)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential gitee-secret
Fetching changes from the remote Git repository
Fetching without tags
> git rev-parse --is-inside-work-tree # timeout=10
> git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git
> git --version # timeout=10
> git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master)
Commit message: "v1"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (unit test)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
> git config core.sparsecheckout # timeout=10
> git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
+ mvn clean test
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< io.kubesphere.devops:devops-sample >-----------------
[INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.kubesphere.devops.HelloWorldControllerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.527 s - in io.kubesphere.devops.HelloWorldControllerTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.872 s
[INFO] Finished at: 2022-03-06T04:25:53Z
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (build & push)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ mvn clean package -DskipTests
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< io.kubesphere.devops:devops-sample >-----------------
[INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample ---
[INFO] Deleting /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ devops-sample ---
[INFO] Building jar: /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/devops-sample-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.1.11.RELEASE:repackage (repackage) @ devops-sample ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.580 s
[INFO] Finished at: 2022-03-06T04:26:16Z
[INFO] ------------------------------------------------------------------------
[Pipeline] sh
+ docker build -f Dockerfile-online -t registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 .
Sending build context to Docker daemon 17.52MB
Step 1/4 : FROM java:openjdk-8-jre-alpine
---> fdc893b19a14
Step 2/4 : WORKDIR /home
---> Using cache
---> 1e0dc3b20420
Step 3/4 : COPY target/*.jar /home
---> fd0896260f0e
Step 4/4 : ENTRYPOINT java -jar *.jar
---> Running in 27d46c26a862
Removing intermediate container 27d46c26a862
---> 8867ca53b5ca
Successfully built 8867ca53b5ca
Successfully tagged registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3
[Pipeline] withCredentials
Masking supported pattern matches of $DOCKER_USERNAME or $DOCKER_PASSWORD
[Pipeline] {
[Pipeline] sh
+ echo ****
+ docker login registry.cn-hangzhou.aliyuncs.com -u **** --password-stdin
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
20dd87a4c2ab: Layer already exists
78075328e0da: Layer already exists
9f8566ee5135: Layer already exists
0e1bd0a16495: Pushed
SNAPSHOT-master-3: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (push latest)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
78075328e0da: Layer already exists
20dd87a4c2ab: Layer already exists
9f8566ee5135: Layer already exists
0e1bd0a16495: Layer already exists
latest: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy to dev)
[Pipeline] input
deploy to dev?
Proceed or Abort
Approved by admin
[Pipeline] container
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] sh
+ envsubst
+ kubectl apply -f -
deployment.apps/ks-sample-dev configured
service/ks-sample-dev unchanged
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (push with tag)
[Pipeline] container
[Pipeline] {
[Pipeline] input
release image with tag?
Proceed or Abort
Approved by admin
[Pipeline] withCredentials
Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[Pipeline] {
[Pipeline] sh
+ git config --global user.email kubesphere@yunify.com
[Pipeline] sh
+ git config --global user.name kubesphere
[Pipeline] sh
+ git tag -a v1 -m v1
[Pipeline] sh
+ git push http://****:****@gitee.com/****/devops-maven-sample.git --tags --ipv4
remote: Powered by [01;33mGITEE.COM [0m[[01;35mGNK-6.3[0m][0m
To http://gitee.com/****/devops-maven-sample.git
* [new tag] v1 -> v1
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] sh
+ docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
9f8566ee5135: Layer already exists
0e1bd0a16495: Layer already exists
20dd87a4c2ab: Layer already exists
78075328e0da: Layer already exists
v1: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy to production)
[Pipeline] input
deploy to production?
Proceed or Abort
Approved by admin
[Pipeline] container
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] sh
+ envsubst
+ kubectl apply -f -
deployment.apps/ks-sample created
service/ks-sample created
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
View Code
1》jenkins 查看日志如下
日志如下：
Started by user admin
> git rev-parse --is-inside-work-tree # timeout=10
Setting origin to https://gitee.com/Qiao-Zhi/devops-maven-sample.git
> git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching origin...
Fetching upstream changes from origin
> git --version # timeout=10
> git --version # 'git version 2.11.0'
> git config --get remote.origin.url # timeout=10
using GIT_ASKPASS to set credentials
> git fetch --tags --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
Obtained Jenkinsfile from 67401e9a817a371f9da32d5606d591e7b3e0598f
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Still waiting to schedule task
‘maven-2whbc’ is offline
Agent maven-2whbc is provisioned from template maven
---
apiVersion: "v1"
kind: "Pod"
metadata:
annotations: {}
labels:
jenkins: "slave"
jenkins/label-digest: "f02c587acd12db3d7b4a28edb5c2eae5f526ce28"
jenkins/label: "maven"
name: "maven-2whbc"
spec:
affinity:
nodeAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- preference:
matchExpressions:
- key: "node-role.kubernetes.io/worker"
operator: "In"
values:
- "ci"
weight: 1
containers:
- args:
- "********"
- "maven-2whbc"
command:
- "jenkins-slave"
env:
- name: "JENKINS_SECRET"
value: "********"
- name: "JENKINS_TUNNEL"
value: "devops-jenkins-agent.kubesphere-devops-system:50000"
- name: "JENKINS_AGENT_NAME"
value: "maven-2whbc"
- name: "JENKINS_NAME"
value: "maven-2whbc"
- name: "JENKINS_AGENT_WORKDIR"
value: "/home/jenkins/agent"
- name: "JENKINS_URL"
value: "http://devops-jenkins.kubesphere-devops-system:80/"
image: "jenkins/jnlp-slave:3.27-1"
imagePullPolicy: "IfNotPresent"
name: "jnlp"
resources:
limits:
memory: "1536Mi"
cpu: "500m"
requests:
memory: "400Mi"
cpu: "50m"
tty: false
volumeMounts:
- mountPath: "/root/.sonar/cache"
name: "volume-2"
readOnly: false
- mountPath: "/root/.m2"
name: "volume-1"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
- command:
- "cat"
image: "kubesphere/builder-maven:v3.2.0"
imagePullPolicy: "IfNotPresent"
name: "maven"
resources:
limits:
ephemeral-storage: "10Gi"
memory: "8192Mi"
cpu: "4000m"
requests:
ephemeral-storage: "1Gi"
memory: "100Mi"
cpu: "100m"
tty: true
volumeMounts:
- mountPath: "/opt/apache-maven-3.5.3/conf/settings.xml"
name: "config-volume"
subPath: "settings.xml"
- mountPath: "/root/.sonar/cache"
name: "volume-2"
readOnly: false
- mountPath: "/root/.m2"
name: "volume-1"
readOnly: false
- mountPath: "/var/run/docker.sock"
name: "volume-0"
readOnly: false
- mountPath: "/home/jenkins/agent"
name: "workspace-volume"
readOnly: false
workingDir: "/home/jenkins/agent"
nodeSelector: {}
restartPolicy: "Never"
securityContext:
fsGroup: 1000
tolerations:
- effect: "NoSchedule"
key: "node.kubernetes.io/ci"
operator: "Exists"
- effect: "PreferNoSchedule"
key: "node.kubernetes.io/ci"
operator: "Exists"
volumes:
- hostPath:
path: "/var/run/docker.sock"
name: "volume-0"
- hostPath:
path: "/var/data/jenkins_sonar_cache"
name: "volume-2"
- hostPath:
path: "/var/data/jenkins_maven_cache"
name: "volume-1"
- emptyDir:
medium: ""
name: "workspace-volume"
- configMap:
items:
- key: "MavenSetting"
path: "settings.xml"
name: "ks-devops-agent"
name: "config-volume"
Running on maven-2whbc in /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential gitee-secret
Cloning the remote Git repository
Cloning with configured refspecs honoured and without tags
Cloning repository https://gitee.com/Qiao-Zhi/devops-maven-sample.git
> git init /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master # timeout=10
Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git
> git --version # timeout=10
> git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Avoid second fetch
Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master)
> git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
> git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
> git config core.sparsecheckout # timeout=10
> git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
Commit message: "v1"
> git rev-list --no-walk 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (checkout scm)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
using credential gitee-secret
Fetching changes from the remote Git repository
Fetching without tags
> git rev-parse --is-inside-work-tree # timeout=10
> git config remote.origin.url https://gitee.com/Qiao-Zhi/devops-maven-sample.git # timeout=10
Fetching upstream changes from https://gitee.com/Qiao-Zhi/devops-maven-sample.git
> git --version # timeout=10
> git --version # 'git version 2.11.0'
using GIT_ASKPASS to set credentials
> git fetch --no-tags --progress -- https://gitee.com/Qiao-Zhi/devops-maven-sample.git +refs/heads/*:refs/remotes/origin/* # timeout=10
Checking out Revision 67401e9a817a371f9da32d5606d591e7b3e0598f (master)
Commit message: "v1"
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (unit test)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
> git config core.sparsecheckout # timeout=10
> git checkout -f 67401e9a817a371f9da32d5606d591e7b3e0598f # timeout=10
+ mvn clean test
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< io.kubesphere.devops:devops-sample >-----------------
[INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample ---
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample ---
[INFO]
[INFO] -------------------------------------------------------
[INFO] T E S T S
[INFO] -------------------------------------------------------
[INFO] Running io.kubesphere.devops.HelloWorldControllerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.527 s - in io.kubesphere.devops.HelloWorldControllerTest
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15.872 s
[INFO] Finished at: 2022-03-06T04:25:53Z
[INFO] ------------------------------------------------------------------------
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (build & push)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ mvn clean package -DskipTests
[INFO] Scanning for projects...
[INFO]
[INFO] -----------------< io.kubesphere.devops:devops-sample >-----------------
[INFO] Building devops-sample :: HelloWorld Demo 0.0.1-SNAPSHOT
[INFO] --------------------------------[ jar ]---------------------------------
[INFO]
[INFO] --- maven-clean-plugin:3.1.0:clean (default-clean) @ devops-sample ---
[INFO] Deleting /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target
[INFO]
[INFO] --- jacoco-maven-plugin:0.8.2:prepare-agent (agent-for-ut) @ devops-sample ---
[INFO] argLine set to -javaagent:/root/.m2/repository/org/jacoco/org.jacoco.agent/0.8.2/org.jacoco.agent-0.8.2-runtime.jar=destfile=/home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/jacoco.exec,append=true
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:resources (default-resources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/main/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:compile (default-compile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 2 source files to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/classes
[INFO]
[INFO] --- maven-resources-plugin:3.1.0:testResources (default-testResources) @ devops-sample ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/src/test/resources
[INFO]
[INFO] --- maven-compiler-plugin:3.8.1:testCompile (default-testCompile) @ devops-sample ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 1 source file to /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/test-classes
[INFO]
[INFO] --- maven-surefire-plugin:2.22.2:test (default-test) @ devops-sample ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:3.1.2:jar (default-jar) @ devops-sample ---
[INFO] Building jar: /home/jenkins/agent/workspace/59wjg_devops-maven-sample_master/target/devops-sample-0.0.1-SNAPSHOT.jar
[INFO]
[INFO] --- spring-boot-maven-plugin:2.1.11.RELEASE:repackage (repackage) @ devops-sample ---
[INFO] Replacing main artifact with repackaged archive
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14.580 s
[INFO] Finished at: 2022-03-06T04:26:16Z
[INFO] ------------------------------------------------------------------------
[Pipeline] sh
+ docker build -f Dockerfile-online -t registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 .
Sending build context to Docker daemon 17.52MB
Step 1/4 : FROM java:openjdk-8-jre-alpine
---> fdc893b19a14
Step 2/4 : WORKDIR /home
---> Using cache
---> 1e0dc3b20420
Step 3/4 : COPY target/*.jar /home
---> fd0896260f0e
Step 4/4 : ENTRYPOINT java -jar *.jar
---> Running in 27d46c26a862
Removing intermediate container 27d46c26a862
---> 8867ca53b5ca
Successfully built 8867ca53b5ca
Successfully tagged registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3
[Pipeline] withCredentials
Masking supported pattern matches of $DOCKER_USERNAME or $DOCKER_PASSWORD
[Pipeline] {
[Pipeline] sh
+ echo ****
+ docker login registry.cn-hangzhou.aliyuncs.com -u **** --password-stdin
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
20dd87a4c2ab: Layer already exists
78075328e0da: Layer already exists
9f8566ee5135: Layer already exists
0e1bd0a16495: Pushed
SNAPSHOT-master-3: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (push latest)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
+ docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:latest
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
78075328e0da: Layer already exists
20dd87a4c2ab: Layer already exists
9f8566ee5135: Layer already exists
0e1bd0a16495: Layer already exists
latest: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy to dev)
[Pipeline] input
deploy to dev?
Proceed or Abort
Approved by admin
[Pipeline] container
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] sh
+ envsubst
+ kubectl apply -f -
deployment.apps/ks-sample-dev configured
service/ks-sample-dev unchanged
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (push with tag)
[Pipeline] container
[Pipeline] {
[Pipeline] input
release image with tag?
Proceed or Abort
Approved by admin
[Pipeline] withCredentials
Masking supported pattern matches of $GIT_USERNAME or $GIT_PASSWORD
[Pipeline] {
[Pipeline] sh
+ git config --global user.email kubesphere@yunify.com
[Pipeline] sh
+ git config --global user.name kubesphere
[Pipeline] sh
+ git tag -a v1 -m v1
[Pipeline] sh
+ git push http://****:****@gitee.com/****/devops-maven-sample.git --tags --ipv4
remote: Powered by [01;33mGITEE.COM [0m[[01;35mGNK-6.3[0m][0m
To http://gitee.com/****/devops-maven-sample.git
* [new tag] v1 -> v1
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] sh
+ docker tag registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:SNAPSHOT-master-3 registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1
[Pipeline] sh
+ docker push registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample:v1
The push refers to repository [registry.cn-hangzhou.aliyuncs.com/qlq_repository/devops-maven-sample]
0e1bd0a16495: Preparing
20dd87a4c2ab: Preparing
78075328e0da: Preparing
9f8566ee5135: Preparing
9f8566ee5135: Layer already exists
0e1bd0a16495: Layer already exists
20dd87a4c2ab: Layer already exists
78075328e0da: Layer already exists
v1: digest: sha256:91d99fde2a9b9a4f15ead003c8446fb3532aeb7c3173ebaad09d69237eb22756 size: 1159
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy to production)
[Pipeline] input
deploy to production?
Proceed or Abort
Approved by admin
[Pipeline] container
[Pipeline] {
[Pipeline] withCredentials
Masking supported pattern matches of $KUBECONFIG
[Pipeline] {
[Pipeline] sh
+ envsubst
+ kubectl apply -f -
deployment.apps/ks-sample created
service/ks-sample created
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS
View Code
2》 kubernetes 环境查看相关的资源
[root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-devops-dev No resources found in kubesphere-devops-dev namespace. [root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-sample-dev NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ks-sample-dev 1/1 1 1 12h NAME READY STATUS RESTARTS AGE pod/ks-sample-dev-5dc9786dcc-6f9fx 1/1 Running 0 5m48s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ks-sample-dev NodePort 10.1.66.88 <none> 8080:30861/TCP 14h [root@k8smaster01 ~]# kubectl get deployments,pods,svc -n kubesphere-sample-prod NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/ks-sample 2/2 2 2 4m57s NAME READY STATUS RESTARTS AGE pod/ks-sample-68b68dfbc6-dpkwk 1/1 Running 0 4m57s pod/ks-sample-68b68dfbc6-qs2sw 1/1 Running 0 4m57s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/ks-sample NodePort 10.1.21.72 <none> 8080:30961/TCP 4m57s
3》gitee 查看打的tag 信息
4》阿里镜像仓库查看版本信息如下：
5》测试访问
xx@xx MINGW64 /d/study/devops-maven-sample (master)
$ curl http://192.168.13.107:30861
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2 100 2 0 0 166 0 --:--:-- --:--:-- --:--:-- 181v1
xx@xx MINGW64 /d/study/devops-maven-sample (master)
$ curl http://192.168.13.107:30961
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 2 100 2 0 0 68 0 --:--:-- --:--:-- --:--:-- 71v1
11. 总结
(0) 前置
其实通过kubesphere 流水线操作，最终还是都会到Jenkins中进行操作，其操作和自定义流水线一样。大致就是打代码、测试代码、打包、制作镜像并打tag、传送到镜像仓库、创建kubernetes 相关资源。在jenkins 执行过程中，会创建一个代理pod，比如：
[root@k8smaster01 ~]# kubectl get pods -A| grep maven kubesphere-devops-worker maven-cfqfb 2/2 Running 0 47s kubesphere-devops-worker maven-gzjdm 1/2 Error 0 18h
其中git 拉取代码、maven打包都是在下面pods 操作的。进入一个pod，查看其环境如下：
(1)Jenkins file文件查看分析
pipeline {
agent {
node {
label 'maven'
}
}
parameters {
string(name:'TAG_NAME',defaultValue: '',description:'')
}
environment {
DOCKER_CREDENTIAL_ID = 'ali-registery'
GITHUB_CREDENTIAL_ID = 'gitee-secret'
KUBECONFIG_CREDENTIAL_ID = 'demo-kubeconfig'
REGISTRY = 'registry.cn-hangzhou.aliyuncs.com'
DOCKERHUB_NAMESPACE = 'qlq_repository'
GITHUB_ACCOUNT = 'qiao-zhi'
APP_NAME = 'devops-maven-sample'
}
stages {
stage ('checkout scm') {
steps {
checkout(scm)
}
}
stage ('unit test') {
steps {
container ('maven') {
sh 'mvn clean test'
}
}
}
stage ('build & push') {
steps {
container ('maven') {
sh 'mvn clean package -DskipTests'
sh 'docker build -f Dockerfile-online -t $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER .'
withCredentials([usernamePassword(passwordVariable : 'DOCKER_PASSWORD' ,usernameVariable : 'DOCKER_USERNAME' ,credentialsId : "$DOCKER_CREDENTIAL_ID" ,)]) {
sh 'echo "$DOCKER_PASSWORD" | docker login $REGISTRY -u "$DOCKER_USERNAME" --password-stdin'
sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER'
}
}
}
}
stage('push latest'){
when{
branch 'master'
}
steps{
container ('maven') {
sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:latest '
}
}
}
stage('deploy to dev') {
when{
branch 'master'
}
steps {
input(id: 'deploy-to-dev', message: 'deploy to dev?')
container ('maven') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
variable: 'KUBECONFIG')
]) {
sh 'envsubst < deploy/dev-all-in-one/devops-sample.yaml | kubectl apply -f -'
}
}
}
}
stage('push with tag'){
when{
expression{
return params.TAG_NAME =~ /v.*/
}
}
steps {
container ('maven') {
input(id: 'release-image-with-tag', message: 'release image with tag?')
withCredentials([usernamePassword(credentialsId: "$GITHUB_CREDENTIAL_ID", passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
sh 'git config --global user.email "kubesphere@yunify.com" '
sh 'git config --global user.name "kubesphere" '
sh 'git tag -a $TAG_NAME -m "$TAG_NAME" '
sh 'git push http://$GIT_USERNAME:$GIT_PASSWORD@gitee.com/$GITHUB_ACCOUNT/$APP_NAME.git --tags --ipv4'
}
sh 'docker tag $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME '
sh 'docker push $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME '
}
}
}
stage('deploy to production') {
when{
expression{
return params.TAG_NAME =~ /v.*/
}
}
steps {
input(id: 'deploy-to-production', message: 'deploy to production?')
container ('maven') {
withCredentials([
kubeconfigFile(
credentialsId: env.KUBECONFIG_CREDENTIAL_ID,
variable: 'KUBECONFIG')
]) {
sh 'envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -'
}
}
}
}
}
}
前面定义了一些全局环境变量，按阶段进行分析
1》checkout scm 拉取代码
2》unit test 执行单元测试
3》build & push: 执行mvn打包， 然后基于Dockerfile-online 构造本地镜像，镜像的版本用代码分支和buildnumber 做区分，然后将镜像推送代码镜像仓库
4》push latest： 当代码分支是master 的时候，将3》的镜像打包为latest版本的镜像，然后推到镜像仓库
5》deploy to dev：当代码分支是master 的时候，相当于是执行 deploy/dev-all-in-one/devops-sample.yaml文件，相当于到kubernetes 创建资源
6》push with tag：当我们传递的参数是vxxx的时候，将代码打上相应的标签并推送到代码仓库；将镜像也打上相应的标签并推送到相应的镜像仓库
7》deploy to production：当我们传递的参数是vxxx的时候， 需要我们验证下是否需要将相关资源创建到kubernetes环境的prod 仓库，使用的文件是deploy/prod-all-in-one/devops-sample.yaml。
envsubst < deploy/prod-all-in-one/devops-sample.yaml | kubectl apply -f -
envsubst 是用shell格式字符串中的值替换环境变量。要替换的变量应位于${var}或$var格式。例如
[root@k8smaster01 ~]# export MYKEY=123456 [root@k8smaster01 ~]# echo $MYKEY 123456 [root@k8smaster01 ~]# cat test.txt $MYKEY [root@k8smaster01 ~]# envsubst < test.txt 123456
(2) 几个重要kubernetes 文件如下：
deploy/dev-all-in-one/devops-sample.yaml：
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kubesphere
component: ks-sample-dev
tier: backend
name: ks-sample-dev
namespace: kubesphere-sample-dev
spec:
progressDeadlineSeconds: 600
replicas: 1
selector:
matchLabels:
app: kubesphere
component: ks-sample-dev
tier: backend
template:
metadata:
labels:
app: kubesphere
component: ks-sample-dev
tier: backend
spec:
containers:
- env:
- name: CACHE_IGNORE
value: js|html
- name: CACHE_PUBLIC_EXPIRATION
value: 3d
image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER
readinessProbe:
httpGet:
path: /
port: 8080
timeoutSeconds: 10
failureThreshold: 30
periodSeconds: 5
imagePullPolicy: Always
name: ks-sample
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: 300m
memory: 600Mi
requests:
cpu: 100m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kubesphere
component: ks-sample-dev
name: ks-sample-dev
namespace: kubesphere-sample-dev
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
nodePort: 30861
selector:
app: kubesphere
component: ks-sample-dev
tier: backend
sessionAffinity: None
type: NodePort
View Code
deploy/prod-all-in-one/devops-sample.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kubesphere
component: ks-sample
tier: backend
name: ks-sample
namespace: kubesphere-sample-prod
spec:
progressDeadlineSeconds: 600
replicas: 2
selector:
matchLabels:
app: kubesphere
component: ks-sample
tier: backend
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 100%
type: RollingUpdate
template:
metadata:
labels:
app: kubesphere
component: ks-sample
tier: backend
spec:
containers:
- env:
- name: CACHE_IGNORE
value: js|html
- name: CACHE_PUBLIC_EXPIRATION
value: 3d
image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME
readinessProbe:
httpGet:
path: /
port: 8080
timeoutSeconds: 10
failureThreshold: 30
periodSeconds: 5
imagePullPolicy: Always
name: ks
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: 300m
memory: 600Mi
requests:
cpu: 100m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kubesphere
component: ks-sample
name: ks-sample
namespace: kubesphere-sample-prod
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
nodePort: 30961
selector:
app: kubesphere
component: ks-sample
tier: backend
sessionAffinity: None
type: NodePort
View Code
代码仓库和修改后配置文件：https://gitee.com/Qiao-Zhi/devops-maven-sample
补充： 在安装过程中遇到一些磁盘不足，报错信息如下:
Warning Evicted 49m kubelet The node had condition: [DiskPressure].
解决参考：
https://blog.51cto.com/riverxyz/2758421
vm中对centos 扩容: https://ld246.com/article/1566021346577
补充： 如果是阿里私有镜像仓库，需要创建secret 然后拉取，如下
1. 创建私有镜像仓库: 账号密码记得替换
kubectl create secret docker-registry registry-aliyun --namespace=kubesphere-sample-dev \
--docker-server=registry.cn-hangzhou.aliyuncs.com --docker-username=usernamre \
--docker-password=pwd --docker-email=xxx@163.com
2. 修改yaml文件，增加镜像拉取secret
(1) deploy/dev-all-in-one/devops-sample.yaml：
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kubesphere
component: ks-sample-dev
tier: backend
name: ks-sample-dev
namespace: kubesphere-sample-dev
spec:
progressDeadlineSeconds: 600
replicas: 1
selector:
matchLabels:
app: kubesphere
component: ks-sample-dev
tier: backend
template:
metadata:
labels:
app: kubesphere
component: ks-sample-dev
tier: backend
spec:
containers:
- env:
- name: CACHE_IGNORE
value: js|html
- name: CACHE_PUBLIC_EXPIRATION
value: 3d
image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:SNAPSHOT-$BRANCH_NAME-$BUILD_NUMBER
readinessProbe:
httpGet:
path: /
port: 8080
timeoutSeconds: 10
failureThreshold: 30
periodSeconds: 5
imagePullPolicy: Always
name: ks-sample
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: 300m
memory: 600Mi
requests:
cpu: 100m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullSecrets:
- name: registry-aliyun
dnsPolicy: ClusterFirst
restartPolicy: Always
terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kubesphere
component: ks-sample-dev
name: ks-sample-dev
namespace: kubesphere-sample-dev
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
nodePort: 30861
selector:
app: kubesphere
component: ks-sample-dev
tier: backend
sessionAffinity: None
type: NodePort
View Code
(2) deploy/prod-all-in-one/devops-sample.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: kubesphere
component: ks-sample
tier: backend
name: ks-sample
namespace: kubesphere-sample-prod
spec:
progressDeadlineSeconds: 600
replicas: 2
selector:
matchLabels:
app: kubesphere
component: ks-sample
tier: backend
strategy:
rollingUpdate:
maxSurge: 100%
maxUnavailable: 100%
type: RollingUpdate
template:
metadata:
labels:
app: kubesphere
component: ks-sample
tier: backend
spec:
containers:
- env:
- name: CACHE_IGNORE
value: js|html
- name: CACHE_PUBLIC_EXPIRATION
value: 3d
image: $REGISTRY/$DOCKERHUB_NAMESPACE/$APP_NAME:$TAG_NAME
readinessProbe:
httpGet:
path: /
port: 8080
timeoutSeconds: 10
failureThreshold: 30
periodSeconds: 5
imagePullPolicy: Always
name: ks
ports:
- containerPort: 8080
protocol: TCP
resources:
limits:
cpu: 300m
memory: 600Mi
requests:
cpu: 100m
memory: 100Mi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
imagePullSecrets:
- name: registry-aliyun
dnsPolicy: ClusterFirst
restartPolicy: Always
terminationGracePeriodSeconds: 30
---
apiVersion: v1
kind: Service
metadata:
labels:
app: kubesphere
component: ks-sample
name: ks-sample
namespace: kubesphere-sample-prod
spec:
ports:
- name: http
port: 8080
protocol: TCP
targetPort: 8080
nodePort: 30961
selector:
app: kubesphere
component: ks-sample
tier: backend
sessionAffinity: None
type: NodePort
View Code
标签：INFO,maven,Pipeline,git,kubesphere,devops,sample,安装 来源： https://www.cnblogs.com/qlqwjy/p/15965008.html