其他分享
首页 > 其他分享> > 极狐GitLab整合K8S实现GitOps与CI/CD(Agent方式)

极狐GitLab整合K8S实现GitOps与CI/CD(Agent方式)

作者:互联网

极狐GitLab整合K8S实现GitOps与CI/CD(Agent方式)

1. 简介

参考文档:

极狐GitLab Kubernetes Agent 是用安全和云原生方式实现极狐GitLab 与 Kubernetes 集成的组件,不同于老版本使用证书连接 Kubernetes 集群的方式。从 v14 版本开始,极狐gitlab 将逐渐废除使用证书连接 Kubernetes 集群的方式。

它实现的功能:

它包含两部分:

要基于 Agent 执行 GitOps 部署,需要满足下列条件:

注意:Kubernetes Agent 在 14.5 之前只支持 ee 以上收费版本,14.5 版本后虽然在 免费版本能够看到此功能,但是添加后使用 gitops 还是会出现错误:project not found,原因是清单文件的同步不是免费的、是专业版以上才能用,本教程使用极狐gitlab v14.7.3

2. 主要功能简介

本部分只是介绍 Agent 使用场景,没有实际操作步骤;

2.1 GitOps workflow

2.1.1 GitOps 简介

这里采用 RedHat 对 GitOps 的定义

2.1.2 演示环境

2.1.3 演示效果

在该项目仓库里编写任意 k8s 对象的描述文件 (.yaml 或者 .json 格式),都能实时部署到 k8s 集群中,无需执行 ci/cd ;

2.2 CI/CD workflow

2.2.1 演示环境

在上述仓库中增加 ci/cd 配置文件:.gitlab-ci.yml

2.2.2 演示效果

3. 启用 Kubernetes Agent

3.1 Omnibus 方式

首先,修改 /etc/gitlab/gitlab.rb 配置

gitlab_kas['enable'] = true

然后运行生效:gitlab-ctl reconfigure

3.2 helm chart 方式

helm upgrade --install gitlab gitlab/gitlab \
  --set global.kas.enabled=true

4. GitOps 演示

4.1 创建仓库

gitops/agentk

4.2 创建配置文件

gitops/agentk 库中创建:.gitlab/agents/agent1/config.yaml

gitops:
  manifest_projects:
  - id: "gitops/agentk"
    paths:
    - glob: '/**/*.{yaml,json}'

4.3 创建资源清单文件

定义 k8s 资源描述文件,例如:deploy/nginx.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: test-agent
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  namespace: test-agent
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 3
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.14.2
        ports:
        - containerPort: 80

4.4 安装 Agent

操作步骤:

agent01

image-20220228121426324

**注意:如果安装命令的 --kas-address= 的地址结尾缺少斜线 /,必须手动添加后再执行,否则 agent 安装后无法连接到 kas **

$ docker run --pull=always --rm \
    registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate \
    --agent-token=ab7HZQiAzBuV2n-7cZoVeEi6jByjHRMzFQu1c_Mx_s8TQGWCJA \
    --kas-address=wss://gitlab.example.com/-/kubernetes-agent/ \
    --agent-version stable \
    --namespace gitlab-kubernetes-agent  > resources.yaml

# resources.yaml 就是 agent 的资源定义文件

$ kubectl apply -f resources.yaml

$ kubectl get pod -A|grep gitlab-agent
gitlab-kubernetes-agent   gitlab-agent-77475dcdd8-4lhcr             1/1     Running     0          39s

web 界面查看 agent 并未正常连接到极狐gitlab

image-20220228122523276

查看日志

$ kubectl -n gitlab-kubernetes-agent logs -f gitlab-agent-77475dcdd8-4lhcr
...
{"level":"error","time":"2022-02-28T04:25:38.901Z","msg":"Error handling a connection","mod_name":"reverse_tunnel","error":"Connect(): rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing failed to WebSocket dial: failed to send handshake request: Get \\\"https://gitlab.example.com/-/kubernetes-agent/\\\": dial tcp: lookup gitlab.example.com on 10.43.0.10:53: no such host\""}

原因是 pod 无法解析 gitlab.example.com,解决方法如下:

然后在集群 coredns 中添加 hosts 解析

$ kubectl -n kube-system edit configmaps coredns
...
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        hosts /etc/coredns/NodeHosts {
          10.10.10.60 gitlab.example.com
          ttl 60    
          reload 15s 
          fallthrough
        }               
...

# 删除 coredns pod 生效
$ kubectl get pod -n kube-system
NAME                                      READY   STATUS      RESTARTS   AGE
helm-install-traefik-g4xsp                0/1     Completed   0          151m
metrics-server-86cbb8457f-8wghh           1/1     Running     0          151m
local-path-provisioner-5ff76fc89d-r87zg   1/1     Running     0          151m
coredns-854c77959c-nm28h                  1/1     Running     0          151m

$ kubectl -n kube-system delete pod coredns-854c77959c-nm28h
pod "coredns-854c77959c-nm28h" deleted

删除旧 pod ,重建新 pod 生效

kubectl -n gitlab-kubernetes-agent delete pod gitlab-agent-77475dcdd8-4lhcr

查看 pod 日志还是报错:

x509: certificate signed by unknown authority

原因是 ssl 证书是自签的,查看 resources.yaml 发现是使用的 registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:stable 运行 agent,查看帮助,发现使用 --ca-cert-file 可以解决

$ docker run --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:stable --help
GitLab Kubernetes Agent

Usage:
  agentk [flags]

Flags:
      --as string                      Username to impersonate for the operation. User could be a regular user or a service account in a namespace.
      --as-group stringArray           Group to impersonate for the operation, this flag can be repeated to specify multiple groups.
      --as-uid string                  UID to impersonate for the operation.
      --ca-cert-file string            Optional file with X.509 certificate authority certificate in PEM format
      --cache-dir string               Default cache directory (default "/home/nonroot/.kube/cache")
      --certificate-authority string   Path to a cert file for the certificate authority
      --client-certificate string      Path to a client certificate file for TLS
      --client-key string              Path to a client key file for TLS
      --cluster string                 The name of the kubeconfig cluster to use
      --context string                 The name of the kubeconfig context to use
  -h, --help                           help for agentk
      --insecure-skip-tls-verify       If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
      --kas-address string             GitLab Kubernetes Agent Server address
      --kubeconfig string              Path to the kubeconfig file to use for CLI requests.
  -n, --namespace string               If present, the namespace scope for this CLI request
      --request-timeout string         The length of time to wait before giving up on a single server request. Non-zero values should contain a corresponding time unit (e.g. 1s, 2m, 3h). A value of zero means don't timeout requests. (default "0")
  -s, --server string                  The address and port of the Kubernetes API server
      --tls-server-name string         Server name to use for server certificate validation. If it is not provided, the hostname used to contact the server is used
      --token string                   Bearer token for authentication to the API server
      --token-file string              File with access token
      --user string                    The name of the kubeconfig user to use
  -v, --version                        version for agentk
# 删除资源
kubectl delete -f resources.yaml

# 创建 gitlab.example.com 的证书 secret
kubectl create namespace gitlab-kubernetes-agent
kubectl -n gitlab-kubernetes-agent create secret generic gitlab.example.com.crt --from-file=gitlab.example.com.crt
# 也可以是签发 gitlab.example.com.crt 的根证书 ca.pem

编辑 resources.yaml 添加 secret 挂载

$ vi resources.yaml
...
      - args:
        - --ca-cert-file=/cafile/gitlab.example.com.crt
        - --token-file=/config/token
        - --kas-address
        - wss://gitlab.example.com/-/kubernetes-agent/
...
        volumeMounts:
        - mountPath: /cafile
          name: ca-file
...
      volumes:
      - name: ca-file
        secret:
          secretName: gitlab.example.com.crt
...

重新应用即可,pod 日志未看到报错

$ kubectl apply -f resources.yaml
$ kubectl get pod -A|grep gitlab-kubernetes-agent
gitlab-kubernetes-agent   gitlab-agent-7f7978db94-rfhql             1/1     Running     0          59s

但是在极狐gitlab web 界面上还是可以看到未正常连接

image-20220228122523276

查看 gitlab 主机上的 kas 服务侧日志发现一样证书报错,kas 无法正常连接极狐gitlab 实例

$ tail /var/log/gitlab/gitlab-kas/current
...
2022-02-28_06:27:25.40431 {"level":"error","time":"2022-02-28T14:27:25.404+0800","msg":"AgentInfo()","correlation_id":"01FWZFTJJTQCCG3SNHEPZ09DPW","grpc_service":"gitlab.agent.agent_configuration.rpc.AgentConfiguration","grpc_method":"GetConfiguration","error":"Get \"https://gitlab.example.com/api/v4/internal/kubernetes/agent_info\": x509: certificate signed by unknown authority"}

解决方法暂时只有极狐gitlab 使用有效机构颁发的有效证书即可。

image-20220228141343963

正常的话,这时候 k8s 集群会根据 deploy/nginx.yaml 创建相关资源

4.5 删除 Agent

可以在 web 界面删除(14.7 后才添加的功能)或者使用 graphql 删除,具体参考:Working with the agent for Kubernetes | GitLab

5. CI/CD 演示

还是在 gitops/agentk (已安装 agent )基础上进行

5.1 添加仓库

gitops/p1

5.2 添加 agent 认证

如果就是当前项目下(gitops/agentk)使用 agent,则不需要添加认证

编辑 gitops/agentk 项目 .gitlab/agents/agent1/config.yaml 添加

ci_access:
  projects:
  - id: gitops/p1

也可以添加组

ci_access:
  projects:
  - id: gitops/g1
  - id: gitops/g2

5.3 添加 .gitlab-ci.yml

gitops/p1 下添加

stages:
  - build

build:
  stage: build
  image:
    name: bitnami/kubectl:1.20.7
    entrypoint: [""]
  script:
  - kubectl config get-contexts
  - kubectl config use-context gitops/agentk:agent1
  - kubectl get pod -A

如果不添加前面的认证任务会报错:

$ kubectl config use-context gitops/agentk:agent1
error: no context exists with the name: "gitops/agentk:agent1"
ERROR: Job failed: command terminated with exit code 1

运行 job 发现报错:

...
$ kubectl config get-contexts
CURRENT   NAME           CLUSTER   AUTHINFO   NAMESPACE
          g2/p2:agent1   gitlab    agent:5    
$ kubectl config use-context g2/p2:agent1
Switched to context "g2/p2:agent1".
$ kubectl get pod
error: You must be logged in to the server (the server has asked for the client to provide credentials)
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: command terminated with exit code 1

根据官方文档:Using GitLab CI/CD with a Kubernetes cluster | GitLab 的说法,是因为自建的狐gitlab 未启用 https 的原因。

这就有个矛盾的地方了,如果使用自签的 https 证书,无法正常完成 kas 的搭建,不使用 https 又无法在 CI/CD workflow 中连接 k8s 集群,唯一的解决方案自有去阿里云或者腾讯云购买一个域名,然后申请1年免费的ssl证书再来尝试搭建了。

更换有效证书搭建极狐gitlab 后,job 成功运行:

...
Running on runner-5hmzywb8-project-4-concurrent-0t5n99 via gitlab-runner-gitlab-runner-7c99894b66-v6fdn...
Getting source from Git repository
00:00
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/gitops/p1/.git/
Created fresh repository.
Checking out 044a80aa as main...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:01
$ kubectl config get-contexts
CURRENT   NAME                   CLUSTER   AUTHINFO   NAMESPACE
          gitops/agentk:agent1   gitlab    agent:1    
$ kubectl config use-context gitops/agentk:agent1
Switched to context "gitops/agentk:agent1".
$ kubectl get pod -A
NAMESPACE                 NAME                                           READY   STATUS      RESTARTS   AGE
kube-system               helm-install-traefik-2xwdx                     0/1     Completed   0          21m
kube-system               svclb-traefik-wl6cb                            2/2     Running     0          21m
kube-system               local-path-provisioner-5ff76fc89d-sppxw        1/1     Running     0          21m
kube-system               traefik-6f9cbd9bd4-46lwd                       1/1     Running     0          21m
kube-system               metrics-server-86cbb8457f-ps94b                1/1     Running     0          21m
kube-system               coredns-854c77959c-jhq64                       1/1     Running     0          18m
default                   gitlab-runner-gitlab-runner-7c99894b66-v6fdn   1/1     Running     0          17m
gitlab-kubernetes-agent   gitlab-agent-79d47cbb87-rm9zr                  1/1     Running     0          9m53s
default                   runner-5hmzywb8-project-4-concurrent-0t5n99    2/2     Running     0          53s
Cleaning up project directory and file based variables
00:01
Job succeeded

标签:kubectl,GitOps,gitlab,--,GitLab,agentk,CI,agent,gitops
来源: https://www.cnblogs.com/leffss/p/16575428.html