其他分享
首页 > 其他分享> > 极狐gitlab集成 SonarQube 7.6

极狐gitlab集成 SonarQube 7.6

作者:互联网

极狐gitlab集成 sonarqube 7.6

场景:

这里,我们引进非常流行的扫描工具 sonarqube 社区版为例加以说明。

环境介绍:

1. 部署 sonarqube ce

采用 docker 方式部署

1.1 安装 postgresql

创建目录

mkdir -p /home/sonar/postgres/postgresql
mkdir -p /home/sonar/postgres/data

创建网络

docker network create sonarqube-network

部署 pg

docker run --name postgres -d -p 5432:5432 --network sonarqube-network \
-v /home/sonar/postgres/postgresql:/var/lib/postgresql \
-v /home/sonar/postgres/data:/var/lib/postgresql/data \
-v /etc/localtime:/etc/localtime:ro \
-e POSTGRES_USER=sonar \
-e POSTGRES_PASSWORD=sonar \
-e POSTGRES_DB=sonar \
-e TZ=Asia/Shanghai \
--restart always \
--privileged=true \
--network-alias postgres \
postgres:14.2

1.2 安装 sonarqube

创建工作目录

mkdir -p /data/sonarqube_dir

修改系统参数

echo "vm.max_map_count=262144" >> /etc/sysctl.conf
sysctl -p

运行测试容器

docker run -d --name sonartest sonarqube:7.6-community

拷贝必须文件到本地,并修改权限为777

docker cp sonartest:/opt/sonarqube/conf /data/sonarqube_dir
docker cp sonartest:/opt/sonarqube/data /data/sonarqube_dir
docker cp sonartest:/opt/sonarqube/logs /data/sonarqube_dir
docker cp sonartest:/opt/sonarqube/extensions /data/sonarqube_dir

chmod -R 777 /data/sonarqube_dir/

删除容器

docker stop sonartest
docker rm sonartest

启动 SonarQube,其中SONARQUBE_JDBC_URL需要修改为实际 Postgresql 数据库的 IP

docker run -d --name sonar -p 9000:9000 \
-e ALLOW_EMPTY_PASSWORD=yes \
-e SONARQUBE_DATABASE_USER=sonar \
-e SONARQUBE_DATABASE_NAME=sonar \
-e SONARQUBE_DATABASE_PASSWORD=sonar \
-e SONARQUBE_JDBC_URL="jdbc:postgresql://<postgresql_server_ip>:5432/sonar" \
--privileged=true \
--network sonarqube-network \
--restart always \
-v /data/sonarqube_dir/logs:/opt/sonarqube/logs \
-v /data/sonarqube_dir/conf:/opt/sonarqube/conf \
-v /data/sonarqube_dir/data:/opt/sonarqube/data \
-v /data/sonarqube_dir/extensions:/opt/sonarqube/extensions \
sonarqube:7.6-community

新版本可以 docker-compose 一键部署,docker-compose.yml 参考

version: "3.0"
services:
  postgresql:
    image: postgres:14.2
    restart: always
    container_name: postgres
    networks:
      - sonarnet
    environment:
      POSTGRES_DB: sonar
      POSTGRES_USER: sonar
      POSTGRES_PASSWORD: sonar
      TZ: Asia/Shanghai
    volumes:
      - /data/sonarqube/postgres/postgresql:/var/lib/postgresql
      - /data/sonarqube/postgres/postgresql_data:/var/lib/postgresql/data
      - /etc/localtime:/etc/localtime:ro

  sonarqube:
    image: sonarqube:9.4-community
    container_name: sonar
    depends_on:
      - postgresql
    ports:
      - "9000:9000"
    networks:
      - sonarnet
    environment:
      TZ: Asia/Shanghai
      SONARQUBE_JDBC_USERNAME: sonar
      SONARQUBE_JDBC_PASSWORD: sonar
      SONARQUBE_DATABASE_NAME: sonar
      SONARQUBE_JDBC_URL: jdbc:postgresql://postgresql:5432/sonar
    volumes:
      - /data/sonarqube/conf:/opt/sonarqube/conf
      - /data/sonarqube/data:/opt/sonarqube/data
      - /data/sonarqube/log:/opt/sonarqube/log
      - /data/sonarqube/extensions:/opt/sonarqube/extensions

networks:
  sonarnet:
    driver: bridge

1.3 生成 token

登陆地址:http://<sonarqube_server_ip>:9000

默认账号:admin/admin

设置必须登陆后才能查看信息:

登陆 -- Administration -- Security -- 开启 Force user authentication -- save

生成 token:

admin 登陆后点击右上角 My Account - Security 中生成Token

2. 扫描器镜像制作

2.1 下载 sonar-scanner

参考官方:SonarScanner | SonarQube Docs

下载:

curl -O https://binaries.sonarsource.com/Distribution/sonar-scanner-cli/sonar-scanner-cli-4.7.0.2747-linux.zip
unzip sonar-scanner-cli-4.7.0.2747-linux.zip

简单使用方法:

./sonar-scanner \
-Dsonar.host.url=http://<sonarqube_server_ip>:9000 \
-Dsonar.login=333f3410ce3e575d559329e8f3d0a5d4ec8a499d \
-Dsonar.projectKey=my:test \
-Dsonar.sources=/path_to_codes

如果想要在本地生成json格式报告,则增加如下参数

-Dsonar.report.export.path=report.json \
-Dsonar.analysis.mode=preview

2.2 json 格式转换

sonar-scanner 默认生成 json 格式:

{
    "version":"7.6.0.21501",
    "issues":[
        {
            "key":"01803B75AC297CF54F",
            "component":"my:test:main.py",
            "line":94,
            "startLine":94,
            "startOffset":4,
            "endLine":94,
            "endOffset":52,
            "message":"Remove this commented out code.",
            "severity":"MAJOR",
            "rule":"python:S125",
            "status":"OPEN",
            "isNew":true
        },
        {
            "key":"01803B75AC297CF550",
            "component":"my:test:main.py",
            "line":28,
            "startLine":28,
            "startOffset":4,
            "endLine":28,
            "endOffset":13,
            "message":"Rename this local variable \"startLine\" to match the regular expression ^[_a-z][a-z0-9_]*$.",
            "severity":"MINOR",
            "rule":"python:S117",
            "status":"OPEN",
            "isNew":true
        },
        {
            "key":"01803B75AC297CF551",
            "component":"my:test:main.py",
            "line":29,
            "startLine":29,
            "startOffset":4,
            "endLine":29,
            "endOffset":11,
            "message":"Rename this local variable \"endLine\" to match the regular expression ^[_a-z][a-z0-9_]*$.",
            "severity":"MINOR",
            "rule":"python:S117",
            "status":"OPEN",
            "isNew":true
        }
    ],
    "components":[
        {
            "key":"my:test"
        },
        {
            "key":"my:test:main.py",
            "path":"main.py",
            "status":"ADDED"
        }
    ],
    "rules":[
        {
            "key":"python:S125",
            "rule":"S125",
            "repository":"python",
            "name":"Sections of code should not be commented out"
        },
        {
            "key":"python:S117",
            "rule":"S117",
            "repository":"python",
            "name":"Local variable and function parameter names should comply with a naming convention"
        }
    ],
    "users":[

    ]
}

而 gitlab 报告 json 格式:

{
  "category": "test",
  "message": "这个问题不怎么严重",
  "cve": "python-webhook/MicroService/Service.py:960662f9bd521d32692b07bd8d5b10538924c23c37cec891847f40e436c5c2f:B104",
  "severity": "Medium",
  "confidence": "Medium",
  "scanner": {
    "id": "test",
    "name": "test"
  },
  "location": {
    "file": "python-webhook/MicroService/Service.py",
    "start_line": 26,
    "end_line": 28
  },
  "identifiers": [
    {
      "type": "bandit_test_id",
      "name": "Bandit Test ID B104",
      "value": "B104",
      "url": "https://bandit.readthedocs.io/en/latest/plugins/b104_hardcoded_bind_all_interfaces.htl"
    }
  ]
}

2.2.1 转换器 converter.py

# coding=utf-8
from datetime import datetime
import json
import hashlib


# {u'INFO': 50, u'BLOCKER': 3, u'MAJOR': 5724, u'CRITICAL': 1089, u'MINOR': 1103}
severity_mapper = {
    'INFO': 'Info',
    'BLOCKER': 'Unknown',
    'MAJOR': 'High',
    'CRITICAL': 'Critical',
    'MINOR': 'Low'
}

# gitlab values ["Ignore", "Unknown", "Experimental", "Low", "Medium", "High", "Confirmed"]
confidence_mapper = {
    'INFO': 'Ignore',
    'BLOCKER': 'Unknown',
    'MAJOR': 'High',
    'CRITICAL': 'Confirmed',
    'MINOR': 'Low'
}


def conv(issue):
    component = issue.get('component')
    start_line = issue.get('startLine')
    end_line = issue.get('endLine')
    message = issue.get('message')
    severity = issue.get('severity')
    rule = issue.get('rule')
    ret = {
        'category': 'sast',
        'message': message,
        'cve': '',
        'severity': severity_mapper.get(severity, 'Unknown'),
        'confidence': confidence_mapper.get(severity, 'Unknown'),
        'scanner': {
            'id': 'sonarqube',
            'name': 'sonarqube'
        },
        'location': {
            'file': component.split(':')[-1],
            'start_line': start_line,
            'end_line': end_line
        },
        'identifiers': [
            {
                'type': rule,
                'name': rule,
                'value': rule,
                # 'url': ''
            }
        ]
    }
    hash_id = hashlib.sha256(json.dumps(ret, sort_keys=True).encode('utf-8')).hexdigest()
    ret['id'] = hash_id
    return ret


def sonarqube2gitlab(source_file, destination_file):
    datetime_obj = datetime.now()
    time_str = datetime_obj.strftime('%Y-%m-%dT%H:%M:%S')
    gitlab_sast_report = {
        'version': '3.0.0',
        'vulnerabilities': [],
        'remediations': [],
        'scan': {
            'scanner': {
                'id': 'sonarqube',
                'name': 'SonarQube',
                'url': 'https://docs.sonarqube.org/',
                'vendor': {
                    "name": 'GitLab'
                },
                'version': '1.7.0'
            },
            'type': 'sast',
            'start_time': time_str,
            'end_time': time_str,
            'status': 'success'
        }
    }

    with open(source_file, 'r') as f:
        report = json.loads(f.read())
    issues = report.get('issues', list())

    for issue in issues:
        issue_gitlab = conv(issue)
        gitlab_sast_report['vulnerabilities'].append(issue_gitlab)

    with open(destination_file, 'w') as gitlab_sast_report_file:
        gitlab_sast_report_file.write(json.dumps(gitlab_sast_report, indent=4, sort_keys=True))


if __name__ == '__main__':
    sonarqube2gitlab('.scannerwork/report.json', 'gl-sast-report.json')

2.3 打包镜像

新建 scan.sh

/sonar-scanner/bin/sonar-scanner -Dsonar.host.url=$sonar_host_url -Dsonar.login=$sonar_login -Dsonar.report.export.path=report.json -Dsonar.analysis.mode=preview
python /sonar-scanner/converter.py

Dockerfile

from python:3.9.12-slim
workdir /sonar-scanner
copy sonar-scanner-4.7.0.2747-linux /sonar-scanner
add converter.py /sonar-scanner
add scan.sh /sonar-scanner
ENV LANG C.UTF-8
ENV TZ='Asia/Shanghai'
RUN echo 'Asia/Shanghai' > /etc/timezone

构建

docker build -t mysonarscanner:4.7 .

3. 扫描测试

3.1 创建 python 代码

创建测试项目,添加代码 main.py:

# coding=utf-8
# Copyright 2022 Xuefeng Yin, All Rights Reserved

from datetime import datetime
import json
import hashlib

f = open(".scannerwork/report.json", "r")

report = json.loads(f.read())
issues = report.get("issues")


# {u'INFO': 50, u'BLOCKER': 3, u'MAJOR': 5724, u'CRITICAL': 1089, u'MINOR': 1103}
severitys_mapper = {
    "INFO": "info",
    "BLOCKER":"Unknown",
    "MAJOR":"High",
    "CRITICAL":"Critical",
    "MINOR":"Low",
}


def conv(issue):
    component = issue.get("component")
    startLine = issue.get("startLine")
    endLine = issue.get("endLine")
    message = issue.get("message")
    severity = issue.get("severity")
    rule = issue.get("rule")

    ret = {
        "category": "sast",
        "message": message,
        "cve": "",
        "severity": severitys_mapper.get(severity, "Unknown"),
        "confidence": severitys_mapper.get(severity, "Unknown"),
        "scanner": {
            "id": "sonarqube",
            "name": "sonarqube"
        },
        "location": {
            "file": component.split(":")[-1],
            "start_line": startLine,
            "end_line": endLine
        },
        "identifiers": [
            {
                "type": rule,
                "name": rule,
                "value": rule,
                "url": ""
            }
        ]
    }

    id = hashlib.sha256(json.dumps(ret, sort_keys=True)).hexdigest()
    ret["id"] = id

    return ret


dateTimeObj = datetime.now()
timeStr = dateTimeObj.strftime("%Y-%m-%dT%H:%M:%S")

gl_sast_report = {
  "version": "3.0.0",
  "vulnerabilities": [],
  "remediations": [],
  "scan": {
    "scanner": {
      "id": "sonarqube",
      "name": "SonarQube",
      "url": "https://docs.sonarqube.org/",
      "vendor": {
        "name": "GitLab"
      },
      "version": "1.7.0"
    },
    "type": "sast",
    "start_time": timeStr,
    "end_time": timeStr,
    "status": "success"
  }
}


for i, issue in enumerate(issues[:]):
    #print("Issue No. %s ---------------------" % i)
    #print("SonarQube: %s" % issue)
    issue_gitlab = conv(issue)
    #print("GitLab: %s" % issue_gitlab)
    gl_sast_report["vulnerabilities"].append(issue_gitlab)


gl_sast_report_file = open("gl-sast-report.json", "w")
gl_sast_report_file.write(json.dumps(gl_sast_report, indent=4, sort_keys=True))
gl_sast_report_file.close()

3.2 添加 .gitlab-ci.yml

variables:
  sonar_host_url: http://<sonarqube_server_ip>:9000
  sonar_login: <sonarqube_server_token>

sonarqube:
  image: mysonarscanner:4.7
  script:
    # 生成扫描参数
    - echo -e "sonar.sourceEncoding=UTF-8\nsonar.projectKey=$CI_PROJECT_NAME\nsonar.sources=." >> sonar-project.properties
    # 开始扫描并转换 json 格式
    - sh /sonar-scanner/scan.sh
  artifacts:
    reports:
      sast:
        - gl-sast-report.json

扫描结果:

如果将检查设置为合并时执行,还可以将结果显示到 GitLab 合并请求小部件中:

.gitlab-ci.yml

variables:
  sonar_host_url: http://<sonarqube_server_ip>:9000
  sonar_login: <sonarqube_server_token>

sonarqube:
  image: mysonarscanner:4.7
  script:
    - echo "sonar.sourceEncoding=UTF-8\nsonar.projectKey=$CI_PROJECT_NAME\nsonar.sources=." >> sonar-project.properties
    - sh /sonar-scanner/scan.sh
  artifacts:
    reports:
      sast:
        - gl-sast-report.json
  only:
    - merge_requests

扫描结果:

4. Findbugs 支持

sonarqube 7.6 最高支持 4.0.0 版本:SpotBugs. 4.0.0, sb-contrib 7.4.7, and findsecbugs 1.10.1,2020.2.28 发布;

最新版本 4.1.4:SpotBugs. 4.6.0, sb-contrib 7.4.7, and findsecbugs 1.12.0,2022.4.28 发布;

4.1 安装插件

下载插件:

wget https://github.com/spotbugs/sonar-findbugs/releases/download/4.0.0/sonar-findbugs-plugin-4.0.0.jar

复制插件到 sonarqube server 插件目录

cp sonar-findbugs-plugin-4.0.0.jar /data/sonarqube_dir/extensions/plugins
chmod 777 /data/sonarqube_dir/extensions/plugins/sonar-findbugs-plugin-4.0.0.jar

重启 sonarqube server 生效

docker restart sonar

4.2 启用插件

查看插件

这里设置 FindBugs + FB-Contrib 为默认,也可根据需求选择其他配置

4.3 测试 maven 项目

下载测试项目 push 到 gitlab 示例:https://jihulab.com/ffli/simple-java-maven-app

添加 .gitlab-ci.yml

stages:
  - compile
  - scan
  - package
  - test

variables:
  sonar_host_url: http://<sonar_server_ip>:9000
  sonar_login: <sonar_server_token>

# java 项目需要编译后才能进行扫描
# 这里使用 maven 编译后,然后使用
# gitlab 的 artifacts 功能将结果
# 传递给下一个 sonarqube 任务扫描
compile:
  image: maven:3.8.5-jdk-11
  stage: compile
  variables:
    MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
    MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
  # 开启 job cache 缓存,可以极大的提高第 2 次以后的编译速度
  cache:
    paths:
      - .m2/repository/
      - target/
    key: $CI_PROJECT_NAME
  script:
    # - mvn package
    - mvn compile
  artifacts:
    paths:
      - target/
  only:
    - merge_requests

# sonarqube 扫描任务
sonarqube:
  image: mysonarscanner:4.7
  stage: scan
  script:
    # 生成扫描参数,这里的 sonar.sources 设置源码目录,可以设置多个,sonar.java.binaries 设置编译后目录    
    - echo "sonar.sourceEncoding=UTF-8" >> sonar-project.properties
    - echo "sonar.projectKey=$CI_PROJECT_NAME" >> sonar-project.properties
    - echo "sonar.sources=./src" >> sonar-project.properties
    - echo "sonar.java.binaries=./target" >> sonar-project.properties
    - echo "sonar.language=java" >> sonar-project.properties
    - echo "sonar.exclusions=**/*.js" >> sonar-project.properties
    # 开始扫描并转换 json 格式
    - sh /sonar-scanner/scan.sh
  artifacts:
    reports:
      sast:
        - gl-sast-report.json
  only:
    - merge_requests

# 打包
package:
  image: maven:3.8.5-jdk-11
  stage: package
  variables:
    MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
    MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
  cache:
    paths:
      - .m2/repository/
      - target/
    key: $CI_PROJECT_NAME
  script:
    - mvn -B -DskipTests clean package
  artifacts:
    paths:
      - target/
  only:
    - merge_requests

# 测试,并使用 gitlab artifacts 功能将 junit 测试结果显示到 gitlab web 页面上
test:
  image: maven:3.8.5-jdk-11
  stage: test
  variables:
    MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode"
    MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository"
  cache:
    paths:
      - .m2/repository/
      - target/
    key: $CI_PROJECT_NAME
  script:
    - mvn test
  artifacts:
    reports:
      junit: target/surefire-reports/*.xml
  only:
    - merge_requests

结果:

5. 总结

优点:

缺点:

标签:sonarqube,gitlab,json,极狐,report,7.6,sonar,data
来源: https://www.cnblogs.com/leffss/p/16476268.html