jenkins_CICD部署.md

一套基于 jenkins 的部署方法

前言

以我个人使用经验来说,Jenkins 使用的很痛苦,一边写一边骂。目前只是做了一套简单的方案,还有更多复杂的东西并没有做。
目前做到的是,本地搭建 Jenkins 环境,使用 BlueOcean,Jenkins pipeline,shared library 多环境部署
主要精力在标准 SpringBoot 项目的部署。

Jenkins 部署就不说了,我是直接在物理机部署的 Jenkins

规范

标准的 Springboot 项目,最外层应该有以下几个文件
pom.xml 文件,该文件是 maven 的配置文件
Dockerfile 文件,Docker 容器文件
jenkinsfile 文件,jenkins pipeline 文件。

然后开发一个公共库,这里命名为 jenkins-pipeline-lib,并在 Jenkins 全局配置 Global Pipeline Libraries

jenkinsfile

1
2
3
4
library 'relengxing-library' // 在Global Pipeline Libraries中配置的library name
def map = [:]
// 此处可以添加参数,会传入方法
build_v2(map)

dockerfile

底层容器

1
2
3
4
5
6
7
8
9
10
11
12
13
FROM java:8-jdk-alpine
RUN apk add --update ttf-dejavu fontconfig
VOLUME /tmp
RUN mkdir -p /application/agent
# 挂载时区
ENV TZ=Asia/Shanghai
RUN ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && echo $TZ > /etc/timezone
#定义jvm参数变量
ENV JAVA_OPTS=""
# 复制SkyWalking
ADD agent/ /application/agent/
#启动命令
#ENTRYPOINT cd /application && java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -jar app.jar

自定义容器

1
2
3
4
5
6
7
8
9
10
11
#FROM 192.168.1.138:5000/supay-java8-base:latest
FROM 192.168.1.138:5000/xxxx-java8-skywalking-base:latest

ARG JAR_FILE
COPY ${JAR_FILE} /application/app.jar

#启动命令
ENTRYPOINT cd /application && java ${JAVA_OPTS} -Djava.security.egd=file:/dev/./urandom -jar app.jar
# 此处填写该项目需要暴露的端口号
EXPOSE 20002

shared pipeline

build_v2.groovy 文件
groovy 仅作为胶水语言,实际的编译执行等,通过 sh 和 python 来完成
项目结构
一些参考文章:
Jenkins pipeline 中优雅的执行 shell/python/groovy 脚本

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
def call(configMap) {

pipeline {
agent any
options {
// 禁止同时运行多个流水线
disableConcurrentBuilds()
}
//常量参数,初始确定后一般不需更改
// environment {
// projectName = "${configMap.projectName}"
// java_opts = "${configMap.java_opts}"
// }

stages {
stage("Pre work") {
input {
message "请输入版本号?"
ok "确定"
// submitter "alice,bob"
parameters {
string(name: 'OLD_VERSION', defaultValue: 'latest', description: '旧版本号?')
string(name: 'NEW_VERSION', defaultValue: 'latest', description: '新版本号?')
}
}

steps {
echo "Pre work echo"
dir("cicddir") {
git credentialsId: 'bfxxxxx0-xxxx-4259-xxxx-22946xxxxxce', url: 'https://gitee.com/XXXXXXXX/jenkins-pipeline-lib.git'
sh("chmod -R +x ./cicd/*")
// sh("cicd/pre.sh")
sh("python3 cicd/read_project_info.py ${env.BRANCH_NAME} ${OLD_VERSION} ${NEW_VERSION}")
}
}
}

stage('Docker Build') {
steps {
script {
dir("cicddir") {
sh("python3 cicd/build.py")
}
}
}
}

stage("Deploy") {
steps {
script {
dir("cicddir") {
sh("python3 cicd/deploy.py")
}
}
}
}
}
post {
always {
echo 'I will always say Hello!'
}
aborted {
echo 'I was aborted'
}
failure {
echo 'I was failure'
}
}
}

自定义脚本

read_project_info.py 会从 pom 文件中读取项目名称等信息,从 dockerfile 读端口号, 从全局配置中读取配置信息,生成一个本地文件,供后面的脚本使用
build.py 执行编译命令,并推送镜像到服务器
deploy.py 执行部署命令,可部署到普通 docker 服务器和 K8s 服务器

read_project_inf.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
#!/usr/bin/python3

"""
该脚本用于读取JAVA项目信息
会从 Pom.xml Dockerfile yaml 中读取项目部署所需要的信息,生成一个json文件存到本地,供其他脚本使用
"""
import json
import os
import xml.etree.ElementTree as xml

import sys
import yaml

branch = sys.argv[1]
old_version = sys.argv[2]
new_version = sys.argv[3]

GLOBAL_PROJECT_INFO = 'global_project_info.json'


def getMappingsNode(node, nodeName):
if node.findall('*'):
for n in node.findall('*'):
if nodeName in n.tag:
return n
else:
return getMappingsNode(n, nodeName)


def get_project_info(filepath):
"""
解析java Pom.xml 文件,获取项目名称,版本号,service模块所在路径
:param filepath: Pom文件地址
:return:
"""
pomFile = xml.parse(filepath)
root = pomFile.getroot()
artifactId = getMappingsNode(root, 'artifactId').text
version = getMappingsNode(root, 'version')
version = str(version.text).replace('-SNAPSHOT', '')
moduleList = getMappingsNode(root, 'modules')
serviceModule = None
if moduleList != None:
for module in moduleList:
module = module.text
if module.endswith("app") or module.endswith("service"):
serviceModule = module
return {
"project_name": artifactId,
"version": version,
"service_module": serviceModule
}


def get_yaml_file(branch, project_name):
"""
根据所选分支和项目名称获取全局配置信息
:param branch:
:param project_name:
:return:
"""
profile = None
if branch == 'develop':
profile = 'dev'
elif branch == 'test':
profile = 'test'
elif branch == 'master':
profile = 'master'
filename = 'resources/deploy_' + profile + '.yaml'
fs = open(os.path.join(filename), encoding="UTF-8")
datas = yaml.load(fs)
project_info_list = datas['project']
for project_info in project_info_list:
if project_info['project_name'] == project_name:
project_info['profile'] = profile
return project_info
return None


def get_docker_file():
"""
从 dockerfile 中读取,看是否使用了skywalking镜像,使用expose的端口来映射,仅读取第一个export
:return:
"""
expose = ''
skywalking_image = False
filename = '../Dockerfile'
fs = open(os.path.join(filename), encoding="UTF-8")
for line in fs:
if line.lower().startswith("from") and line.lower().find("skywalking") >= 0:
skywalking_image = True
if line.lower().startswith("expose"):
expose = expose + line.replace("EXPOSE", "").replace("expose", "").strip()
break
fs.close()
return {
"expose": expose,
"skywalking_image": skywalking_image
}


project_info = get_project_info('../pom.xml')
project_info.update(get_yaml_file(branch, project_info['project_name']))
project_info.update(get_docker_file())

project_info['branch'] = branch
project_info['old_version'] = old_version
project_info['new_version'] = new_version

project_info['image'] = project_info["project_name"] + ":" + project_info['new_version']
project_info['remote_image'] = project_info['registry'] + "/" + project_info['image']

if project_info['service_module'] == None:
project_info["jar_path"] = "./target/"
else:
project_info["jar_path"] = "./" + project_info['service_module'] + "/target/"

print(project_info)
with open(GLOBAL_PROJECT_INFO, 'w', encoding='utf-8') as f:
f.write(json.dumps(project_info, ensure_ascii=False) + '\n')

build.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
#!/usr/bin/python3

# 部署脚本
import json
import os

"""
export PAHT=$PATH:/bin:/usr/bin
/bin/mvn clean package -Dmaven.test.skip=true -U
docker build -t ${projectName} --build-arg JAR_FILE=${jarPath} .
"""
GLOBAL_PROJECT_INFO = 'global_project_info.json'

with open(GLOBAL_PROJECT_INFO, 'r', encoding='utf-8') as f:
global_project_info = json.load(f)

# print(global_project_info)

# os.system("cd ../")
# os.system("export PAHT=$PATH:/bin:/usr/bin")
# os.system("cd .. | /bin/mvn clean package -Dmaven.test.skip=true -U")

build_result = os.system("""
cd ../
export PAHT=$PATH:/bin:/usr/bin
/bin/mvn clean package -Dmaven.test.skip=true -U
""")

if build_result != 0:
raise Exception("Jar包编译失败")


def traversalDir_FirstDir(path):
list = []
if (os.path.exists(path)):
files = os.listdir(path)
for file in files:
m = os.path.join(path, file)
if (os.path.isfile(m)):
h = os.path.split(m)
if h[1].endswith(".jar") and not h[1].endswith("sources.jar"):
list.append(h[1])
return list


jar_list = traversalDir_FirstDir("../" + global_project_info["jar_path"])
print(jar_list)
if len(jar_list) == 0:
raise Exception("编译错误,没有合适的jar包")

docker_build_shell = "docker build -t %s --build-arg JAR_FILE='%s' ." % (
global_project_info["image"], global_project_info["jar_path"] + jar_list[0])

print(docker_build_shell)
docker_build_result = os.system("""
cd ../
%s
""" % (docker_build_shell))

if docker_build_result != 0:
raise Exception("Docker镜像编译失败")

print("build image success")

docker_push_result = os.system("""
docker tag %s %s
docker push %s
""" % (global_project_info["image"], global_project_info["remote_image"], global_project_info["remote_image"]))
print("push image success")

if docker_push_result != 0:
raise Exception("Docker镜像推送失败")

deploy.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
# 部署脚本
import json

from deploy_to_docker import deploy_to_docker
from deploy_to_k8s import deploy_to_k8s

GLOBAL_PROJECT_INFO = 'global_project_info.json'

with open(GLOBAL_PROJECT_INFO, 'r', encoding='utf-8') as f:
global_project_info = json.load(f)


def develop_process():
"""
开发环境使用docker部署
:return:
"""
deploy_to_docker(global_project_info)


def test_process():
"""
测试环境部署到 K8s环境
:return:
"""
deploy_to_k8s(global_project_info)


def master_process():
pass


if __name__ == '__main__':
if global_project_info["branch"] == "develop":
develop_process()
if global_project_info["branch"] == "test":
test_process()
if global_project_info["branch"] == "master":
master_process()

deploy_to_docker

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
from remote_cmd import ssh_exec_command


def deploy_to_docker(global_project_info: dict, remote_user="root"):
"""
部署到docker环境
部署实例数仅machine有效
:param global_project_info: 全局项目信息
:param remote_ip_list:
:param port:
:param remote_user:
:return:
"""

project_name = global_project_info['project_name']
java_opts = global_project_info['java_opts']
profile = global_project_info['profile']
remote_image = global_project_info['remote_image']
port = global_project_info['expose']

remote_ip_str = global_project_info["machine"]
remote_ip_list = remote_ip_str.split(",")
for remote_ip in list(set(remote_ip_list)):
if global_project_info["skywalking"] and global_project_info["skywalking_image"]:
skywalking_host = global_project_info["skywalking_host"]
skywalking_java_opts = '-javaagent:agent/skywalking-agent.jar -Dskywalking.agent.service_name=' + project_name + ' -Dskywalking.collector.backend_service=' + skywalking_host
else:
skywalking_java_opts = ""

cmd = "docker rm -f %s || true" % project_name
cmd_result = ssh_exec_command(remote_ip, remote_user, None, cmd)
# if not cmd_result:
# raise Exception("删除容器失败")

cmd = "docker pull %s" % remote_image
print(remote_ip, remote_user)
cmd_result = ssh_exec_command(remote_ip, remote_user, None, cmd)
if not cmd_result:
raise Exception("拉取镜像失败")

cmd = "docker run --name=%s --restart=always -p %s:%s -d -e JAVA_OPTS='%s %s -Dspring.profiles.active=%s' %s" % (
project_name, port, port, skywalking_java_opts, java_opts, profile, remote_image)
cmd_result = ssh_exec_command(remote_ip, remote_user, None, cmd)
if not cmd_result:
raise Exception("运行容器失败")
print(cmd_result)

deploy_to_k8s

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
import json

from remote_cmd import ssh_exec_command


def template_process(template, project_name, remote_image, port, replicas, env):
kuber_file = str(template).replace("${project_name}", project_name)
kuber_file = kuber_file.replace("${remote_image}", remote_image)
kuber_file = kuber_file.replace("${port}", port)
# kuber_file = kuber_file.replace("${node_port}", node_port)
kuber_file = kuber_file.replace("${replicas}", str(replicas))
kuber_file = kuber_file.replace("${java_opts}", str(env))
return kuber_file


def deploy_to_k8s(global_project_info: dict, remote_user="root"):
"""
部署到 k8s环境
machine 仅代表 master所在节点,
replicas 用来控制实例数
env JAVA_OPTS='%s %s -Dspring.profiles.active=%s'
:param global_project_info: 全局项目信息
:param remote_ip_list:
:param port:
:param remote_user:
:return:
"""
java_opts = global_project_info['java_opts']
profile = global_project_info['profile']
project_name = global_project_info['project_name']

if global_project_info["skywalking"] and global_project_info["skywalking_image"]:
skywalking_host = global_project_info["skywalking_host"]
skywalking_java_opts = '-javaagent:agent/skywalking-agent.jar -Dskywalking.agent.service_name=' + project_name + ' -Dskywalking.collector.backend_service=' + skywalking_host
else:
skywalking_java_opts = ""
env = "%s %s -Dspring.profiles.active=%s" % (skywalking_java_opts, java_opts, profile)

template_file = "cicd/java_k8s_template.yaml"

k8s_custom = global_project_info.get('k8s_custom')
if k8s_custom is not None:
with open("../" + k8s_custom, 'r') as f:
template = f.read()
kuber = template
else:
with open(template_file, 'r') as f:
template = f.read()
kuber = template_process(template, global_project_info["project_name"], global_project_info["remote_image"],
global_project_info["expose"], global_project_info["replicas"], env)

kuber_cmd = """
source .bash_profile
echo "%s" | kubectl apply -f -
kubectl get po | grep -m1 %s | awk '{print $1}' | xargs -n1 -I{} kubectl delete po {}
""" % (kuber, project_name)
remote_user = "root"
# print(kuber_cmd)
remote_ip = global_project_info["machine"]
ssh_port = global_project_info.get('ssh_port')
res = ssh_exec_command(remote_ip, remote_user, '', kuber_cmd,port=ssh_port)
if not res:
raise Exception("执行 K8s 命令失败")
print(res)


if __name__ == '__main__':
GLOBAL_PROJECT_INFO = 'global_project_info.json'
with open(GLOBAL_PROJECT_INFO, 'r', encoding='utf-8') as f:
global_project_info = json.load(f)
deploy_to_k8s(global_project_info)

java_k8s_template.yaml

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
apiVersion: apps/v1
kind: Deployment
metadata:
name: ${project_name}
spec:
replicas: ${replicas}
strategy:
type: Recreate
selector:
matchLabels:
app: ${project_name}
template:
metadata:
labels:
app: ${project_name}
spec:
containers:
- name: ${project_name}
image: ${remote_image}
env:
- name: JAVA_OPTS
value: '${java_opts}'
imagePullPolicy: Always
ports:
- containerPort: ${port}
imagePullSecrets:
- name: jiezsecret
---
apiVersion: v1
kind: Service
metadata:
name: ${project_name}
labels:
app: ${project_name}
spec:
type: NodePort
ports:
- port: ${port}
targetPort: ${port}
nodePort: ${port}
name: http-port
selector:
app: ${project_name}

deploy_test.yaml 部分配置

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
project:
- project_name: xxxx-basics-eureka
replicas: 1
limit_memory: 1G
registry: registry-intl.cn-hangzhou.aliyuncs.com/default_au
machine: xx.xx.xx.195
ssh_port: 22
java_opts: '-Xms512m -Xmx512m'
skywalking: false
skywalking_host: 192.168.0.xx:11800
k8s_custom: kuber-test.yaml
- project_name: superpay-basics-config
replicas: 1
limit_memory: 1G
registry: registry-intl.cn-hangzhou.aliyuncs.com/default_au
machine: xx.xx.xx.xx
ssh_port: 22
java_opts: '-Xms512m -Xmx512m'
skywalking: false
skywalking_host: 192.168.0.xx:11800