Jenkins+harbor+gitlab+k8s deploy maven project

Keywords: Python jenkins

Jenkins+harbor+gitlab+k8s deploy maven project

1, Overview

The deployment flow chart of maven project is as follows:




Environment introduction

operating system ip role edition
ubuntu-16.04.4-server-amd64 Jenkins+harbor Jenkins 2.176.2,harbor 1.8.1
ubuntu-16.04.4-server-amd64 gitlab gitlab-ce_12.1.4
ubuntu-16.04.4-server-amd64 k8s master kubernetes 1.15.2
ubuntu-16.04.4-server-amd64 k8s node kubernetes 1.15.2








Due to the shortage of resources, Jenkins+harbor was merged into one. In fact, it should be deployed separately.

For k8s installation, please refer to the link:


For gitlab installation, please refer to the link:


For harbor installation, please refer to the link:


For Jenkins installation, please refer to the link:


Note: after Jenkins is installed, the following configuration is required. Otherwise, the function of this article cannot be realized!!!

For the k8s configuration of Jenkins based on https, please refer to the link:


GitLab+Jenkins continuous integration, please refer to the link:


Jenkins configures maven, please refer to the link:


2, Creating maven project with gitlab

  First, find a maven based project from github. The address is as follows:

Use the git client to download the project code


Create project

Log in to gitlab and create a project called spring boot demo



Click Set -- > member



Add two members, one is the jenkins user, which is used to pull the code. One is for me to submit code.

Note: my account has higher permissions and can be submitted directly to the master branch.



Submit the code downloaded by github to the project just created. The effects are as follows:



3, harbor create users and projects

Create jenkins user

Log in to the harbor background and click create user



Note that each item should be filled in. In particular, passwords must meet the requirements of password complexity.


Create project

Click new item



Enter the name, java. This is a private project.


  Click java



Click member -- > User



Enter jenkins and you will be prompted. Select the developer for the role.


4, jenkins modify default user

Insufficient permissions may occur when using jenkins to execute shell commands.

View jenkins default users

cat /etc/default/jenkins

The contents are as follows:

# pulled in from the init script; makes things easier.

# arguments to pass to java

# Allow graphs etc. to work even when an X server is present


# make jenkins listen on IPv4 address


# user and group to be invoked as (default to jenkins)

jenkins can be found_ User and jenkins_ The value of the group variable is jenkins


Modify the default user to root

Change the following two variables to root



Restart jenkins service

service jenkins restart


5, jenkins and k8s master are ssh free

Why do you want to ssh with k8s master? Because jenkins needs to log in to k8s master to do some pod operations.

Generate secret key

Log in to the jenkins server and generate the secret key

ssh-keygen -t rsa -P "" -f ~/.ssh/id_rsa


copy key



Test root password free


If no password is prompted, it means success!


6, jenkins logs in to harbor

Modify the docker configuration file and add a harbor Address

vim /etc/docker/daemon.json

The contents are as follows:

{"insecure-registries": [""]}


Reload docker configuration

/etc/init.d/docker reload


Test login

root@ubuntu:~/docker_dir# docker login -u jenkins -p Jenkins@1234
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See

Login Succeeded


If present

Error response from daemon: Get dial tcp connect: connection refused

express  / Incorrect modification of etc/docker/daemon.json file


7, jenkins creates maven project

New task

Log in to jenkins background and click   New Item

Enter a name and choose to build a maven project. Note: the task name should preferably be the project name, because the following deploy_ The script will call this variable.





Set up parametric build

Discard the old build and keep it for 2 days



Set source code

Enter the warehouse download address and select user root



Set maven build command

Enter the command: clean package


The mvn clean package successively executes seven phases: clean, resources, compile, testResources, testCompile, test, jar (packaging).

The package command completes the functions of project compilation, unit testing and packaging, but does not deploy the typed executable jar package (war package or other forms of package) to the local maven warehouse and the remote maven private server warehouse


The jar package is packaged into a docker image and pushed

Execute shell commands

When build is complete, choose to execute the shell command



Post the following content




cp ${WORKSPACE}/target/${jarName} ${docker_path}
bash -x /root/docker_dir/ ${harborPro} ${projectName} ${docker_path} ${jarName}


Content explanation:

Jarname the file name of the jar package, whatever the name is

jarFolder jar package folder

harborPro   For the project in harbor, in the above steps, create a java private project.

projectName project name

docker_ Working directory of the path docker command


Finally, a shell script is executed. Let's talk about it!


The effects are as follows:




Define shell script

Log in to the jenkins server and create a directory

mkdir /root/docker_dir


Edit script

cd /root/docker_dir/

The contents are as follows:

# maven $workspace $jarname
# ${harborPro} ${projectName} ${docker_path} ${jarName}

set -e

# harbor authentication user

# harbor Address and tag
tag=$(date +%s)
echo ${taget_image}

# Log in to docker
cd ${docker_path}
sudo docker login ${harbor_server} -u ${user_name} -p ${password}

# Generate an image and push it to harbor, and finally delete the local image
sudo docker build --build-arg app=${appName} -t ${taget_image} .
sudo docker tag ${taget_image} ${server_path}/${harbor_project}/${projectName}
echo "The name of image is ${server_path}/${harbor_project}/${projectName}"
sudo docker push ${server_path}/${harbor_project}/${projectName}:latest
sudo docker rmi -f $(docker images|grep ${projectName}|grep ${tag}|awk '{print $3}'|head -n 1)


Add execution permission

chmod 755 /root/docker_dir/


yaml files are copied to k8s master and applied

k8s master operation

Log in to k8s master and create a directory

mkdir kube-conf


Delete kube.yaml in gitlab springbootdemo project and add two yaml files.


apiVersion: v1
kind: Service
  name: ph-service
  type: NodePort
  - name: ph
    port: 8080
    nodePort: 31002
    targetPort: 8080
    protocol: TCP
    app: ph



apiVersion: v1
kind: ReplicationController
  name: ph-rc
  replicas: 1
        app: ph
        - name: harborsecret
        - name: ph
          imagePullPolicy: Always
          - containerPort: 8080
            - name: key
              value: "value"


Submit 2 yaml files to gitlab


k8s generate imagePullSecrets, please refer to the following link

Note: specify jenkins when logging in with docker

docker login -u jenkins -p Jenkins@1234


jenkins operation

Log in to the jenkins server and add a restart application script

cd /root/docker_dir

The contents are as follows:


scp ${WORKSPACE}/*.yaml ${MASTER}:${CONF_DIR}
# Pod running process
RUN=$(ssh $MASTER kubectl get po|grep -w ${ProJ}|wc -l)
if [ $RUN -eq 0 ];then
        ssh ${MASTER} kubectl apply -f ${CONF_DIR}/${ProJ}-rc.yaml
        ssh ${MASTER} kubectl apply -f ${CONF_DIR}/${ProJ}-service.yaml
        # Delete pod
        podname=$(ssh ${MASTER} kubectl get po|grep -w ${ProJ}|awk '{print $1}')
        ssh ${MASTER} kubectl delete po $podname --grace-period=0 --force
        ssh ${MASTER} kubectl get po|grep ${ProJ}


Add execution permission

chmod 755 /root/docker_dir/


Add execute shell command



  Post the following content


bash -x /root/docker_dir/


The effects are as follows:




Build applications manually

Click Build Now


Click #1


Click console output


It will download some components by itself and need to wait for some time.


The spring icon appears, indicating that the application starts to build.


An error occurred at the end


Why? View the Dockerfile file of spring bootdemo and the first line


Need to added to  / In / etc/docker/daemon.json


Log in to jenkins server and modify  / etc/docker/daemon.json

vim /etc/docker/daemon.json

The contents are as follows:

{"insecure-registries": ["",""]}


Reload docker

/etc/init.d/docker reload


Manually build it again and click Build Now

View the build process

The following prompt appears indicating that the image is being downloaded  


Finally, prompt for execution completion



View harbor warehouse

You will find an additional image, which was submitted when you just built it.


8, Access k8s applications

View Pod status

root@k8s-master:~# kubectl get pods -o wide
NAME                                READY   STATUS             RESTARTS   AGE     IP              NODE        NOMINATED NODE   READINESS GATES
ph-rc-nj7j2   0/1     ImagePullBackOff   0          5m19s   k8s-node1   <none>           <none>


It is found that pod is running on k8s-node1 this server and the status is ImagePullBackOff


View pod details

kubectl describe po maven-deployment-7bd87867f8-6tlgp


  Type     Reason     Age                     From                Message
  ----     ------     ----                    ----                -------
  Normal   Scheduled  7m15s                   default-scheduler   Successfully assigned default/maven-deployment-7bd87867f8-6tlgp to k8s-node1
  Normal   Pulling    5m49s (x4 over 7m14s)   kubelet, k8s-node1  Pulling image ""
  Warning  Failed     5m49s (x4 over 7m14s)   kubelet, k8s-node1  Failed to pull image "": rpc error: code = Unknown desc = Error response from daemon: Get dial tcp connect: connection refused
  Warning  Failed     5m49s (x4 over 7m14s)   kubelet, k8s-node1  Error: ErrImagePull
  Normal   BackOff    5m36s (x6 over 7m13s)   kubelet, k8s-node1  Back-off pulling image ""
  Warning  Failed     2m11s (x20 over 7m13s)  kubelet, k8s-node1  Error: ImagePullBackOff



Get dial tcp connect: connection refused

Description, k8s-node1   The / etc/docker/daemon.json of this server has not been changed


Log in to k8s-node1   This server, modify the file

vim /etc/docker/daemon.json

The contents are as follows:

{"insecure-registries": [""]}


Reload docker

/etc/init.d/docker reload


Reload pod

Log in to k8s master server, delete pod and reapply

root@k8s-master:~# cd /root/kube-conf/
root@k8s-master:~/kube-conf# kubectl delete -f ph-rc.yaml
service "maven-service" deleted
deployment.extensions "maven-deployment" deleted
root@k8s-master:~/kube-conf# kubectl apply -f ph-rc.yaml
service/maven-service created
deployment.extensions/maven-deployment created


View status again

root@k8s-master:~/kube-conf# kubectl get pods -o wide
NAME                                READY   STATUS    RESTARTS   AGE   IP              NODE        NOMINATED NODE   READINESS GATES
ph-rc-7bd7475544-q9b6j   1/1     Running   0          27s   k8s-node1   <none>           <none>

Found in Running status


Access the application page

View svc exposed ports

root@k8s-master:~/kube-conf# kubectl get svc|grep maven
ph-service   NodePort   <none>        8080:31002/TCP   4m

It can be found that the exposed port is 31002


Access pages using Google


The effects are as follows:


ip using k8s-node1

The effect is the same as above!


9, Update application code

Open the local project springbootdemo and modify the file

The path is as follows:



Modify the following 2 lines and add 1 to the number

jsonObject.put("welcome2", "2");
jsonObject.put("welcome7", "7");


Resubmit to gitlab, and then build it again. Click Build Now

After the build is successful, refresh the page


  Found content has been updated!


Text reference link:


Classification: python operation and maintenance development
label: Jenkins

Posted by krotkruton on Wed, 24 Nov 2021 08:01:51 -0800