background
Although there are rising stars such as Jenkins X, Drone and Tekton in the cloud native era, Jenkins, an old CI/CD tool, is still the mainstream use scheme of major companies.
For example, the packaging and release of our private cloud products was completed by this old guy. However, the traditional Jenkins Slave one master multi slave mode has some pain points, such as:
- The configuration environment of each Slave is different to complete the compilation and packaging of different languages, but these differentiated configurations make it very inconvenient to manage and difficult to maintain
- Resource allocation is uneven. Some Slave job s to be run are queued, while others are idle
- Resources are wasted. Each Slave may be a physical machine or a virtual machine. When the Slave is idle, resources will not be completely released.
Just because the Jenkins slave above has these pain points, we are eager for a more efficient and reliable way to complete the CI/CD process, and Docker virtualization container technology can well solve this pain point, especially in the Kubernetes cluster environment.
The following figure is a simple diagram of building Jenkins slave cluster based on Kubernetes:
As can be seen from the figure, the Jenkins Master runs on a node in the form of docker compose.
Jenkins Slave runs on the nodes of Kubernetes cluster in the form of Pod, and it is not always running. It will be dynamically created and automatically deleted according to the requirements.
The workflow of this method is as follows: when the Jenkins Master receives the Build request, it will dynamically create a Jenkins Slave running in the Pod according to the configured Label and register it with the Master. After running the Job, the Slave will be logged off, and the Pod will be automatically deleted and restored to its original state.
Then we use this method to bring the following benefits:
- Dynamically scale and reasonably use resources. Each time a Job is run, a Jenkins Slave will be automatically created. After the Job is completed, the Slave will automatically log off and delete the container, and the resources will be automatically released. Moreover, Kubernetes will dynamically allocate the Slave to the idle node according to the usage of each resource, so as to reduce the occurrence of high resource utilization of a node, It is also queued at the node.
- Good scalability. When the Kubernetes cluster is seriously short of resources and leads to Job queuing, it is easy to add a Kubernetes Node to the cluster to realize expansion.
kubernetes cluster
For kubernetes cluster deployment, it is most convenient to use kubedm for deployment. Please refer to the article "experience K8s with kubedm rapid deployment" I wrote a long time ago. Here is just a brief introduction:
- Use kubedm to create a kubernets cluster with a single master node
root@jenkins:~ # kubeadm init --pod-network-cidr=10.244.0.0/16 --apiserver-advertise-address=192.168.20.11
- After the cluster is successfully deployed, the following prompt will appear:
Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
- Check whether the node status and pod are normal
root@jenkins:~ # kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system coredns-f9fd979d6-9t6qp 1/1 Running 0 89s kube-system coredns-f9fd979d6-hntm8 1/1 Running 0 89s kube-system etcd-jenkins 1/1 Running 0 106s kube-system kube-apiserver-jenkins 1/1 Running 0 106s kube-system kube-controller-manager-jenkins 1/1 Running 0 106s kube-system kube-proxy-8pzkz 1/1 Running 0 89s kube-system kube-scheduler-jenkins 1/1 Running 0 106s root@jenkins:~ # kubectl get node NAME STATUS ROLES AGE VERSION jenkins Ready master 119s v1.19.8
- Remove the stain on the master node and allow other pods to be scheduled on the master node, otherwise the pods created by Jenkins later will not be scheduled on this node.
kubectl taint nodes $(hostname) node-role.kubernetes.io/master:NoSchedule-
Jenkins master
As for the deployment method of Jenkins master, I personally recommend using docker compose. There is nothing wrong with running in kubernetes cluster. You can refer to the blog based on Jenkins CI/CD (I). However, in terms of personal operation and maintenance, it is more convenient to deploy Jenkins master independently of kubernetes cluster 😂.
- docker-compose.yaml
version: '3.6'
services:
jenkins:
image: jenkins/jenkins:2.263.4-lts-slim
container_name: jenkins
restart: always
volumes:
- ./jenkins_home:/var/jenkins_home
network_mode: host
user: root
environment:
- JAVA_OPTS=-Duser.timezone=Asia/Shanghai- Use docker compose up to start. After successful startup, you will be prompted as follows. The key output from the log is the default password of the admin user. Use it to log in to Jenkins for the first time.
jenkins | 2021-03-06 02:22:31.741+0000 [id=41] INFO jenkins.install.SetupWizard#init: jenkins | jenkins | ************************************************************* jenkins | ************************************************************* jenkins | ************************************************************* jenkins | jenkins | Jenkins initial setup is required. An admin user has been created and a password generated. jenkins | Please use the following password to proceed to installation: jenkins | jenkins | 4c2361968cd94323acdde17f7603d8e1 jenkins | jenkins | This may also be found at: /var/jenkins_home/secrets/initialAdminPassword jenkins | jenkins | ************************************************************* jenkins | ************************************************************* jenkins | *************************************************************
After logging in, it is recommended to select plug-ins to install. Install as few plug-ins as possible and install them as needed.
Next, start to configure how Uncle Jenkins holds hands with captain kubernetes 🧑🤝🧑 :-). kubernets can be configured in system management > Node Management > configure clouds. Click Add a new cloud to add a kubernetes cluster.
- Configure connection parameters
- Add the kubeconfig file to Jenkins' credentials. Select Secret file as the credential type, and then upload the kubeconfig generated above using kubedm deployment here.
- Click connection test. If you are prompted with connected to kubernetes v1.19.8, you have successfully connected to the kubernetes cluster.
- About pod templates
In fact, it is to configure the pod template run by Jenkins Slave. Personally, it is not recommended to use the template in the plug-in. It is recommended to put the pod template in Jenkins file, because these configurations are closely related to our pipeline, so it is not convenient to store the pod configuration in Jenkins plug-in;
Inconvenient for subsequent migration and backup; These configurations may also be lost after subsequent plug-in upgrades. Therefore, it is recommended to directly define the configuration of the pod template in the Jenkins file, which is more flexible and will not be affected by the Jenkins plug-in upgrade. In short, using code to manage these pod configuration maintenance costs will be much less.
Jenkinsfile
- Pipeline Jenkinsfile. The following is a simple task to build the docker image of webp server go [7] project.
// Kubernetes pod template to run.
def JOB_NAME = "${env.JOB_NAME}"
def BUILD_NUMBER = "${env.BUILD_NUMBER}"
def POD_NAME = "jenkins-${JOB_NAME}-${BUILD_NUMBER}"
podTemplate(
# The pod template is defined here
)
{ node(POD_NAME) {
container(JOB_NAME) {
stage("Build image") {
sh """#!/bin/bash
git clone https://github.com/webp-sh/webp_server_go /build
cd /build
docker build -t webps:0.3.2-rc.1 .
"""
}
}
}
}- The pod template is as follows. Copy and paste the contents of the template into the Jenkins file above. To build an image in the container, we use the scheme of dind: mount the docker sock file of the host computer where the pod is located into the container of the pod. As long as the docker cli tool is installed in the pod container, the docker can be used directly like the host computer.
podTemplate(
cloud: "kubernetes",
namespace: "default",
name: POD_NAME,
label: POD_NAME,
yaml: """
apiVersion: v1
kind: Pod
spec:
containers:
- name: ${JOB_NAME}
image: "debian:buster-docker"
imagePullPolicy: IfNotPresent
tty: true
volumeMounts:
- name: dockersock
mountPath: /var/run/docker.sock
- name: jnlp
args: ["\$(JENKINS_SECRET)", "\$(JENKINS_NAME)"]
image: "jenkins/inbound-agent:4.3-4-alpine"
imagePullPolicy: IfNotPresent
volumes:
- name: dockersock
hostPath:
path: /var/run/docker.sock
""",
)- Build Debian: Buster docker image. Use it to build docker image in pod container. The Dockerfile used is as follows:
FROM debian:buster
RUN apt update \
&& apt install -y --no-install-recommends \
vim \
curl \
git \
make \
ca-certificates \
gnupg \
&& rm -rf /var/lib/apt/lists/*
RUN curl -fsSL "https://download.docker.com/linux/debian/gpg" | apt-key add -qq - >/dev/null \
&& echo "deb [arch=amd64] https://download.docker.com/linux/debian buster stable" > /etc/apt/sources.list.d/docker.list \
&& apt update -qq \
&& apt-get install -y -qq --no-install-recommends docker-ce-cli \
&& rm -rf /var/lib/apt/lists/*After defining the Jenkins file and building the image in the pod template, we begin to use it to create pipelined tasks.
Assembly line
- Create a new task on Jenkins and select the type of task as pipeline
Copy and paste the defined Jenkinsfile into Pipelinescript and click save. Click build now on the new Job page to run the pipeline task.
- Use the kubectl command on the kubernetes cluster machine to check whether the pod is Running normally
root@jenkins:~ # kubectl get pod NAME READY STATUS RESTARTS AGE jenkins-webps-9-bs78x-5x2042/2Running066s
- The job is running normally and the status is green, indicating that the job has been successfully executed.
- Check whether the docker image is successfully built on the kubernetes cluster machine
root@jenkins:~ # docker images | grep webps webps 0.3.2-rc.1 f68f496c0444 20 minutes ago 13.7MB
Stepping pit
- pod cannot run normally
Running
in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Created Pod: kubernetes default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r][Scheduled] Successfully assigned default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r to jenkins
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r][Pulling] Pulling image "debian:buster"
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r][Pulled] Successfully pulled image "debian:buster" in 2.210576896s
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r][Created] Created container debian
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r][Started] Started container debian
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r][Pulling] Pulling image "jenkins/inbound-agent:4.3-4-alpine"
Still waiting to schedule task
'debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r' is offline
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r][Pulled] Successfully pulled image "jenkins/inbound-agent:4.3-4-alpine" in 3.168311973s
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r][Created] Created container jnlp
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-9wm0r][Started] Started container jnlp
Created Pod: kubernetes default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-qdw4m
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-qdw4m][Scheduled] Successfully assigned default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-qdw4m to jenkins
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-qdw4m][Pulled] Container image "debian:buster" already present on machine
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-qdw4m][Created] Created container debian
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-qdw4m][Started] Started container debian
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-qdw4m][Pulled] Container image "jenkins/inbound-agent:4.3-4-alpine" already present on machine
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-qdw4m][Created] Created container jnlp
[Normal][default/debian-35a11b49-087b-4a8c-abac-bd97d7eb5a1f-fkmzq-qdw4m][Started] Started container jnlpThis is because the jnlp container in the Jenkins pod cannot connect to the Jenkins master. You can check whether the Jenkins address and Jenkins channel parameters in system management > Node Management > configure clouds on the Jenkins master are configured correctly.
end
So far, we're done. Let uncle Jenkins hold hands with captain kubernetes 🧑🤝🧑 la The above uses a simple example to show how to run Jenkins' Job task on kubernetes cluster, but the situation encountered in actual work may be more complex, and the pipeline needs to configure more parameters. Then I will talk about the advanced usage in the next blog: using Jenkins to package the kubespreay offline installation package.
Original link: https://blog.k8s.li/jenkins-with-kubernetes.html