Building Jenkins as code
In our company, we try to use the "everything is code" mode, which involves replicable infrastructure, monitoring, tasks, etc. But in this article, I'll show you how to apply this pattern to Jenkins. Yes, I mean for Jenkins' fully replicable configuration, as well as infrastructure, plug-ins, credentials, tasks, and other things in the code. In addition, you will solve the following questions in this article:
-
Has our Jenkins become more stable?
-
Can we change Jenkins and task configuration frequently?
-
Is upgrading Jenkins and its plug-ins no longer a pain for us?
-
Have we managed all the changes on Jenkins?
-
After the failure, can we recover Jenkins quickly?
My name is Amet Umerov. It's a Preply.com DevOps engineer. Let's start!
Preliminary introduction
When we talk about DevOps tools, the first thing that comes to mind is a CI/CD system. We use Jenkins in Preply because we have hundreds of tasks every day. Many of the features we use cannot be provided in other systems. Even if these features are provided, they will be some simplified functions.
We want to fully code Jenkins as well as infrastructure, configuration, tasks, and plug-ins. And, we've had previous experience in running Kubernetes, but because Jenkins architecture And our own purpose is to find that it doesn't fit us.
This is what we want to achieve
Building the underlying architecture for Jenkins
We're using AWS to manage all our infrastructure using Terraform, as well as other tools from HashiStack, such as Packer Or Vault.
As I mentioned earlier, we tried to use Kubernetes to host Jenkins, but we ran into problems in expanding PVC, resources, and some non considered architectures.
Here, we use AWS resources, such as EC2 instance, SSL authentication, load balancing, CloudFront allocation, etc. AMI is built by Packer with perfect integration of Terraform and Vault.
{ "variables": { "aws_access_key": "{{vault `packer/aws_access_key_id` `key`}}", "aws_secret_key": "{{vault `packer/aws_secret_access_key` `key`}}", "aws_region": "{{vault `packer/aws_region` `key`}}", "vault_token": "{{env `VAULT_TOKEN`}}" }, "builders": [{ "access_key": "{{ user `aws_access_key` }}", "secret_key": "{{ user `aws_secret_key` }}", "region": "{{ user `aws_region` }}", "type": "amazon-ebs", "communicator": "ssh", "ssh_username": "ubuntu", "instance_type": "c5.xlarge", "security_group_id": "sg-12345", "iam_instance_profile": "packer-role-profile", "ami_name": "packer-jenkins-master-{{timestamp}}", "ami_description": "Jenkins master image", "launch_block_device_mappings": [{ "device_name": "/dev/sda1", "volume_size": 50, "volume_type": "gp2", "delete_on_termination": true }], "source_ami_filter": { "filters": { "virtualization-type": "hvm", "name": "ubuntu/images/*ubuntu-bionic-18.04-amd64-server-*", "root-device-type": "ebs" }, "owners": ["099720109477"], "most_recent": true } }], "provisioners": [{ "type": "shell", "environment_vars": ["VAULT_TOKEN={{ user `vault_token` }}"], "scripts": ["packer_bootstrap.sh"] }] }
This is how the packer is configured
The Bootstrap file, packer ﹣ Bootstrap.sh, contains all the commands to pre install the software in AMI. For example, we use Docker, Docker compose, and vaultenv Or install the Datadog node for monitoring.
Considering the architecture of this AMI, we can use Terraform, CloudFormation, Pulumi and even Ansible. This is one of the possible architectures to use Jenkins on AWS.
Users access Jenkins through internal LB and GitHub webhook through public lb.
The Jenkins we use integrate GitHub, so we should provide some Jenkins URL s for GitHub through the external network. There are many operational options( IP white list , URL or token whitelist, etc.) and we combine Cloudfront to allow path and token validation.
After doing these things, we have a ready-made infrastructure with AMI. The Vault that provides the possibility for monitoring and for obtaining company credentials is also available.
Using Docker to manage Jenkins and its plug-in version
OK, the next step is Jenkins and plug-ins. We had a lot of problems upgrading the Jenkins plug-in before, so the main goal is to fix the version for them.
Docker helped us a lot at this time. We use Jenkins image built in advance Use it as the base image for our installation.
FROM jenkins/jenkins:2.215 ENV CASC_JENKINS_CONFIG /jenkins_configs USER root # Install additional packages RUN apt update && \ apt install -y python3 python3-pip && \ pip3 install awscli jenkins-job-builder jjb-reactive-choice-param --no-cache-dir USER jenkins VOLUME /jenkins_configs VOLUME /var/jenkins_home # Install plugins COPY plugins.txt /usr/share/jenkins/ref/ RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt
We installed some additional installation packages for Job Builder, which we will use later, and passed a data volume for Jenkins and plug-ins installation.
We are https://our-jenkins-url/script The Groovy code from above pastes and saves it to plugins.txt, through which we can easily get the plug-in list.
Jenkins.instance.pluginManager.plugins.each{ plugin -> println ("${plugin.getShortName()}:${plugin.getVersion()}") }
Finally, the Docker compose configuration runs Jenkins in Docker (we also use vaultenv to transfer credentials from Vault to Docker compose):
version: "3" services: jenkins: build: . container_name: jenkins restart: always ports: - "50000:50000" - "8080:8080" volumes: - ./configs/:/jenkins_configs/:ro - ./jenkins_home/:/var/jenkins_home/:rw environment: - VAULT_TOKEN - GITHUB_TOKEN - AWS_ACCESS_KEY_ID - AWS_SECRET_ACCESS_KEY - JAVA_OPTS=-Xms4G -Xmx8G -Xloggc:/var/jenkins_home/gc-%t.log -XX:NumberOfGCLogFiles=5 -XX:+UseGCLogFileRotation -XX:GCLogFileSize=20m -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1 volumes: configs: driver: local jenkins_home: driver: local
It's important to remember that some Java parameters can help us deal with some garbage collection and resource limitations. This article It's great for tuning Jenkins.
Of course, we can run Jenkins locally to install the plug-in, or a new version of Jenkins. It's also great.
Now we have Jenkins with the plug-in installed, and even we can run it locally and easily deploy it to the production environment. Let's add more configuration to it.
Configure Jenkins as a code (JCasC) plug-in for the primary node
Jenkins configuration is code( JCasC )The plug-in stores the configuration in a readable format.
This plug-in allows us to describe security configuration, credentials, plug-in configuration, nodes, views, and many other things.
This YAML configuration file is divided into five parts:
-
credentials (system credential description)
-
jenkins (authorization, cloud settings, global parameters, nodes, security domains, and views)
-
Security (global security configuration, such as script permissions)
-
tool (configuration of external tools, such as git, allure, etc.)
-
unclassified (other configurations, such as Slack integration)
We can import the configuration from the existing Jenkins installation process
It supports Different credential providers Used to manage credentials, but we also need to use environment variables.
credentials: system: domainCredentials: - credentials: - usernamePassword: description: "AWS credentials" id: "aws-creds" password: ${AWS_SECRET_ACCESS_KEY} scope: GLOBAL username: ${AWS_ACCESS_KEY_ID} - string: description: "Vault token" id: "vault-token" scope: GLOBAL secret: ${VAULT_TOKEN} ...
We will also Amazon EC2 plug in For bootstrap agent on AWS, its configuration can also be managed using this plug-in. Matrix based authorization allows us to manage the permissions of users in a code way.
jenkins: authorizationStrategy: projectMatrix: permissions: - "Overall/Administer:steve.a@example.org" - "Credentials/View:john.d@example.org" ... clouds: - amazonEC2: cloudName: "AWS" privateKey: ${EC2_PRIVATE_KEY} region: "${AWS_REGION}" templates: - ami: "ami-12345678" amiType: unixData: sshPort: "22" connectionStrategy: PRIVATE_IP deleteRootOnTermination: true description: "jenkins_agent" idleTerminationMinutes: "20" instanceCapStr: "100" minimumNumberOfInstances: 0 mode: EXCLUSIVE numExecutors: 1 remoteAdmin: "jenkins" remoteFS: "/home/jenkins" securityGroups: "sg-12345678" subnetId: "subnet-12345678" type: C52xlarge ...
We also use a lot of cool things. If we have a process to test Jenkins' local changes, we can find and fix bug s before putting them into production.
So we installed a reusable Jenkins configuration, and last but not least our task
Integrating Job Builder for freestyle tasks
When we talk about freestyle tasks, there are several different ways to create them in Jenkins:
-
Using the GUI (the easiest way to do this, just click on it)
-
Direct use REST API
-
Use similar Job DSL Or plug-in of JJB wrapper
Jenkins Job Builder (JJB) allows us to configure tasks into a readable text format (YAML or JSON). It's very comfortable to use SCM to manage these tasks. Basically, we can use JJB to create a CI/CD process for our CI/CD tool.
. ├── config.ini ├── jobs │ ├── Job1.yaml │ | ... │ └── Job2.yaml └── scripts ├── job1.sh | ... └── job2.sh
Here, we can describe it in a Job1.yaml file Definition of tasks , the task steps are in the script (for example, job1.sh).
- job: name: Job1 project-type: freestyle auth-token: mytoken disabled: false concurrent: false node: jenkins_agent triggers: - timed: '0 3 * * *' builders: - shell: !include-raw: ../scripts/job1.sh
This is a To configure Examples of documents:
$ cat config.ini [job_builder] ignore_cache=True exclude=jobs/Job2 [jenkins] url=https://jenkins.example.org user=some_user password=some_password $ jenkins-jobs --conf config.ini test -r jobs/ $ jenkins-jobs --conf config.ini update -r jobs/
It should be easy to run in the jenkins task upgrade command
Of course, our Jenkins users need permission to create and configure tasks. We just need to run a subtask on the primary node to import from JJB to all Jenkins configurations.
JJB is not a panacea, because there are still some plug-ins that are not very commonly used that are not supported. But it's still a very flexible plug-in. In addition, it can use macroses Configure.
conclusion
Now that we have seen a general outline of the "everything is code" pattern and how we use Jenkins, we can go back to the questions mentioned at the beginning of the article. Have you found the answer? Perhaps, it is obvious that the answer to all five questions is "yes".
We just want to share with you our experience in parameter configuration and Jenkins best practices We did not go into depth.
