Building Jenkins as code

Keywords: Programming jenkins AWS Docker Ubuntu

This article starts from: Jenkins Chinese community
Original link By Amet Umerov
Translator: s1mple
 

Building Jenkins as code

This paper mainly introduces the specific method of Jenkins "configuration as code" mode, and explains the mode with tools and scripts

In our company, we try to use the "everything is code" mode, which involves replicable infrastructure, monitoring, tasks, etc. But in this article, I'll show you how to apply this pattern to Jenkins. Yes, I mean for Jenkins' fully replicable configuration, as well as infrastructure, plug-ins, credentials, tasks, and other things in the code. In addition, you will solve the following questions in this article:

  • Has our Jenkins become more stable?

  • Can we change Jenkins and task configuration frequently?

  • Is upgrading Jenkins and its plug-ins no longer a pain for us?

  • Have we managed all the changes on Jenkins?

  • After the failure, can we recover Jenkins quickly?

My name is Amet Umerov. It's a Preply.com DevOps engineer. Let's start!

Preliminary introduction  

When we talk about DevOps tools, the first thing that comes to mind is a CI/CD system. We use Jenkins in Preply because we have hundreds of tasks every day. Many of the features we use cannot be provided in other systems. Even if these features are provided, they will be some simplified functions.

We want to fully code Jenkins as well as infrastructure, configuration, tasks, and plug-ins. And, we've had previous experience in running Kubernetes, but because Jenkins architecture And our own purpose is to find that it doesn't fit us.

This is what we want to achieve

Building the underlying architecture for Jenkins  

We're using AWS to manage all our infrastructure using Terraform, as well as other tools from HashiStack, such as Packer Or Vault.

As I mentioned earlier, we tried to use Kubernetes to host Jenkins, but we ran into problems in expanding PVC, resources, and some non considered architectures.

Here, we use AWS resources, such as EC2 instance, SSL authentication, load balancing, CloudFront allocation, etc. AMI is built by Packer with perfect integration of Terraform and Vault.

{

"variables": {

"aws_access_key": "{{vault `packer/aws_access_key_id` `key`}}",

"aws_secret_key": "{{vault `packer/aws_secret_access_key` `key`}}",

"aws_region": "{{vault `packer/aws_region` `key`}}",

"vault_token": "{{env `VAULT_TOKEN`}}"

},

"builders": [{

"access_key": "{{ user `aws_access_key` }}",

"secret_key": "{{ user `aws_secret_key` }}",

"region": "{{ user `aws_region` }}",

"type": "amazon-ebs",

"communicator": "ssh",

"ssh_username": "ubuntu",

"instance_type": "c5.xlarge",

"security_group_id": "sg-12345",

"iam_instance_profile": "packer-role-profile",

"ami_name": "packer-jenkins-master-{{timestamp}}",

"ami_description": "Jenkins master image",

"launch_block_device_mappings": [{

"device_name": "/dev/sda1",

"volume_size": 50,

"volume_type": "gp2",

"delete_on_termination": true

}],

"source_ami_filter": {

"filters": {

"virtualization-type": "hvm",

"name": "ubuntu/images/*ubuntu-bionic-18.04-amd64-server-*",

"root-device-type": "ebs"

},

"owners": ["099720109477"],

"most_recent": true

}

}],

"provisioners": [{

"type": "shell",

"environment_vars": ["VAULT_TOKEN={{ user `vault_token` }}"],

"scripts": ["packer_bootstrap.sh"]

}]

}

This is how the packer is configured

The Bootstrap file, packer ﹣ Bootstrap.sh, contains all the commands to pre install the software in AMI. For example, we use Docker, Docker compose, and vaultenv Or install the Datadog node for monitoring.

Considering the architecture of this AMI, we can use Terraform, CloudFormation, Pulumi and even Ansible. This is one of the possible architectures to use Jenkins on AWS.

Users access Jenkins through internal LB and GitHub webhook through public lb.

The Jenkins we use integrate GitHub, so we should provide some Jenkins URL s for GitHub through the external network. There are many operational options( IP white list , URL or token whitelist, etc.) and we combine Cloudfront to allow path and token validation.

After doing these things, we have a ready-made infrastructure with AMI. The Vault that provides the possibility for monitoring and for obtaining company credentials is also available.

Using Docker to manage Jenkins and its plug-in version  

OK, the next step is Jenkins and plug-ins. We had a lot of problems upgrading the Jenkins plug-in before, so the main goal is to fix the version for them.

Docker helped us a lot at this time. We use Jenkins image built in advance Use it as the base image for our installation.

FROM jenkins/jenkins:2.215

ENV CASC_JENKINS_CONFIG /jenkins_configs

USER root

# Install additional packages

RUN apt update && \

apt install -y python3 python3-pip && \

pip3 install awscli jenkins-job-builder jjb-reactive-choice-param --no-cache-dir

USER jenkins

VOLUME /jenkins_configs

VOLUME /var/jenkins_home

# Install plugins

COPY plugins.txt /usr/share/jenkins/ref/

RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

We installed some additional installation packages for Job Builder, which we will use later, and passed a data volume for Jenkins and plug-ins installation.

We are https://our-jenkins-url/script The Groovy code from above pastes and saves it to plugins.txt, through which we can easily get the plug-in list.

Jenkins.instance.pluginManager.plugins.each{

plugin ->

println ("${plugin.getShortName()}:${plugin.getVersion()}")

}

Finally, the Docker compose configuration runs Jenkins in Docker (we also use vaultenv to transfer credentials from Vault to Docker compose):

version: "3"

services:

jenkins:

build: .

container_name: jenkins

restart: always

ports:

- "50000:50000"

- "8080:8080"

volumes:

- ./configs/:/jenkins_configs/:ro

- ./jenkins_home/:/var/jenkins_home/:rw

environment:

- VAULT_TOKEN

- GITHUB_TOKEN

- AWS_ACCESS_KEY_ID

- AWS_SECRET_ACCESS_KEY

- JAVA_OPTS=-Xms4G -Xmx8G -Xloggc:/var/jenkins_home/gc-%t.log -XX:NumberOfGCLogFiles=5 -XX:+UseGCLogFileRotation -XX:GCLogFileSize=20m -XX:+PrintGC -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintHeapAtGC -XX:+PrintGCCause -XX:+PrintTenuringDistribution -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy -XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1

volumes:

configs:

driver: local

jenkins_home:

driver: local

It's important to remember that some Java parameters can help us deal with some garbage collection and resource limitations. This article It's great for tuning Jenkins.

Of course, we can run Jenkins locally to install the plug-in, or a new version of Jenkins. It's also great.

Now we have Jenkins with the plug-in installed, and even we can run it locally and easily deploy it to the production environment. Let's add more configuration to it.

Configure Jenkins as a code (JCasC) plug-in for the primary node  

Jenkins configuration is code( JCasC )The plug-in stores the configuration in a readable format.

This plug-in allows us to describe security configuration, credentials, plug-in configuration, nodes, views, and many other things.

This YAML configuration file is divided into five parts:

  • credentials (system credential description)

  • jenkins (authorization, cloud settings, global parameters, nodes, security domains, and views)

  • Security (global security configuration, such as script permissions)

  • tool (configuration of external tools, such as git, allure, etc.)

  • unclassified (other configurations, such as Slack integration)

We can import the configuration from the existing Jenkins installation process

It supports Different credential providers Used to manage credentials, but we also need to use environment variables.

credentials:

system:

domainCredentials:

- credentials:

- usernamePassword:

description: "AWS credentials"

id: "aws-creds"

password: ${AWS_SECRET_ACCESS_KEY}

scope: GLOBAL

username: ${AWS_ACCESS_KEY_ID}

- string:

description: "Vault token"

id: "vault-token"

scope: GLOBAL

secret: ${VAULT_TOKEN}

...

We will also Amazon EC2 plug in For bootstrap agent on AWS, its configuration can also be managed using this plug-in. Matrix based authorization allows us to manage the permissions of users in a code way.

jenkins:

authorizationStrategy:

projectMatrix:

permissions:

- "Overall/Administer:steve.a@example.org"

- "Credentials/View:john.d@example.org"

...

clouds:

- amazonEC2:

cloudName: "AWS"

privateKey: ${EC2_PRIVATE_KEY}

region: "${AWS_REGION}"

templates:

- ami: "ami-12345678"

amiType:

unixData:

sshPort: "22"

connectionStrategy: PRIVATE_IP

deleteRootOnTermination: true

description: "jenkins_agent"

idleTerminationMinutes: "20"

instanceCapStr: "100"

minimumNumberOfInstances: 0

mode: EXCLUSIVE

numExecutors: 1

remoteAdmin: "jenkins"

remoteFS: "/home/jenkins"

securityGroups: "sg-12345678"

subnetId: "subnet-12345678"

type: C52xlarge
...

We also use a lot of cool things. If we have a process to test Jenkins' local changes, we can find and fix bug s before putting them into production.

So we installed a reusable Jenkins configuration, and last but not least our task

Integrating Job Builder for freestyle tasks  

When we talk about freestyle tasks, there are several different ways to create them in Jenkins:

  • Using the GUI (the easiest way to do this, just click on it)

  • Direct use REST API

  • Use similar Job DSL Or plug-in of JJB wrapper

Jenkins Job Builder (JJB) allows us to configure tasks into a readable text format (YAML or JSON). It's very comfortable to use SCM to manage these tasks. Basically, we can use JJB to create a CI/CD process for our CI/CD tool.

.

├── config.ini

├── jobs

│ ├── Job1.yaml

│ | ...

│ └── Job2.yaml

└── scripts

├── job1.sh

| ...

└── job2.sh

Here, we can describe it in a Job1.yaml file Definition of tasks , the task steps are in the script (for example, job1.sh).

- job:

name: Job1

project-type: freestyle

auth-token: mytoken

disabled: false

concurrent: false

node: jenkins_agent

triggers:

- timed: '0 3 * * *'

builders:

- shell:

!include-raw: ../scripts/job1.sh

This is a To configure Examples of documents:

$ cat config.ini

[job_builder]

ignore_cache=True

exclude=jobs/Job2

[jenkins]

url=https://jenkins.example.org

user=some_user

password=some_password

$ jenkins-jobs --conf config.ini test -r jobs/

$ jenkins-jobs --conf config.ini update -r jobs/

It should be easy to run in the jenkins task upgrade command

Of course, our Jenkins users need permission to create and configure tasks. We just need to run a subtask on the primary node to import from JJB to all Jenkins configurations.

JJB is not a panacea, because there are still some plug-ins that are not very commonly used that are not supported. But it's still a very flexible plug-in. In addition, it can use macroses Configure.

conclusion  

Now that we have seen a general outline of the "everything is code" pattern and how we use Jenkins, we can go back to the questions mentioned at the beginning of the article. Have you found the answer? Perhaps, it is obvious that the answer to all five questions is "yes".

We just want to share with you our experience in parameter configuration and Jenkins best practices We did not go into depth.

Posted by GregArtemides on Wed, 08 Apr 2020 02:15:11 -0700