In the past work, we have built a technical platform using microservices, containerization and service choreography. In order to improve the R & D efficiency of the development team, we also provide a CICD platform to quickly deploy the code to the Openshift (enterprise class Kubernetes) cluster.
The first step of deployment is the containerization of the application. The delivery of continuous integration has changed from the previous jar package and webpack to the container image. Containerization packages the software code and all required components (libraries, frameworks and running environments), so that it can run consistently on any infrastructure in any environment and "isolate" from other applications.
Our code needs to be completed in the pipeline of CICD from source code to compilation to final runnable image and even deployment. Initially, we added three files to each code warehouse and also injected them into the new project through the project generator (similar to Spring Initializer):
- Jenkins file. Groovy: used to define the Pipeline of Jenkins. There are multiple versions for different languages
- Manifest YAML: used to define Kubernetes resources, that is, descriptions of workloads and their operations
- Dockerfile: used to build objects
These three files also need to evolve in the work. At first, when there are few projects (more than a dozen), our basic team can go to each code warehouse to maintain and upgrade. With the explosive growth of the project, the cost of maintenance is becoming higher and higher. We iterated on the CICD platform, removed "Jenkinsfile.groovy" and "manifest YAML" from the project, and retained the Dockerfile with few changes.
With the evolution of the platform, we need to consider decoupling the only "nail" Dockerfile from the code, and upgrade the Dockerfile if necessary. Therefore, after investigating buildpacks, we have today's article.
What is Dockerfile
Docker automatically builds an image by reading the instructions in the dockerfile. Dockerfile is a text file that contains instructions that docker can execute to build images. We used it before Test Tekton's Java project Take Dockerfile of as an example:
FROM openjdk:8-jdk-alpine RUN mkdir /app WORKDIR /app COPY target/*.jar /app/app.jar ENTRYPOINT ["sh", "-c", "java -Xmx128m -Xms64m -jar app.jar"]
Mirror layering
You may have heard that the Docker image contains multiple layers. Each layer corresponds to each command in Dockerfile, such as RUN, COPY and ADD. Some specific instructions will create a new layer. During the image construction process, if some layers do not change, they will be obtained from the cache.
In the following build pack, image layering and cache are also used to speed up the construction of images.
What is Buildpack
BuildPack Is a program that can convert the source code into a container image and run in any cloud environment. Usually, buildpack encapsulates a single language ecological tool chain. Applicable to Java, Ruby, Go, NodeJs, Python, etc.
What is Builder?
After some buildpacks are combined in order, the builder is added in addition to buildpacks life cycle And stack container mirroring.
The stack container image consists of two images: the image build image used to run the buildpack and the basic image run image used to build the application image. As shown in the figure above, it is the running environment in builder.
How Buildpack works
Each buildpack runtime consists of two phases:
1. Detection stage
Check some specific files / data in the source code to determine whether the current buildpack is applicable. If applicable, it will enter the construction phase; Otherwise it will quit. For example:
- Java maven's buildpack will check whether there is pom.xml in the source code
- Python's buildpack will check whether there are requirements.txt or setup.py files in the source code
- Node buildpack looks for the package-lock.json file.
2. Construction stage
During the construction phase, the following operations will be performed:
- Setting up the build environment and runtime environment
- Download dependencies and compile the source code (if needed)
- Set the correct entry point and startup script.
For example:
- Java maven buildpack will execute mvn clean install -DskipTests after checking the pom.xml file
- After Python buildpack checks that there are requrements.txt, it will execute pip install -r requrements.txt
- After checking that there is package-lock.json in the Node build pack, execute npm install
Get started with BuildPack
How to use builder pack to build an image without Dockerfile. After reading the above, we can basically understand that the core is in the preparation and use of buildpack.
In fact, there are many open source Buildpacks that can be used. There is no need to write them manually without specific customization. For example, the following major manufacturers open source and maintain Buildpacks:
However, before we formally introduce the open source buildpacks in detail, we still have a deep understanding of how buildpacks work by creating our own buildpacks. As for the test project, we still use it Test Tekton's Java project.
All the following contents have been submitted to Github On, you can access: https://github.com/addozhang/buildpacks-sample Get the relevant code.
The final directory buildpacks sample structure is as follows:
├── builders
│ └── builder.toml
├── buildpacks
│ └── buildpack-maven
│ ├── bin
│ │ ├── build
│ │ └── detect
│ └── buildpack.toml
└── stacks
├── build
│ └── Dockerfile
├── build.sh
└── run
└── Dockerfile
Create buildpack
pack buildpack new examples/maven \
--api 0.5 \
--path buildpack-maven \
--version 0.0.1 \
--stacks io.buildpacks.samples.stacks.bionic
Look at the generated buildpack Maven Directory:
buildpack-maven ├── bin │ ├── build │ └── detect └── buildpack.toml
Each file is the default preliminary test data, which is of little use. Some contents need to be added:
bin/detect:
#!/usr/bin/env bash
if [[ ! -f pom.xml ]]; then
exit 100
fi
plan_path=$2
cat >> "${plan_path}" <<EOL
[[provides]]
name = "jdk"
[[requires]]
name = "jdk"
EOL
bin/build:
#!/usr/bin/env bash
set -euo pipefail
layers_dir="$1"
env_dir="$2/env"
plan_path="$3"
m2_layer_dir="${layers_dir}/maven_m2"
if [[ ! -d ${m2_layer_dir} ]]; then
mkdir -p ${m2_layer_dir}
echo "cache = true" > ${m2_layer_dir}.toml
fi
ln -s ${m2_layer_dir} $HOME/.m2
echo "---> Running Maven"
mvn clean install -B -DskipTests
target_dir="target"
for jar_file in $(find "$target_dir" -maxdepth 1 -name "*.jar" -type f); do
cat >> "${layers_dir}/launch.toml" <<EOL
[[processes]]
type = "web"
command = "java -jar ${jar_file}"
EOL
break;
done
buildpack.toml:
api = "0.5" [buildpack] id = "examples/maven" version = "0.0.1" [[stacks]] id = "com.atbug.buildpacks.example.stacks.maven"
Create stack
To build Maven project, Java and Maven environment are preferred. We use maven:3.5.4-jdk-8-slim as the base image of build image. The application needs JAVA environment when running, so openjdk:8-jdk-slim is used as the base image of run image.
Create build and run directories respectively in the stacks Directory:
build/Dockerfile
FROM maven:3.5.4-jdk-8-slim
ARG cnb_uid=1000
ARG cnb_gid=1000
ARG stack_id
ENV CNB_STACK_ID=${stack_id}
LABEL io.buildpacks.stack.id=${stack_id}
ENV CNB_USER_ID=${cnb_uid}
ENV CNB_GROUP_ID=${cnb_gid}
# Install packages that we want to make available at both build and run time
RUN apt-get update && \
apt-get install -y xz-utils ca-certificates && \
rm -rf /var/lib/apt/lists/*
# Create user and group
RUN groupadd cnb --gid ${cnb_gid} && \
useradd --uid ${cnb_uid} --gid ${cnb_gid} -m -s /bin/bash cnb
USER ${CNB_USER_ID}:${CNB_GROUP_ID}
run/Dockerfile
FROM openjdk:8-jdk-slim
ARG stack_id
ARG cnb_uid=1000
ARG cnb_gid=1000
LABEL io.buildpacks.stack.id="${stack_id}"
USER ${cnb_uid}:${cnb_gid}
Then use the following command to build two mirrors:
export STACK_ID=com.atbug.buildpacks.example.stacks.maven
docker build --build-arg stack_id=${STACK_ID} -t addozhang/samples-buildpacks-stack-build:latest ./build
docker build --build-arg stack_id=${STACK_ID} -t addozhang/samples-buildpacks-stack-run:latest ./run
Create Builder
With buildpack and stack, you can create a Builder. First, create the builder.toml file and add the following contents:
[[buildpacks]] id = "examples/maven" version = "0.0.1" uri = "../buildpacks/buildpack-maven" [[order]] [[order.group]] id = "examples/maven" version = "0.0.1" [stack] id = "com.atbug.buildpacks.example.stacks.maven" run-image = "addozhang/samples-buildpacks-stack-run:latest" build-image = "addozhang/samples-buildpacks-stack-build:latest"
Then execute the command. Note that here we use the -- pull policy if not present parameter, so we don't need to push the two images of the stack to the image warehouse:
pack builder create example-builder:latest --config ./builder.toml --pull-policy if-not-present
test
With the builder, we can use the created builder to build the image.
The -- pull policy if not present parameter is also added here to use the local builder image:
# The directory buildpacks sample is the same level as Tekton test, and execute the following commands in buildpacks sample pack build addozhang/tekton-test --builder example-builder:latest --pull-policy if-not-present --path ../tekton-test
If you see something like the following, it means that the image is successfully built (the first time you build an image may take a long time because you need to download maven dependency, and it will be very fast later. You can perform two verifications):
... ===> EXPORTING [exporter] Adding 1/1 app layer(s) [exporter] Reusing layer 'launcher' [exporter] Reusing layer 'config' [exporter] Reusing layer 'process-types' [exporter] Adding label 'io.buildpacks.lifecycle.metadata' [exporter] Adding label 'io.buildpacks.build.metadata' [exporter] Adding label 'io.buildpacks.project.metadata' [exporter] Setting default process type 'web' [exporter] Saving addozhang/tekton-test... [exporter] *** Images (0d5ac1158bc0): [exporter] addozhang/tekton-test [exporter] Adding cache layer 'examples/maven:maven_m2' Successfully built image addozhang/tekton-test
Start the container and you will see that the spring boot application starts normally:
docker run --rm addozhang/tekton-test:latest . ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v2.2.3.RELEASE) ...
summary
In fact, there are many open source Buildpacks that can be used. There is no need to write them manually without specific customization. For example, the following major manufacturers open source and maintain Buildpacks:
The contents of the above buildpacks libraries are relatively comprehensive, and the implementation will be slightly different. For example, Heroku's execution phase uses Shell scripts, while Paketo uses Golang. The latter is highly scalable, supported by the Cloud Foundry foundation and has a full-time core development team sponsored by VMware. These small modular buildpacks can be combined and extended to use different scenarios.
Of course, it's still that sentence. Writing one by yourself will make it easier to understand the working mode of Buildpack.
The article is unified in the official account of the cloud.
