Custom OpenShift s2i image and template -- OracleJDK8

Keywords: Docker

Objective of this paper

Because the image and template (OpenJDK8) officially provided by OpenShift does not fully meet the business needs:

  • Flight recording function is not included. Only OpenJDK11 and above are open source by Oracle

  • The generated heap dump is very large and slow (the minimum JVM heap of the company project is 16G and the maximum is 32G), the export compression transmission is long (more than half an hour), and the business pause time is too long

  • There are few commands built into the image to obtain performance indicators, and there are no conventional network status query commands such as netstat and ping

  • The internal s2i process of the image is too complex to write, and the image construction process has too much cache and is not removed, resulting in the general excessive volume of the final image (for 350M business applications, the image is 1.1G...)

In order to facilitate locating performance problems, the author carefully read the s2i construction process, image customization and other relevant documents of OpenShift v3.11, and customized his own template and image. At present, it has been verified in the production environment. Here are some records. If it helps you, please pay attention and praise before you go~

The main idea of this article is to use the s2i tool to create a basic build debugging environment, modify the Dockerfile and build script to generate an image, and then push the image and template to the previous article OpenShift local development environment configuration (based on Minishift) In the environment, test the final effect with a demo.

Custom s2i mirror section

Installation s2i tool

Official Release Page of s2i tool: https://github.com/openshift/source-to-image/releases

#Download s2i tools
wget https://github.com/openshift/source-to-image/releases/download/v1.3.1/source-to-image-v1.3.1-a5a77147-linux-amd64.tar.gz
#Unzip the s2i tool to / usr/bin
sudo tar zxf source-to-image-v1.3.1-a5a77147-linux-amd64.tar.gz -C /usr/bin
#View s2i version
s2i version

Initialize s2i directory structure

The author started the operation in the ~ / WorkSpace/openshift directory. Let's say it in advance so that readers don't make mistakes.

#Generate s2i script directory structure. The create parameter is create, the third parameter is image name, and the third parameter is generated directory name
s2i create oracle-jdk-8 oracle-jdk-8-catelog
#Enter the generated directory
cd oracle-jdk-8-catelog
#View basic directory structure
tree -L 3
.
├── Dockerfile #Image Dockerfile
├── Makefile #Makefile build script
├── README.md
├── s2i
│   └── bin #s2i script directory
│       ├── assemble #Assembly script
│       ├── run #Run script
│       ├── save-artifacts #Optional, save the artifact script and use it in incremental construction
│       └── usage #Instruction script
└── test
    ├── run #Test run script
    └── test-app #Build the source code (in this case, it can be a jar package)
        └── index.html

Here, the author does not use S2i / bin / save artifacts, because there is no need to build incrementally and delete them; Test / test app / index.html is also unavailable. Delete it.

rm -rf s2i/bin/save-artifacts
rm -rf test/test-app/index.html

Download and unzip the OracleJDK8 installation package, which is moderately streamlined

Since the author has downloaded the jdk8 installation package, download the paid version of OracleJDK and the previous version, please refer to My blog Here, you can directly unzip it from ~ / Downloads to ~ / workspace / openshift / oracle-jdk-8-catalog

#Unzip the jdk installation package
tar zxvf ~/Downloads/jdk-8u181-linux-x64.tar.gz -C ~/WorkSpace/openshift/oracle-jdk-8-catelog
#Enter the jdk decompression directory
cd /home/hellxz/WorkSpace/openshift/oracle-jdk-8-catelog/jdk1.8.0_181
#Moderately streamlined, delete the source package, and can be 26M smaller
rm -rf javafx-src.zip src.zip
#Return to the directory of template initialization
cd ..

Modify Dockerfile

vim Dockerfile, delete the default and add the following content:

FROM debian:buster

ENV BUILDER_VERSION 1.0

# Set labels used in OpenShift to describe the builder image
LABEL description="Source To Image (S2I) image for Hellxz Providing Oracle JDK 8" \
      maintainer="Hellxz Zhang <hellxz@foxmail.com>" \
      io.k8s.description="Platform for building and running Java applications with Oracle JDK 8" \
      io.k8s.display-name="Java Applications" \
      io.openshift.expose-services="8080:http" \
      io.openshift.tags="builder,java" \
      io.openshift.s2i.scripts-url="image:///usr/libexec/s2i" \
      io.openshift.s2i.destination="/tmp"

# Copy Oracle JDK8
COPY ./jdk1.8.0_181 /usr/lib/jvm

# Copy scripts to s2i build path, must same as `io.openshift.s2i.scripts-url` label specified.
COPY ./s2i/bin/ /usr/libexec/s2i

# Environments
ENV JAVA_HOME="/usr/lib/jvm" \
    PATH="/usr/lib/jvm/bin:${PATH}" \
    TZ="Asia/Shanghai"

# Create oraclejdk user with home dir /deployments
# Grant user exec privilege
# Set apt mirror and install some utils then clean 
# Generate UTF-8 locales
RUN mkdir /deployments && \
    useradd -M -d /deployments -u 1001  -s /bin/bash oraclejdk8 && \
    chown oraclejdk8 /deployments && chmod 777 /deployments && \
    bash -c "echo -e \"deb http://mirrors.163.com/debian/ buster main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-updates main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-backports main non-free contrib\ndeb http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\" > /etc/apt/sources.list" && \
    apt-get update -y && apt-get install -y net-tools locales procps && \
    apt-get clean all -y && rm -rf /var/lib/apt/lists/* && \
    sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen

# To avoid build image error tips set LC env after locale.gen
ENV LC_ALL="en_US.UTF-8"

# Switch user
USER 1001

# Switch WorkDir
WORKDIR /deployments

EXPOSE 8080 8443

# Set the default CMD for the image
CMD ["/usr/libexec/s2i/usage"]

What does the Dockerfile above do?

  • Based on very stable Debian10
  • Label the image, which will be read during s2i construction
  • Copy Oracle JDK directory and s2i script
  • Configure JDK environment variables, UTF-8 character set, and specify China time zone
  • Create deployment directory, create running users, change domestic sources, and install necessary tools for collecting performance indicators (netstat and ps)
  • Specify working directory, running user, default exposed port, default CMD entry

The modification of Dockerfile is not finished yet. We also need to modify s2i build and run scripts to adapt to the image configuration just modified

Modify s2i script

The s2i script is in the s2i/bin directory. We need to modify the assembly script to change the assembly process, and modify the run script to replace the startup command.

Modify assembly

vim s2i/bin/assemble

#!/bin/bash -e
#
# S2I assemble script for the 'oracle-jdk-8' image.
# The 'assemble' script builds your application source so that it is ready to run.
#
# For more information refer to the documentation:
#	https://github.com/openshift/source-to-image/blob/master/docs/builder_image.md
#

# Prevent some horror bugs. example: rm -rf ${SOME_UNEXISTS_VARIABLE}/* 
set -e

# If the 'oracle-jdk-8' assemble script is executed with the '-h' flag, print the usage.
if [[ "$1" == "-h" ]]; then
	exec /usr/libexec/s2i/usage
fi

echo "---> Moving Artifact from source..."
mv /tmp/src/*.jar /deployments/

echo "---> Clearing tmp dir..."
rm /tmp/src/.git -rf

echo "<--- Build Success..."

The above script mainly does two things: one is to copy the jar package from / tmp/src to the deployment directory. In the previous article, we introduced the workflow of s2i. During construction, we will pull the product or source code from git to / tmp/src; The other is to delete the / tmp/src/.git directory. The reason is that git's. Git directory will also save the compressed product or source file. Delete it to reduce the image volume.

Modify run

vim s2i/bin/run

#!/bin/bash -e
#
# S2I run script for the 'oracle-jdk-8' image.
# The run script executes the server that runs your application.
#
# For more information see the documentation:
#	https://github.com/openshift/source-to-image/blob/master/docs/builder_image.md
#

exec java ${JVM_OPTS} -Djava.security.egd=file:/dev/./urandom -jar /deployments/*.jar

The run script is only used for startup. In order to facilitate the transmission of JVM tuning parameters, the JVM is customized_ The opts environment variable is available for use. The random number configuration in the parameter can improve the ability of the service to generate random numbers and improve the startup speed.

Modify Makefile

The purpose of modifying Makefile is to simplify the command of building image. The author has modified the image name and added the parameter of pushing private image warehouse. In this regard, npm is a bit similar to Makefile.

IMAGE_NAME = 192.168.99.1:5000/oracle-jdk-8

.PHONY: test
test:
	docker build -t $(IMAGE_NAME)-candidate .
	IMAGE_NAME=$(IMAGE_NAME)-candidate test/run

.PHONY: push
push:
	docker build -t $(IMAGE_NAME) .
	docker push $(IMAGE_NAME)

For more ways to write Makefile, please refer to "write makefile with me" written by left ear mouse Chen Hao

Write test scripts

The test/run test script is equivalent to completing the process of s2i to deployment. First prepare the test image, and then run the s2i build process incremental build (assemble the test product program and the image, and call the assembly script inside the container), then start the container, run the scheduled test script to access the container port address, and it will be successful.

In the default test script, whether the test service is available is the access address to determine whether the return value is 200 status code. I provide a demo of springboot. After deployment, access the / test endpoint to return the results. First build the demo:

#Clone test demo
git clone https://github.com/hellxz/cicd-demo.git test/test-app
#Manual build
cd test/test-app
mvn package -DskipTests
#Copy the jar package and delete anything unrelated to this article
mv target/*.jar .
rm -rf *.sh *file *.xml *.md src target .git *.yaml .gitignore .mvn
#Go back to the directory where Makefile is located
cd ../..

Modify the test/run script, adjust the detection interval and the accessed interface address. In this demo, you need to modify 116 behavior 5 and add / test to the 120 and 121 access paths

Perform test

make test

We can see the line "starting cicddemoapplication v0.0.1 using java 1.8.0" under the banner of springboot_ 181 on f770c60456f4 with pid 1 (/ deployments / cicd-demo-0.0.1. Jar started by oraclejdk8 in / deployments), which can prove that the tested demo is indeed run successfully by oraclejdk8 user, and the pid is 1. Looking down again, there is no error detection information, indicating that the test has passed and the Minishift test can be used.

Push image to warehouse

#Execute the build push command defined in Makefile
make push

hellxz@debian:~/WorkSpace/openshift/oracle-jdk-8-catelog$ make push
docker build -t 192.168.99.1:5000/oracle-jdk-8 .
Sending build context to Docker daemon  389.2MB
Step 1/12 : FROM debian:buster
 ---> 2b6f409b1d24
Step 2/12 : ENV BUILDER_VERSION 1.0
 ---> Using cache
 ---> 46de72742080
Step 3/12 : LABEL description="Source To Image (S2I) image for Hellxz Providing Oracle JDK 8"       maintainer="Hellxz Zhang <hellxz@foxmail.com>"       io.k8s.description="Platform for building and running Java applications with Oracle JDK 8"       io.k8s.display-name="Java Applications"       io.openshift.expose-services="8080:http"       io.openshift.tags="builder,java"       io.openshift.s2i.scripts-url="image:///usr/libexec/s2i"       io.openshift.s2i.destination="/tmp"
 ---> Using cache
 ---> fcbadb0d0d39
Step 4/12 : COPY ./jdk1.8.0_181 /usr/lib/jvm
 ---> Using cache
 ---> ebb551cc9d4c
Step 5/12 : COPY ./s2i/bin/ /usr/libexec/s2i
 ---> Using cache
 ---> 5f08421b9527
Step 6/12 : ENV JAVA_HOME="/usr/lib/jvm"     PATH="/usr/lib/jvm/bin:${PATH}"     TZ="Asia/Shanghai"
 ---> Using cache
 ---> 3c663a5370c8
Step 7/12 : RUN mkdir /deployments &&     useradd -M -d /deployments -u 1001  -s /bin/bash oraclejdk8 &&     chown oraclejdk8 /deployments && chmod 777 /deployments &&     bash -c "echo -e \"deb http://mirrors.163.com/debian/ buster main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-updates main non-free contrib\ndeb http://mirrors.163.com/debian/ buster-backports main non-free contrib\ndeb http://mirrors.163.com/debian-security/ buster/updates main non-free contrib\" > /etc/apt/sources.list" &&     apt-get update -y && apt-get install -y net-tools locales procps &&     apt-get clean all -y && rm -rf /var/lib/apt/lists/* &&     sed -i '/en_US.UTF-8/s/^# //g' /etc/locale.gen && locale-gen
 ---> Using cache
 ---> 5e9c181bb69b
Step 8/12 : ENV LC_ALL="en_US.UTF-8"
 ---> Using cache
 ---> 60091bb63284
Step 9/12 : USER 1001
 ---> Using cache
 ---> f8e6a2e96a8a
Step 10/12 : WORKDIR /deployments
 ---> Using cache
 ---> 66f6925e3ab6
Step 11/12 : EXPOSE 8080 8443
 ---> Using cache
 ---> bda814465820
Step 12/12 : CMD ["/usr/libexec/s2i/usage"]
 ---> Using cache
 ---> f30388d3181c
Successfully built f30388d3181c
Successfully tagged 192.168.99.1:5000/oracle-jdk-8:latest
docker push 192.168.99.1:5000/oracle-jdk-8
Using default tag: latest
The push refers to repository [192.168.99.1:5000/oracle-jdk-8]
16970671df83: Pushed 
97a51654d23f: Pushed 
16fd63c1b14c: Pushed 
d6a325d281f2: Pushed 
latest: digest: sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79 size: 1160

Import images into OpenShift internal image warehouse

Here, the Minishift import image to the internal warehouse is consistent with OpenShift. It can also be said that OpenShift.

#The oc command line tool is required for import. Minisshift is provided by default, but it is not added to Path. You can temporarily add oc to Path by executing the following command
eval (minishift oc-env) #If an error is reported, only the commands in parentheses will be executed, and the corresponding commands will be executed according to the prompts
oc import-image 192.168.99.1:5000/oracle-jdk-8:latest --confirm --insecure

hellxz@debian:~/WorkSpace/openshift$ oc import-image 192.168.99.1:5000/oracle-jdk-8:latest --confirm --insecure
imagestream.image.openshift.io/oracle-jdk-8 imported

Name:			oracle-jdk-8
Namespace:		myproject
Created:		1 second ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2021-11-21T14:24:57Z
Docker Pull Spec:	172.30.1.1:5000/myproject/oracle-jdk-8
Image Lookup:		local=false
Unique Images:		1
Tags:			1

latest
  tagged from 192.168.99.1:5000/oracle-jdk-8:latest
    will use insecure HTTPS or HTTP connections

  * 192.168.99.1:5000/oracle-jdk-8@sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79
      1 second ago

Image Name:	oracle-jdk-8:latest
Docker Image:	192.168.99.1:5000/oracle-jdk-8@sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79
Name:		sha256:904e52bcfe7c4ea8a156de4bd9cb073770f4af703ac6acb59a9180c272d41b79
Created:	1 second ago
Annotations:	image.openshift.io/dockerLayersOrder=ascending
Image Size:	221.8MB in 4 layers
Layers:		50.44MB	sha256:07471e81507f7cf1100827f10c60c3c0422d1222430e34e527d97ec72b14a193
		161.8MB	sha256:98b418031cdbc51e52d88a2632516069e753d5293ec440dae46163a45880492a
		899B	sha256:61addb0f8207e85f3a734299eda8b8afae987076de0cf67b60f33c9c9846f6b6
		9.598MB	sha256:09daea12eb7b9cd76b4629e22d566a6646eb6e19ea3c92706c7bc41bf0285384
Image Created:	30 minutes ago
Author:		<none>
Arch:		amd64
Command:	/usr/libexec/s2i/usage
Working Dir:	/deployments
User:		1001
Exposes Ports:	8080/tcp, 8443/tcp
Docker Labels:	description=Source To Image (S2I) image for Hellxz Providing Oracle JDK 8
		io.k8s.description=Platform for building and running Java applications with Oracle JDK 8
		io.k8s.display-name=Java Applications
		io.openshift.expose-services=8080:http
		io.openshift.s2i.destination=/tmp
		io.openshift.s2i.scripts-url=image:///usr/libexec/s2i
		io.openshift.tags=builder,java
		maintainer=Hellxz Zhang <hellxz@foxmail.com>
Environment:	PATH=/usr/lib/jvm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
		BUILDER_VERSION=1.0
		JAVA_HOME=/usr/lib/jvm
		TZ=Asia/Shanghai
		LC_ALL=en_US.UTF-8

If an error is reported, give priority to whether you have logged in to oc, oc login -u developer, and then try to import after execution.

Custom s2i template section

Define template file

The template content is directly provided here. It is modified based on the openjdk8 template:

oraclejdk-8-basic-s2i.yaml

apiVersion: template.openshift.io/v1
kind: Template
message: A new java application has been created in your project.
metadata:
  annotations:
    description: An Java application using Oracle JDK 8.
    iconClass: icon-rh-openjdk
    openshift.io/display-name: OracleJDK 8 by Hellxz
    openshift.io/provider-display-name: Hellxz Zhang <hellxz001@foxmail.com>.
    tags: java
    template.openshift.io/long-description: This template defines resources needed to develop Oracle JDK 8 Java based application.
    template.openshift.io/support-url: hellxz001@foxmail.com
    version: 0.0.1
  labels:
    template: oraclejdk-8-basic-s2i
    xpaas: 0.0.1
  name: oraclejdk-8-basic-s2i
objects:
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      description: The application's http port.
    labels:
      application: ${APPLICATION_NAME}
    name: ${APPLICATION_NAME}
  spec:
    ports:
    - port: 8080
      targetPort: 8080
    selector:
      deploymentConfig: ${APPLICATION_NAME}
- apiVersion: v1
  id: ${APPLICATION_NAME}-http
  kind: Route
  metadata:
    annotations:
      description: Route for application's http service.
    labels:
      application: ${APPLICATION_NAME}
    name: ${APPLICATION_NAME}
  spec:
    host: ${HOSTNAME_HTTP}
    to:
      name: ${APPLICATION_NAME}
- apiVersion: v1
  kind: ImageStream
  metadata:
    labels:
      application: ${APPLICATION_NAME}
    name: ${APPLICATION_NAME}
- apiVersion: v1
  kind: BuildConfig
  metadata:
    labels:
      application: ${APPLICATION_NAME}
    name: ${APPLICATION_NAME}
  spec:
    output:
      to:
        kind: ImageStreamTag
        name: ${APPLICATION_NAME}:latest
    source:
      contextDir: ""
      git:
        ref: ${SOURCE_REPOSITORY_REF}
        uri: ${SOURCE_REPOSITORY_URL}
      type: Git
    strategy:
      sourceStrategy:
        forcePull: true
        from:
          kind: ImageStreamTag
          name: oracle-jdk-8:latest
          namespace: ${IMAGE_STREAM_NAMESPACE}
      type: Source
    triggers:
    - imageChange: {}
      type: ImageChange
    - type: ConfigChange
- apiVersion: v1
  kind: DeploymentConfig
  metadata:
    labels:
      application: ${APPLICATION_NAME}
    name: ${APPLICATION_NAME}
  spec:
    replicas: 1
    selector:
      deploymentConfig: ${APPLICATION_NAME}
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          application: ${APPLICATION_NAME}
          deploymentConfig: ${APPLICATION_NAME}
        name: ${APPLICATION_NAME}
      spec:
        containers:
        - env: []
          image: ${APPLICATION_NAME}
          imagePullPolicy: Always
          name: ${APPLICATION_NAME}
          ports:
          - containerPort: 8778
            name: jolokia
            protocol: TCP
          - containerPort: 8080
            name: http
            protocol: TCP
          - containerPort: 8443
            name: https
            protocol: TCP
        terminationGracePeriodSeconds: 75
    triggers:
    - imageChangeParams:
        automatic: true
        containerNames:
        - ${APPLICATION_NAME}
        from:
          kind: ImageStreamTag
          name: ${APPLICATION_NAME}:latest
      type: ImageChange
    - type: ConfigChange
parameters:
- description: The name for the application.
  displayName: Application Name
  name: APPLICATION_NAME
  required: true
  value: oraclejdk8-app
- description: Git source URI for application
  displayName: Git Repository URL
  name: SOURCE_REPOSITORY_URL
  required: true
- description: Git branch/tag reference
  displayName: Git Reference
  name: SOURCE_REPOSITORY_REF
  value: master
- description: 'Custom hostname for http service route.  Leave blank for default hostname,
    e.g.: <application-name>-<project>.<default-domain-suffix>'
  displayName: Custom http Route Hostname
  name: HOSTNAME_HTTP
- description: Namespace in which the ImageStreams for Red Hat Middleware images are
    installed. These ImageStreams are normally installed in the openshift namespace.
    You should only need to modify this if you've installed the ImageStreams in a
    different namespace/project.
  displayName: ImageStream Namespace
  name: IMAGE_STREAM_NAMESPACE
  required: true
  value: myproject

Brief description of template structure:

  • The definition format is based on k8s and is the Template API type of red hat extension
  • Source data - records the contents of the template, which will be displayed on the UI that opens the template
  • objects - all object definitions from various build to deployment processes
    • Service - k8s native service object
    • Route - Route Object unique to openshift
    • ImageStream - a mirror stream object unique to openshift
    • Buildconfig - a build configuration object unique to openshift
    • Deploymentconfig - the Deployment configuration object unique to openshift, which generates the Deployment object in k8s
  • Parameters - you can modify parameters or variables to be referenced by various objects in objects to generate a series of associated objects

Import template to OpenShift

oc apply -f oraclejdk-8-basic-s2i.yaml

hellxz@debian:~/WorkSpace/openshift$ oc apply -f=oraclejdk-8-basic-s2i.yaml 
template.template.openshift.io/oraclejdk-8-basic-s2i created

Deploying Java programs using custom templates and images

Preparatory work

Since OpenShift is built by pulling products or source code from Git warehouse, it is necessary to upload the test program (cicd-demo-0.0.1.jar used above) to Git version control, which is used here Last article Gitea configured in the Minishift development environment in (you need to create an account and warehouse in advance. The warehouse created here is called demo and the user name is hellxz).

#Enter the directory where the test program jar is located
cd ~/WorkSpace/openshift/oracle-jdk-8-catelog/test/test-app
#Initialize the current directory to git warehouse
git init
#Add the remote warehouse address (make some modifications according to the actual operation)
git remote add origin git remote add origin http://localhost:3000/hellxz/demo.git
#Set user name and email
git config --local user.name hellxz
git config --local user.email hellxz001@foxmail.com
#Submit
git add .
git commit -m "init"
#Push remote warehouse
git push --set-upstream origin master

As shown in the figure below, the push has been completed.

Deploy programs using custom templates

The name of the user-defined image is referenced in the template, and the namespace is also myproject (Minishift is the default. The production environment can be changed to the required name and then uploaded. Adjustment can be imported multiple times)

The first deployment process is troublesome. There are multiple motion pictures recorded here

Create application

There is a problem with the external address configured by Gitea, but the problem is not big

Create a secret to pull Git

You need to tell s2i the user and password of the Git warehouse of the build script, so that you can pull down the artifact. You only need to create it when you create the application for the first time.

Modify the Build configuration yaml and add the Git warehouse secret reference

Execute the build and view the build log

View the running status of the test program

As shown in the figure, the container is already in Ready: true state, indicating that the startup state is normal.

Use the domain name to access the service

Since there is no external dns, the domain name can be replaced by modifying the / etc/hosts file

After passing the test, the user-defined image and template can work normally.

It's not easy to write. If this article is helpful to you, just pay attention and praise before you go. I'm Hellxz. I'll see you next time.

This article synchronously publishes the following two addresses. Reprinting without permission is prohibited.

Posted by phprocket on Sun, 21 Nov 2021 18:04:51 -0800

Hot Keywords

©2024 Programmer Group