How to smoothly and elegantly upgrade cert manager in Rancher 2.x?

Keywords: Kubernetes github Nginx JSON

Author:

Nassos Michas European Dynamics SA, CTO

If you are using Helm Chart provided by Rancher to install cert manager in the Kubernetes cluster managed by Rancher, you may have received a reminder from Let's Encrypt recently:

Check the log of cert manager in the cluster, and you can see that Let's Encrypt refused to update the certificate request, because "your ACME client version is too old, please upgrade to the updated version". Well, Let's start now!

Update with Helm Chart from Rancher

I think the first idea in your mind should be similar to mine: upgrade cert manager with the latest version of Helm Chart. You don't need to consider this option, because the latest version of cert manager Helm Chart provided by Rancher is version 0.5.2, so don't think about one click upgrade!

Upgrade with official Helm Chart

Reference link: https://forums.rancher.com/t/update-on-cert-manager-application-in-the-catalog/15598

The plan is simple: just remove the Helm Chart of the cert manager provided by Rancher and replace it with the chart maintained by Jetstack in Helm.

We need to be cautious before we start. From v0.5.2 to the current version of stable v0.11.0, many contents have changed. The new CRDs and the corresponding configuration format will have a profound impact on your deployment. Therefore, after upgrading, you need to update the resource definition to a new format. Fortunately, cert manager provides us with the upgrade script, which we will use later.

Remove Helm Chart provided by Rancher

1. Log in to your Rancher UI

2. Switch to the project where cert manager was originally installed (maybe System)

3. Click "APS"

4. Click the vertical ellipsis button, and then select Delete.

Now, you have removed the cert manager that you originally installed. Please note that this operation will not affect the certificate that has been created before, and your inress configuration should work as before.

Install Tiller

Tiller is the server-side component of Helm, so in order to use Helm in our CLI, we need to install tiller in the Kubernetes cluster. You can verify that tiller is installed by running the following command:

helm version

If your output is similar to the above, the Tiller is not installed. If you have completed the installation, skip this section.

OK, now let's install Tiller. First, we need to create a service account, grant us the right to install Tiller remotely, and then grant us the right to install Chart.

kubectl -n kube-system create serviceaccount tiller
kubectl create clusterrolebinding tiller \
  --clusterrole=cluster-admin \
  --serviceaccount=kube-system:tiller

To start installing the Tiller:

helm init --service-account tiller

After a few seconds, you can verify that Tiller is installed by retyping the command helm version, or by verifying your Kubernetes Tiller deployment with the following command:

kubectl -n kube-system rollout status deploy/tiller-deploy:

Install cert Manager

Before installing cert manager, we need to do the following preparations:

1. Disable resource validation to allow the webhook component of cert manager to work normally

2. Install new (v0.11.1) CRDs

3. Add Jetstack repos

kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
kubectl apply --validate=false -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml
helm repo add jetstack https://charts.jetstack.io && helm update

At this point, we are ready to install and verify cert Manager

helm install \
  --name cert-manager \
  --namespace cert-manager \
  --version v0.11.0 \
  jetstack/cert-manager
kubectl get pods --namespace cert-manager

CERT manager v0.11.0 installed successfully

Upgrade old resource references and configurations

Reference link:

https://cert-manager.io/docs/installation/upgrading/upgrading-0.10-0.11/

If you check your ingress certificate, you will find that nothing has changed. This is expected because the configuration defined by ingress used in previous versions of v0.5.2 does not apply to v0.11.0. CERT manager provides us with simple code to find which cluster resource still references the old comment:

kubectl get ingress \
      --all-namespaces \
      -o json | \
      jq '.items[] | select(.metadata.annotations| to_entries | map(.key)[] | test("certmanager")) | "Ingress resource \(.metadata.namespace)/\(.metadata.name) contains old annotations: (\( .metadata.annotations | to_entries | map(.key)[] | select( . | test("certmanager") )  ))"'

Depending on the number of your Kubernetes cluster deployments, the above list may be shorter or longer. Trying to manually change old annotations for all deployments can take quite a long time. The following CLI tools automate this process, but it does not make any changes to your cluster:


# First, Download binaries based on your platform
wget -O api-migration https://github.com/jetstack/cert-manager/releases/download/v0.11.0/api-migration-linux
# Or according to Darwin
wget -O api-migration https://github.com/jetstack/cert-manager/releases/download/v0.11.0/api-migration-darwin

# Mark the binary as executable and run the binary against the cluster
chmod +x api-migration && ./api-migration --kubeconfig /path/to/my/kubeconfig.yaml

# View the output of the CLI and check for differences in the file
diff ingress.yaml ingress-migrated.yaml

# Finally, after review ing the new ingress resources, apply manifest
kubectl apply -f ingress-migrated.yaml --kubeconfig /path/to/my/kubeconfig.yaml

Make sure to update all Ingress resources to keep your certificate up to date.

Reintroduce cluster Issuer

Now that we're almost done, the last step is to reintroduce the cluster Issuer (if you only want to change the kind comment to Issuer, you can also select the Issuer for each namespace).

Use Let's Encrypt stage, Production and HTTP01 to create two cluster issuers. Here is the code summary:


---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    email: example@example.com
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
    - http01:
        ingress:
          class: nginx
---
apiVersion: cert-manager.io/v1alpha2
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: example@example.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
    - http01:
        ingress:
          class: nginx

After a minute or two, all of your ingress will be updated to point to the newly issued certificate. But remember, if your previous certificate is not in the renewal window, you will not find any differences.

PS: for the application of Rancher itself, the maximum version supported by cert manager is v0.9. If it is its own application service, it can support the latest version.

PPS: due to well-known reasons, it is not recommended to use cert manager in China. It is recommended to use self signed certificate with 10-year validity.

At 20:30 tonight, the last online training course, Rancher 2.3 cluster templates and new features, will be on time! Cluster template function, Google authentication, decoupling with k8s, k8s1.16 support One class is unlocked! Copy the following address to register in the browser:

http://www.itdks.com/Home/Live/detail?id=28601

Posted by mewhocorrupts on Fri, 06 Dec 2019 21:59:40 -0800