Sometimes we need to expose some services without any security authentication mechanism in Kubernetes, such as Kibana without xpack, Jenkins service without login authentication, etc. We also want to access through domain name, which is more convenient than domain name. More importantly, for services in Kubernetes, it is too convenient to expose a service through Ingress. It can also automatically complete HTTPS through cert-manager. So it is very necessary to verify the security of these services.

Basic Auth Certification

In our previous upgrade to Dashboard, we mentioned two ways to add Basic Auth authentication to our services: haproxy/nginx and traefik/nginx-ingress.

Using haproxy/nginx is very simple, that is, adding basic auth authentication directly, and then forwarding the request to the later service; while traefik/nginx-ingress directly provides basic auth support, we use nginx-ingress to add a basic auth authentication service for Jenkins service.

First, we need to create an htpasswd file for storing user names and passwords:

$ htpasswd -bc auth admin admin321
Adding password for user admin

Then, create a Secret object based on the above htpasswd file:

$ kubectl create secret generic jenkins-basic-auth --from-file=auth -n kube-ops
secret "jenkins-basic-auth" created

Finally, we need to add auth-type: basic and auth-jenkins-basic-auth annotations to the Ingress object: (ingress.yaml)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins
  namespace: kube-ops
  annotations:
    kubernetes.io/ingress.class: nginx
    # Authentication type
    nginx.ingress.kubernetes.io/auth-type: basic
    # Secret name containing user/password
    nginx.ingress.kubernetes.io/auth-secret: jenkins-basic-auth
    # Display an appropriate context information when authenticating
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - admin'
spec:
  rules:
  - host: jenkins.qikqiak.com
    http:
      paths:
      - backend:
          serviceName: jenkins
          servicePort: web

Then update the resource object above:

$ kubectl apply -f ingress.yaml
ingress.extensions "jenkins" configured

Now that the update is complete, we can go to our Jenkins service and see the prompt information that we need to enter the username and password:

OAuth authentication

In addition to the Basic Auth authentication method mentioned above, we can also authenticate through the OAuth services provided by Github, Google, etc. We can call it OAuth2 Proxy Tools to proxy requests by providing a reverse proxy for external authentication are relatively simple to use.

install

First we need to add automated HTTPS to our application, referring to our previous article. Implementing Kubernetes Ingress Automation HTTPS with Let's Encrypt.

Then log in to Github, at https://github.com/settings/applications/new Add a new OAuth application:

Replace it with the domain name you need to use, then add / oauth2/callback to the callback URL, click Register, and record the value of Client ID and Client Secret on the application details page. Then you need to generate a cookie key. Of course, if we install a python environment in our system, we can generate it directly. If we don't, we can run it with a Docker container.

$ docker run -ti --rm python:3-alpine \
    python -c 'import secrets,base64; print(base64.b64encode(base64.b64encode(secrets.token_bytes(16))));'
b'<GENERATED_COOKIE_SECRET>'

Then deploy the OAuth2 Proxy application, where we use Helm directly to simplify the installation:

$ helm install --name authproxy \
    --namespace=kube-system \
    --set config.clientID=<YOUR_CLIENT_ID> \
    --set config.clientSecret=<YOUR_SECRET> \
    --set config.cookieSecret=<GENERATED_COOKIE_SECRET> \
    --set extraArgs.provider=github \
    --set extraArgs.email-domain="*" \
    stable/oauth2-proxy
NAME:   authproxy
LAST DEPLOYED: Sun Apr 14 01:11:50 2019
NAMESPACE: kube-system
STATUS: DEPLOYED

RESOURCES:
==> v1/Secret
NAME                    TYPE    DATA  AGE
authproxy-oauth2-proxy  Opaque  3     0s

==> v1/ConfigMap
NAME                    DATA  AGE
authproxy-oauth2-proxy  1     0s

==> v1/Service
NAME                    TYPE       CLUSTER-IP      EXTERNAL-IP  PORT(S)  AGE
authproxy-oauth2-proxy  ClusterIP  10.109.110.219  <none>       80/TCP   0s

==> v1beta2/Deployment
NAME                    DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
authproxy-oauth2-proxy  1        0        0           0          0s

==> v1/Pod(related)
NAME                                     READY  STATUS             RESTARTS  AGE
authproxy-oauth2-proxy-798cff85fc-pc8x5  0/1    ContainerCreating  0         0s


NOTES:
To verify that oauth2-proxy has started, run:

  kubectl --namespace=kube-system get pods -l "app=oauth2-proxy"

$ # Execute the following command to verify that the installation is successful.
$ kubectl --namespace=kube-system get pods -l "app=oauth2-proxy"
NAME                                     READY     STATUS    RESTARTS   AGE
authproxy-oauth2-proxy-cdb4f675b-wvdg5   1/1       Running   0          1m

test

We also use a Jenkins service here, and you can use any service to authenticate. Of course, it's better to have no authentication function, such as Kibana without x-pack installed.

The key to achieving external service authentication is that nginx-ingress-controller provides auth-url and auth-signin annotations to allow us to configure the entry of external authentication.

We configure the url of service authentication in the core Ingress object of Jenkins: https://$host/oauth2/auth, and then proxy the oauth2 path to the OAuth2 proxy application by creating a homologous Ingress object to process the authentication service:

nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"

Then recreate the two Ingress objects of Jenkins as follows:

$ cat <<EOF | kubectl apply -f -
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: jenkins
  namespace: kube-ops
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
    nginx.ingress.kubernetes.io/auth-url: "https://$host/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://$host/oauth2/start?rd=$escaped_request_uri"
spec:
  rules:
  - host: jenkins.qikqiak.com
    http:
      paths:
      - backend:
          serviceName: jenkins
          servicePort: web
        path: /
  tls:
  - hosts:
    - jenkins.qikqiak.com
    secretName: jenkins-tls

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: authproxy-oauth2-proxy
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "true"
spec:
  rules:
  - host: jenkins.qikqiak.com
    http:
      paths:
      - backend:
          serviceName: authproxy-oauth2-proxy
          servicePort: 80
        path: /oauth2
  tls:
  - hosts:
    - jenkins.qikqiak.com
    secretName: jenkins-tls
EOF

Here we add HTTPS to the service automatically through cert-manager, add the annotation of kubernetes.io/tls-acme=true, and then we open our Jenkins service in the browser, which will normally jump to the GitHub login page:

Then we can jump to our Jenkins service after the certification has passed.

Of course, in addition to GitHub, other OAuth authentication services, such as Google, can be added as needed.

Related links

• OAuth2 Proxy
• ingress-nginx oauth external auth exmaple
• First Experience of Kubernetes Helm
• Implementing Kubernetes Ingress Automation HTTPS with Let's Encrypt