Enabling Auto-TLS on EnRoute with Jack’s Cert Manager

Enabling Auto-TLS on EnRoute with Jack’s Cert Manager

Saaras Inc. April 16, 2021
Enabling Auto-TLS on EnRoute with Jack’s Cert Manager

Introduction - Jack’s Cert Manager

Jack’s Cert Manager is a Kubernetes package that helps working with Certificate Authorities like Let’s Encrypt to automatically sign and renew certificates. Helm is a popular package manager choice for Kubernetes. Installation of software, managing versions, upgrading versions and finding charts from registry are key benefits of Helm.

Installing TLS Certificates for EnRoute Ingress using Jack’s Cert Manager

EnRoute helm chart installs the EnRoute Ingress Controller to define L7 policy for a service running inside Kubernetes. The helm chart provides a fine-grained control to define L7 policies with its ability to enable/disable plugins for a service using configuration options that can be specified when helm is invoked.

EnRoute also supports plugins/filters to extend functionality and enforce policies. The features page lists the available plugins for the Gateway. More details about each of the plugins can also be found on plugin pages.

The helm chart integrates with Jack’s Cert manager to install a TLS certificate for a service. Once Jack’s Cert manager is installed, for example the following command sets up Jack’s Cert manager and issues a certificate -

helm upgrade httpbin-service-policy local-saaras/service-policy \
        --set service.namespace=demo-service                    \
        --set service.name=httpbin                              \
        --set service.prefix=/get                               \
        --set service.port=80                                   \
        --set service.enableTLS=true                            \
        --set autoTLS.certificateCN=httpbin.enroutedemo.com     \
        --set autoTLS.enableProd=true                                 

In the next few sections we describe what artifacts for certificate issue are created and how to debug it.

Artifacts created to support TLS Certificate Generation

Jack’s Cert Manager needs the creation of a ClusterIssuer that represent a certificate authority for certificates. EnRoute helm chart sets up a ClusterIssuer to work with Let’s encrypt staging, and another one to work with Let’s Encrypt production server -

ClusterIssuer

 kubectl describe --all-namespaces clusterissuers.cert-manager.io
Name:         letsencrypt-prod
Namespace:
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  cert-manager.io/v1
Kind:         ClusterIssuer
Metadata:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
  Self Link:         /apis/cert-manager.io/v1/clusterissuers/letsencrypt-prod
  UID:               370db8e7-fb61-4e41-a740-6e0f3bcabc1c
Spec:
  Acme:
    Email:            contact@example.com
    Preferred Chain:
    Private Key Secret Ref:
      Name:  letsencrypt-prod
    Server:  https://acme-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  enroute
      Selector:
Status:
  Acme:
    Last Registered Email:  contact@example.com
    Uri:                    https://acme-v02.api.letsencrypt.org/acme/acct/118404211
  Conditions:
    Last Transition Time:  2021-04-16T23:33:02Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>


Name:         letsencrypt-staging
Namespace:
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  cert-manager.io/v1
Kind:         ClusterIssuer
Metadata:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
  Self Link:         /apis/cert-manager.io/v1/clusterissuers/letsencrypt-staging
  UID:               34db7e69-f9a4-4009-bc68-7f1651f23d93
Spec:
  Acme:
    Email:            contact@example.com
    Preferred Chain:
    Private Key Secret Ref:
      Name:  letsencrypt-staging
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  enroute
      Selector:
Status:
  Acme:
    Last Registered Email:  contact@example.com
    Uri:                    https://acme-staging-v02.api.letsencrypt.org/acme/acct/18972900
  Conditions:
    Last Transition Time:  2021-04-16T23:33:02Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

Certificates

You get a certificate to install on the service by creating a Certificate object. This initiates the certificate signing process. Depending on the Issuer used, the staging or production server is used to get a certificate.

kubectl describe --all-namespaces certificates.cert-manager.io
Name:         httpbin.enroutedemo.com
Namespace:    demo-service
Labels:       app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2021-04-17T00:00:26Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    API Version:  cert-manager.io/v1
    Time:            2021-04-17T00:02:36Z
  Resource Version:  47879688
  Self Link:         /apis/cert-manager.io/v1/namespaces/demo-service/certificates/httpbin.enroutedemo.com
  UID:               e41092bf-6773-49d3-9f5d-4338798c97ab
Spec:
  Common Name:  httpbin.enroutedemo.com
  Dns Names:
    httpbin.enroutedemo.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-prod
  Secret Name:  httpbin.enroutedemo.com
Status:
  Conditions:
    Last Transition Time:  2021-04-17T00:02:36Z
    Message:               Certificate is up to date and has not expired
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2021-07-15T23:02:34Z
  Not Before:              2021-04-16T23:02:34Z
  Renewal Time:            2021-06-15T23:02:34Z
  Revision:                1
Events:                    <none>

Tracing Certificate creation logs

Jack’s Cert Manager installation creates services in the cert manager space.

kubectl get all -n cert-manager
NAME                                           READY   STATUS    RESTARTS   AGE
pod/cert-manager-56f5c44b5d-wshwn              1/1     Running   1          3d23h
pod/cert-manager-cainjector-5f494679f9-q8jhq   1/1     Running   0          3d23h
pod/cert-manager-webhook-5f59d4c7bd-xhqgj      1/1     Running   0          3d23h

NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/cert-manager           ClusterIP   10.96.82.166    <none>        9402/TCP   3d23h
service/cert-manager-webhook   ClusterIP   10.96.239.164   <none>        443/TCP    3d23h

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager              1/1     1            1           3d23h
deployment.apps/cert-manager-cainjector   1/1     1            1           3d23h
deployment.apps/cert-manager-webhook      1/1     1            1           3d23h

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-56f5c44b5d              1         1         1       3d23h
replicaset.apps/cert-manager-cainjector-5f494679f9   1         1         1       3d23h
replicaset.apps/cert-manager-webhook-5f59d4c7bd      1         1         1       3d23h

Tailing logs for the cert-manager service provides logs to debug certificate issue

kubectl logs -n cert-manager cert-manager-56f5c44b5d-wshwn --follow --since=5m
I0415 19:03:14.188036       1 conditions.go:173] Setting lastTransitionTime for Certificate "httpbin.enroutedemo.com" condition "Issuing" to 2021-04-15 19:03:14.188026423 +0000 UTC m=+3382.297097405
I0415 19:03:14.191331       1 conditions.go:173] Setting lastTransitionTime for Certificate "httpbin.enroutedemo.com" condition "Ready" to 2021-04-15 19:03:14.191322087 +0000 UTC m=+3382.300393029
...
I0415 19:03:16.904672       1 pod.go:70] cert-manager/controller/challenges/http01/ensurePod "msg"="creating HTTP01 challenge solver pod" "dnsName"="httpbin.enroutedemo.com" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
...
I0415 19:03:17.018426       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.018685       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vs8r7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.127299       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.127613       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vs8r7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
E0415 19:03:17.139881       1 controller.go:156] ingress 'enroutedemo/cm-acme-http-solver-kd82k' in work queue no longer exists
E0415 19:03:17.150861       1 controller.go:156] ingress 'enroutedemo/cm-acme-http-solver-drhzw' in work queue no longer exists
I0415 19:03:17.269384       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.269793       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vs8r7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.536732       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.537821       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vs8r7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:17.538186       1 ingress.go:92] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-vglq7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1beta1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
...
I0415 19:03:27.071540       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:27.071676       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vs8r7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:27.071761       1 ingress.go:92] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-vglq7" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1beta1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
...
I0415 19:03:38.290830       1 pod.go:118] cert-manager/controller/challenges/cleanupPods "msg"="deleting pod resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
I0415 19:03:38.309099       1 pod.go:126] cert-manager/controller/challenges/cleanupPods "msg"="successfully deleted pod resource" "dnsName"="httpbin.enroutedemo.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-29j68" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579-3409815712" "resource_namespace"="enroutedemo" "resource_version"="v1" "type"="HTTP-01"
E0415 19:03:38.415267       1 controller.go:156] ingress 'enroutedemo/cm-acme-http-solver-vglq7' in work queue no longer exists
I0415 19:03:38.832551       1 acme.go:198] cert-manager/controller/certificaterequests-issuer-acme/sign "msg"="certificate issued" "related_resource_kind"="Order" "related_resource_name"="httpbin.enroutedemo.com-f4xqt-2887721579" "related_resource_namespace"="enroutedemo" "related_resource_version"="v1" "resource_kind"="CertificateRequest" "resource_name"="httpbin.enroutedemo.com-f4xqt" "resource_namespace"="enroutedemo" "resource_version"="v1"
I0415 19:03:38.833597       1 conditions.go:222] Found status change for CertificateRequest "httpbin.enroutedemo.com-f4xqt" condition "Ready": "False" -> "True"; setting lastTransitionTime to 2021-04-15 19:03:38.833587237 +0000 UTC m=+3406.942658189
...