EnRoute Kubernetes Ingress Gateway

Published May 11, 2021
Last Modified February 05, 2022

EnRoute Universal Gateway

EnRoute Universal Gateway is a an API gateway built to support traditional and cloud-native use cases. It is designed to run either as a Kubernetes Ingress Gateway, Standalone Gateway, Horizontally scaling L7 API gateway or a Mesh of Gateways. Depending on the need of the user, the environment, the application, either one or many of these solutions can be deployed.

EnRoute also supports plugins/filters to extend functionality and enforce policies. The features page lists the available plugins for the Gateway. More details about each of the plugins can also be found on plugin pages.

A consistent policy framework across all these network components makes the EnRoute Universal Gateway a versatile and powerful solution.

What this article covers

This article covers how to get started with the EnRoute Kubernetes Ingress Gateway. The minimum requirement is a working Kubernetes cluster.

To get a more detailed understanding of EnRoute Universal Gateway and its architecture, refer to the article here To run EnRoute outside of kubernetes as a standalone gateway, refer to the article on standalone gateway

EnRoute also supports several other topologies including a Standalone Gateway. Only the Kubernetes Ingress topology is covered in this article.

Configure EnRoute using helm

A simple non-SSL example of EnRoute gateway with Lua and Rate-Limit filters can be programmed using the EnRoute helm chart.

Add the helm chart -

helm repo add saaras https://getenroute.io

Check repositories -

helm search repo

NAME                    CHART VERSION   APP VERSION     DESCRIPTION
saaras/enroute          0.5.0           v0.8.0          EnRoute API Gateway
saaras/service-policy   0.3.1           0.8.0           Service L7 Policy using EnRoute API Gateway

Install the helm chart -

helm install enroute-demo saaras/enroute   \
    --set serviceAccount.create=true --create-namespace --namespace enroutedemo

NAME: enroute-demo
LAST DEPLOYED: Tue Dec 28 01:37:05 2021
NAMESPACE: enroutedemo
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
###################
# 𝙴𝚗𝚁𝚘𝚞𝚝𝚎 𝚂𝚎𝚛𝚟𝚒𝚌𝚎 #
###################

It got installed in namespace: enroutedemo

This installs the EnRoute Ingress API Gateway.

kubectl get svc -n enroutedemo

NAME           TYPE           CLUSTER-IP      EXTERNAL-IP    PORT(S)                      AGE
enroute-demo   LoadBalancer   10.43.113.134   212.2.246.47   80:32365/TCP,443:32120/TCP   50s

Questions about the helm chart? Slack Send Us a Note

However we still need to program the gateway to expose a service. Let us start by creating a demo namespace and a demo service.

kubectl create namespace demo-service

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: httpbin
  namespace: demo-service
  labels:
    app: httpbin
spec:
  containers:
  - name: httpbin
    image: kennethreitz/httpbin
    ports:
      - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: demo-service
spec:
  selector:
    app: httpbin
  ports:
  - port: 80
EOF

This creates a demo service in demo-service namepspace.

Now lets expose this service using a simple helm command

helm install httpbin-service-policy saaras/service-policy    \
     --set service.name=httpbin                              \
     --set service.prefix=/get                               \
     --set service.port=80                                   \
     --namespace demo-service

NAME: httpbin-service-policy
LAST DEPLOYED: Tue Dec 28 01:54:03 2021
NAMESPACE: demo-service
STATUS: deployed
REVISION: 1
NOTES:
##########################
# 𝙴𝚗𝚁𝚘𝚞𝚝𝚎 𝚂𝚎𝚛𝚟𝚒𝚌𝚎 𝙿𝚘𝚕𝚒𝚌𝚢 #
##########################

It got installed in namespace: demo-service

The above command makes the service httpbin running on port 80 externally visible on path /get

Let’s check all the helm charts installed

helm ls --all-namespaces

NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
enroute-demo            enroutedemo     1               2021-12-28 01:37:05.978469338 +0000 UTC deployed        enroute-0.5.0           v0.8.0
httpbin-service-policy  demo-service    1               2021-12-28 01:54:03.485778209 +0000 UTC deployed        service-policy-0.3.1    0.8.0

Use the External IP to send traffic

The EnRoute service can be reached using the External-IP and a request on path /get sends it to the httpbin service

curl 212.2.246.47/get

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "212.2.246.47",
    "User-Agent": "curl/7.68.0",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Internal": "true"
  },
  "origin": "10.42.0.41",
  "url": "http://212.2.246.47/get"
}

Trigger rate-limits by sending a burst of traffic

while true; do curl -I 212.2.246.47/get; done;

...

HTTP/1.1 200 OK
server: envoy
date: Tue, 28 Dec 2021 01:57:01 GMT
content-type: application/json
content-length: 266
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 3
vary: Accept-Encoding

HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
vary: Accept-Encoding
date: Tue, 28 Dec 2021 01:57:01 GMT
server: envoy
transfer-encoding: chunked

HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
vary: Accept-Encoding
date: Tue, 28 Dec 2021 01:57:01 GMT
server: envoy
transfer-encoding: chunked

...

Questions about rate-limit? Slack Send Us a Note

Understanding EnRoute configuration setup in previous step

`GatewayHost`

GatewayHost is a way to expose your services running inside Kubernetes.

kubectl describe  gatewayhosts.enroute.saaras.io  --all-namespaces

Name:         httpbin-80-gatewayhost
Namespace:    demo-service
Labels:       app=httpbin
              app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: demo-service
API Version:  enroute.saaras.io/v1
Kind:         GatewayHost
Metadata:
...
Spec:
  Routes:
    Conditions:
      Prefix:  /get
    Filters:
      Name:  httpbin-80-rl2
      Type:  route_filter_ratelimit
    Services:
      Health Check:
        Healthy Threshold Count:    3
        Host:                       hc
        Interval Seconds:           5
        Path:                       /
        Timeout Seconds:            3
        Unhealthy Threshold Count:  3
      Name:                         httpbin
      Port:                         80
  Virtualhost:
    Filters:
      Name:  httpbin-80-luatestfilter
      Type:  http_filter_lua
    Fqdn:    *
Events:      <none>

Filters attached to GatewayHost can be used to control traffic flowing to the exposed service

Questions about GatewayHost? Slack Send Us a Note

`Filter`

Filters provide fine-grained control to control access to service. There are several features supported by EnRoute.

Lua Filter

This config invokes a function for an incoming request on request and response path

kubectl describe -n demo-service httpfilters.enroute.saaras.io httpbin-80-luatestfilter

Name:         httpbin-80-luatestfilter
Namespace:    demo-service
Labels:       app=httpbin-app
              app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  enroute.saaras.io/v1
Kind:         HttpFilter
Metadata:
...
Spec:
  Http Filter Config:
    Config:  function envoy_on_request(request_handle)
   request_handle:logInfo("Hello World request");
end

function envoy_on_response(response_handle)
   response_handle:logInfo("Hello World response");
end

  Name:  httpbin-80-luatestfilter
  Type:  http_filter_lua
Events:  <none>

Questions about lua filter? Slack Send Us a Note

Rate Limit config for Filter

The following config limits every http request from a unique IP. The x-forwarded-for provides the remote-ip and x-forwarded-proto is set to http for requests. A combination of IP address (from x-forwarded-for) and protocol (from x-forwarded-proto) is rate-limited to 5 requests per second.

kubectl describe -n demo-service globalconfigs.enroute.saaras.io httpbin-rl-global-config

Name:         httpbin-rl-global-config
Namespace:    demo-service
Labels:       app=httpbin-app
              app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  enroute.saaras.io/v1beta1
Kind:         GlobalConfig
Metadata:
...
Spec:
  Config:  {
  "domain": "enroute",
  "descriptors" :
  [
    {
      "key": "x-forwarded-for",
      "descriptors" :
      [
        {
          "key" : "x-forwarded-proto",
          "value" : "http",
          "descriptors" : [
           {
             "key" : "generic_key",
             "value" : "default_route",
             "rate_limit" : { "unit" : "second", "requests_per_unit" : 5 }
            }
          ]
        },
        {
          "key" : "x-forwarded-proto",
          "value" : "https",
          "descriptors" : [
           {
             "key" : "generic_key",
             "value" : "default_route",
             "rate_limit" : { "unit" : "second", "requests_per_unit" : 2 }
           }
          ]
        }
      ]
    }
  ]
}

  Name:  httpbin-rl-global-config
  Type:  globalconfig_ratelimit
Events:  <none>

Questions about filters? Slack Send Us a Note

Adding SSL

The above config does not add SSL config. For SSL to work, we need a certificate installed with the appropriate domain name.

Let’s install a SSL certificate for both the httpbin and echo service. We will first install Jetstack cert manager. Then invoke EnRoute helm chart to issue and install certificates.

Install Jetstack Cert Manager

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm search repo

We can now see the helm repo for Jetstack cert manager

NAME                                    CHART VERSION   APP VERSION     DESCRIPTION
jetstack/cert-manager                   v1.6.1          v1.6.1          A Helm chart for cert-manager
jetstack/cert-manager-approver-policy   v0.2.0          v0.2.0          A Helm chart for cert-manager-approver-policy
jetstack/cert-manager-csi-driver        v0.2.1          v0.2.0          A Helm chart for cert-manager-csi-driver
jetstack/cert-manager-csi-driver-spiffe v0.1.0          v0.1.0          A Helm chart for cert-manager-csi-driver-spiffe
jetstack/cert-manager-istio-csr         v0.3.1          v0.3.0          A Helm chart for istio-csr
jetstack/cert-manager-trust             v0.1.1          v0.1.0          A Helm chart for cert-manager-trust
saaras/enroute                          0.5.0           v0.8.0          EnRoute API Gateway
saaras/service-policy                   0.3.1           0.8.0           Service L7 Policy using EnRoute API Gateway

Let’s go ahead an install the cert manager

helm install cert-manager jetstack/cert-manager \
    --namespace cert-manager                    \
    --version v1.2.0                            \
    --create-namespace                          \
    --set installCRDs=true

NAME: cert-manager
LAST DEPLOYED: Tue Dec 28 02:18:43 2021
NAMESPACE: cert-manager
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
cert-manager has been deployed successfully!
...

The cert manager chart got installed, lets list all the charts we have

helm ls --all-namespaces

Let’s verify the cert manager chart installation

NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
cert-manager            cert-manager    1               2021-12-28 02:18:43.239487767 +0000 UTC deployed        cert-manager-v1.2.0     v1.2.0
enroute-demo            enroutedemo     1               2021-12-28 01:37:05.978469338 +0000 UTC deployed        enroute-0.5.0           v0.8.0
httpbin-service-policy  demo-service    1               2021-12-28 02:06:25.6445765 +0000 UTC   deployed        service-policy-0.3.1    0.8.0

We can also check the artifacts installed for with the cert manager installation

kubectl get all -n cert-manager

NAME                                           READY   STATUS    RESTARTS   AGE
pod/cert-manager-cainjector-74459fcc56-5r8xr   1/1     Running   0          74s
pod/cert-manager-85f9bbcd97-tk6rv              1/1     Running   0          74s
pod/cert-manager-webhook-57d97ccc67-5m9zt      1/1     Running   0          74s

NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/cert-manager-webhook   ClusterIP   10.43.211.67    <none>        443/TCP    75s
service/cert-manager           ClusterIP   10.43.195.104   <none>        9402/TCP   75s

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager-cainjector   1/1     1            1           75s
deployment.apps/cert-manager              1/1     1            1           75s
deployment.apps/cert-manager-webhook      1/1     1            1           75s

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-cainjector-74459fcc56   1         1         1       75s
replicaset.apps/cert-manager-85f9bbcd97              1         1         1       75s
replicaset.apps/cert-manager-webhook-57d97ccc67      1         1         1       75s

We will use the HTTP01 challenge to issue the cert. For that, we will first need to create a DNS record to map httpbin-service.enroutedemo.com to the external public domain name a3523e69d10ea4da19d50af16f2dc272-1512456672.us-west-1.elb.amazonaws.com

Once you create the DNS record, verify that the DNS name maps to the correct IP

ping httpbin-service.enroutedemo.com

ping httpbin-service.enroutedemo.com
PING httpbin-service.enroutedemo.com (212.2.246.47) 56(84) bytes of data.
...

Now we are ready to install a SSL certificate for this service

Adding SSL to `httpbin` service

helm upgrade -n enroutedemo httpbin-service-policy saaras/service-policy \
      --namespace demo-service                                           \
      --set service.name=httpbin                                         \
      --set service.prefix=/get                                          \
      --set service.port=80                                              \
      --set service.enableTLS=true                                       \
      --set service.createGlobalConfig=false                             \
      --set autoTLS.certificateCN=httpbin-service.enroutedemo.com        \
      --set autoTLS.createIssuers=true                                   \
      --set autoTLS.email=myemail@mydomain.com                           \
      --set autoTLS.enableProd=true

Note: In the step above, the actual email has been redacted.

Release "httpbin-service-policy" has been upgraded. Happy Helming!
NAME: httpbin-service-policy
LAST DEPLOYED: Tue Dec 28 02:24:33 2021
NAMESPACE: demo-service
STATUS: deployed
REVISION: 2
NOTES:
##########################
# 𝙴𝚗𝚁𝚘𝚞𝚝𝚎 𝚂𝚎𝚛𝚟𝚒𝚌𝚎 𝙿𝚘𝚕𝚒𝚌𝚢 #
##########################

It got installed in namespace: demo-service

The above command enables TLS for the service by using the fqdn of httpbin-service.enroutedemo.com

It also issues the certificate using the CN httpbin-service.enroutedemo.com

The flag autoTLS.createIssuers=true creates Issuers, which are needed to work with Jetstack Cert manager. Note that this is only required the very first time. This flag is not required when issuing certificates again in the same cluster.

The autoTLS.email=myemail@mydomain.com needs to be set to a valid email address that can be used by the Cert manager to request a certificate. Note that the current value of myemail@mydomain.com won’t work

Let’s check the status of the certificate.

kubectl get certificates.cert-manager.io --all-namespaces

NAMESPACE      NAME                              READY   SECRET                            AGE
demo-service   httpbin-service.enroutedemo.com   True    httpbin-service.enroutedemo.com   2m6s

The True status above tells us that certificate issue was successful. More details are also captured in troubleshooting and more information about EnRoute with Jetstack Cert manager

Let’s send some traffic to verify the certificate got installed

curl -v https://httpbin-service.enroutedemo.com/get

*   Trying 212.2.246.47:443...
* TCP_NODELAY set
* Connected to httpbin-service.enroutedemo.com (212.2.246.47) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=httpbin-service.enroutedemo.com
*  start date: Dec 28 01:25:01 2021 GMT
*  expire date: Mar 28 01:25:00 2022 GMT
*  subjectAltName: host "httpbin-service.enroutedemo.com" matched cert's "httpbin-service.enroutedemo.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55d62ab17860)
> GET /get HTTP/2
> Host: httpbin-service.enroutedemo.com
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200
< server: envoy
< date: Tue, 28 Dec 2021 02:27:45 GMT
< content-type: application/json
< content-length: 305
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 5
< vary: Accept-Encoding
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "httpbin-service.enroutedemo.com",
    "User-Agent": "curl/7.68.0",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Internal": "true"
  },
  "origin": "10.42.0.41",
  "url": "https://httpbin-service.enroutedemo.com/get"
}
* Connection #0 to host httpbin-service.enroutedemo.com left intact

Let’s check the helm charts installed.

helm ls --all-namespaces

NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
cert-manager            cert-manager    1               2021-12-28 02:18:43.239487767 +0000 UTC deployed        cert-manager-v1.2.0     v1.2.0
enroute-demo            enroutedemo     1               2021-12-28 01:37:05.978469338 +0000 UTC deployed        enroute-0.5.0           v0.8.0
httpbin-service-policy  demo-service    2               2021-12-28 02:24:33.762045299 +0000 UTC deployed        service-policy-0.3.1    0.8.0

Summary of artifacts created

We graphically show the summary of steps, the helm commands and the corresponding artifacts created

A helm chart instance per-service

EnRoute helm chart helps manage policy per-service by installing a version of the chart for each service.

The previous example showed how a simple service can receive external traffic. Note that the previous service GatewayHost installs a fqdn of * which matches traffic destined to all hosts. However, real-world deployments often route on the basis of fqdn first and then the route.

In this example, we demonstrate how traffic destined to a specific service is first routed on the basis of fqdn and then the route for the service.

We start by installing the another example workload

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo-server
  template:
    metadata:
      labels:
        app: echo-server
    spec:
      containers:
        - name: echo-server
          image: jmalloc/echo-server
          ports:
            - name: http-port
              containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: echo-service
spec:
  ports:
    - name: http-port
      port: 80
      targetPort: http-port
      protocol: TCP
  selector:
    app: echo-server
EOF

This creates an echo service in default namespace.

Now lets expose this service using a simple helm command

helm install echo-service-policy saaras/service-policy \
    --set service.name=echo-service                    \
    --set service.prefix=/get                          \
    --set service.enableTLS=false                      \
    --set service.port=80                              \
    --set service.fqdn=echo-service.enroutedemo.com

Note that this time, we use a couple of other flags. service.enableTLS=false disables https on this service. The service.fqdn=echo-service.enroutedemo.com flag sets the fqdn. Only traffic destined to echo-service.enroutedemo.com will be directed to this service

Let’s send some traffic to test this (Note that we did create a DNS CNAME record for echo-service.enroutedemo.com to 212.2.246.47 before sending traffic).

curl echo-service.enroutedemo.com/get

Request served by echo-deployment-869b7bf9c7-xlx44

HTTP/1.1 GET /get

Host: echo-service.enroutedemo.com
User-Agent: curl/7.68.0
X-Request-Id: 9d23891f-7b1d-470a-9b1d-232ca9adf51b
X-Envoy-Expected-Rq-Timeout-Ms: 15000
X-Request-Start: t=1640659237.591
Accept: */*
X-Forwarded-For: 10.42.0.41
X-Forwarded-Proto: http
X-Envoy-Internal: true

Adding SSL to `echo` service

Installing a SSL certificate for the echo service is similar to the previous step, except that this time we don’t need to create issuers. We leave out the autoTLS.createIssuers flag this time. The steps below demonstrate that

First create a DNS CNAME record from echo-service.enroutedemo.com to 212.2.246.47. Then use the helm command below to issue a certificate and install it for the service

helm upgrade echo-service-policy saaras/service-policy        \
    --set service.name=echo-service                           \
    --set service.prefix=/get                                 \
    --set service.port=80                                     \
    --set service.enableTLS=true                              \
    --set autoTLS.certificateCN=echo-service.enroutedemo.com  \
    --set autoTLS.email=myemail@mydomain.com                  \
    --set autoTLS.enableProd=true

Note that we don’t use the createIssuers=true flag above since it is only needed the very first time.

Release "echo-service-policy" has been upgraded. Happy Helming!
NAME: echo-service-policy
LAST DEPLOYED: Tue Dec 28 02:43:00 2021
NAMESPACE: default
STATUS: deployed
REVISION: 2
NOTES:
##########################
# 𝙴𝚗𝚁𝚘𝚞𝚝𝚎 𝚂𝚎𝚛𝚟𝚒𝚌𝚎 𝙿𝚘𝚕𝚒𝚌𝚢 #
##########################

It got installed in namespace: default

Let’s check the status of the new certificate we requested

kubectl get certificates.cert-manager.io --all-namespaces

NAMESPACE      NAME                        READY   SECRET                      AGE
demo-service   httpbin-service.enroutedemo.com   True    httpbin-service.enroutedemo.com   19m
default        echo-service.enroutedemo.com      True    echo-service.enroutedemo.com      49s

Now that the certificate is ready, let’s send some traffic to test it

curl -v https://echo-service.enroutedemo.com/get

*   Trying 212.2.246.47:443...
* TCP_NODELAY set
* Connected to echo-service.enroutedemo.com (212.2.246.47) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=echo-service.enroutedemo.com
*  start date: Dec 28 01:43:28 2021 GMT
*  expire date: Mar 28 01:43:27 2022 GMT
*  subjectAltName: host "echo-service.enroutedemo.com" matched cert's "echo-service.enroutedemo.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x564b0a520860)
> GET /get HTTP/2
> Host: echo-service.enroutedemo.com
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 200
< content-type: text/plain
< date: Tue, 28 Dec 2021 02:44:31 GMT
< content-length: 341
< x-envoy-upstream-service-time: 1
< vary: Accept-Encoding
< server: envoy
<
Request served by echo-deployment-869b7bf9c7-xlx44

HTTP/1.1 GET /get

Host: echo-service.enroutedemo.com
X-Request-Id: 0b204581-4c04-40b1-9ea2-723c89662647
X-Envoy-Expected-Rq-Timeout-Ms: 15000
User-Agent: curl/7.68.0
Accept: */*
X-Forwarded-For: 10.42.0.41
X-Forwarded-Proto: https
X-Envoy-Internal: true
X-Request-Start: t=1640659471.225
* Connection #0 to host echo-service.enroutedemo.com left intact

Creation of SSL certificates is automated using ACME protocol and a Certificate Authority like Let’s Encrypt. EnRoute integrates with Jacks Cert Manager to achieve this. The EnRoute helm chart also provides switches to achieve this.

Questions about SSL/TLS? Slack Send Us a Note

Helm switches to enable/disable features

EnRoute helm chart provides configuration options for multiple clouds and switches to enable/disable plugins or filters. These switches provide a fine-grained control over L7 policy.

The EnRoute Helm chart supports the following filter switches -

Switch	Valid Values	Description
envoySettings.logLevel	One of - ["trace", "debug", "info", "error"]	Log level for underlying Envoy
EnRoute Ingress Controller Service Settings
enrouteService.enable	One of - ["true", "false"]	Disabled by default. Enable when installing EnRoute the first time. Enables creation of EnRoute deployment and service
enrouteService.replicaCount	Number of replicas eg: 3	By default, one instance of EnRoute pod is created Change this value to increase/decrease count of replicas
Cloud Settings
awsSettings.enable	One of - ["true", "false"]	Disabled by default. Enable when running on AWS. Enables propagation of client-IP to backend service
ociSettings.enable	One of - ["true", "false"]	Disabled by default. Enable when running on OCI. Enables propagation of client-IP to backend service
digitaloceanSettings.enable	One of - ["true", "false"]	Disabled by default. Enable when running on DigitalOcean. Enables propagation of client-IP to backend service
Filter Settings
filters.lua.enable	One of - ["true", "false"]	Enabled by default. Enable when you want to install a Lua filter for service.
filters.ratelimit.enable	One of - ["true", "false"]	Enabled by default. Enable when you want to install a RateLimit filter for service.
filters.cors.enable	One of - ["true", "false"]	Disabled by default. Enable when you want to install a CORS filter for service.
filters.jwt.enable	One of - ["true", "false"]	Disabled by default. Enable when you want to install a JWT filter for service.
Service Settings
service.name	A string representing service name in Kubernetes	Default is "hello-enroute" service. Set it to your own service name
service.port	Port on which to access this service	Default is 9090, the port for "hello-enroute" service. Set it to your own service port
service.enableTLS	When securing the service using TLS, use this flag after cert-manager has been setup.	Default is false. Enable this flag to automatically get a certificate for a service from Let's encrypt
service.namespace	The namespace in which the service is running	This setting is used to determine the namespace in which to create EnRoute Filters and ```GatewayHost```
service.prefix	The prefix on which the service is reachable	This installs the prefix route for this service
Auto-TLS Settings
autoTLS.issueCert	One of - ["true", "false"]	Enable this to issue a certificate signed using the ACME protocol (from Let's Encrypt)
autoTLS.certificateCN	Common Name for Certificate	Common Name to use for Certificate. This also gets installed as ```fqdn``` for the service
autoTLS.enableProd	One of - ["true", "false"]	Default is false. When set to false, it uses the ACME staging server `https://acme-staging-v02.api.letsencrypt.org/directory` . When set to true, it uses the ACME production server `https://acme-v02.api.letsencrypt.org/directory`

As filters/plugins are added to EnRoute, simple switches will allow enabling/disabling these features.

Questions about the helm chart? Slack Send Us a Note

Enforcing L7 Policies

EnRoute uses filter/plugin to add functionality to Ingress for a service. These Filters/Plugins loosely map to Envoy filters/plugins and can be enabled/disabled using the helm chart. The complete list of plugins supported in the helm chart and examples are covered in the Enforce L7 Policy blog

Adding JWT to `echo` service

Adding JWT to validate requests sent to the echo service is as simple adding a filters.jwt.enable=true switch below -

helm upgrade echo-service-policy saaras/service-policy \
 --set service.name=echo-service                       \
 --set service.prefix=/get                             \
 --set service.port=80                                 \
 --set service.enableTLS=true                          \
 --set autoTLS.certificateCN=echo-service.enroutedemo.com \
 --set autoTLS.email=myemail@mydomain.com              \
 --set autoTLS.enableProd=true                         \
 --set filters.jwt.enable=true

Now let’s try sending the request again

curl -v https://echo-service.enroutedemo.com/get

*   Trying 212.2.246.47:443...
* TCP_NODELAY set
* Connected to echo-service.enroutedemo.com (212.2.246.47) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=echo-service.enroutedemo.com
*  start date: Dec 28 01:43:28 2021 GMT
*  expire date: Mar 28 01:43:27 2022 GMT
*  subjectAltName: host "echo-service.enroutedemo.com" matched cert's "echo-service.enroutedemo.com"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55be3e627860)
> GET /get HTTP/2
> Host: echo-service.enroutedemo.com
> user-agent: curl/7.68.0
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 2147483647)!
< HTTP/2 401
< www-authenticate: Bearer realm="https://echo-service.enroutedemo.com/get"
< content-length: 14
< content-type: text/plain
< date: Tue, 28 Dec 2021 02:47:34 GMT
< server: envoy
<
* Connection #0 to host echo-service.enroutedemo.com left intact

EnRoute Config Model across Kubernetes Ingress Gateway and Standalone Gateway

EnRoute can be used to protect services outside kubernetes using the standalone gateway or services running inside kubernetes.

EnRoute follows a configuration model similar to Envoy and is extensible using Filters. It uses filters to extend functionality at the global Service level and per-route level. The config objects used for this are - GatewayHost, Route, HttpFilter, RouteFilter and Service as shown here for Kubernetes gateway.

Regardless of where the workload runs, a consistent service policy can be defined once and applied to secure any service running inside or without Kubernetes using Envoy.

EnRoute provides key functionality using modular filters which make it easy to secure any service.

Need Questions Answered? Slack Send Us a Note

Next Steps

Working with EnRoute Ingress API Gateway is organized in following steps

Step-1 Getting Started : Enforcing L7 Policy for a micro-service API using helm chart
Step-2 Getting Started : Filters in EnRoute Ingress Controller
Step-3 Understanding the How and Why of Kubernetes Ingress and Networking
Step-4 Using Self-Signed Certificates with EnRoute OneStep Kubernetes Ingress

kubernetes-ingress

EnRoute Kubernetes Ingress Gateway