Enroute Kubernetes Ingress Gateway

February 11, 2020

Enroute Universal Gateway

Enroute Universal Gateway is a flexible API gateway built to support traditional and cloud-native use cases. It is designed to run either as a [Kubernetes Ingress Gateway, Standalone Gateway, Horizontally scaling L7 API gateway or a Mesh of Gateways] (/blog/enroute-topologies/) . Depending on the need of the user, the environment, the application, either one or many of these solutions can be deployed.

A consistent policy framework across all these network components makes the Enroute Universal Gateway a versatile and powerful solution.

What this article covers

This article covers how to get started with the Enroute Kubernetes Ingress Gateway. There are several options to running a kubernetes ingress gateway. This document only covers a cloud provider where service of type LoadBalancer is supported. Alternative mechanisms include using a NodePort which will be convered in another article.

To get a more detailed understanding of Enroute Universal Gateway and its architecture, refer to the article here To run Enroute outside of kubernetes as a standalone gateway, refer to the article on standalone gateway

Enroute also supports several other topologies including a Standalone Gateway and Gateway Mesh topology. One Enroute control plane can program multiple stateless Enroute data planes. Every Enroute data plane instance runs an instance of Envoy along with it. Only the Kubernetes Ingress topology is covered in this article.

Enroute Config Model

Enroute follows a configuration model similar to Envoy. It uses filters to extend functionality at the global Service level and per-route level. The config objects used for this are - GatewayHost, Route, HttpFilter, RouteFilter and Service as shown here for Kubernetes gateway.

Enroute Kubernetes Ingress Gateway in less than a minute

Use the following command to create all the configuration in one step. A step-by-step walkthrough is shown below -

bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/ic-all.yaml'

The above command does not create the self-signed certificate. You can send non-SSL traffic to test the above configuration. The SSL configuration is shown below.

While this step creates all CRDs, Roles and Enroute Objects, here are the important ones of interest -

`GatewayHost` with `http_filter_lua` and `route_filter_ratelimit`

---
apiVersion: enroute.saaras.io/v1beta1
kind: GatewayHost
metadata:
  labels:
    app: httpbin
  name: httpbin
  namespace: enroute-gw-k8s
spec:
  virtualhost:
    fqdn: demo.saaras.io
    filters:
      - name: luatestfilter
        type: http_filter_lua
  routes:
    - match: /
      services:
        - name: httpbin
          port: 80
      filters:
        - name: rl2
          type: route_filter_ratelimit
---

`GlobalConfig` to configure rate-limit engine

---
apiVersion: enroute.saaras.io/v1beta1
kind: GlobalConfig
metadata:
  labels:
    app: httpbin
  name: rl-global-config
  namespace: enroute-gw-k8s
spec:
  name: rl-global-config
  type: globalconfig_ratelimit
  config: |
        {
          "domain": "enroute",
          "descriptors" :
          [
            {
              "key" : "generic_key",
              "value" : "default",
              "rate_limit" :
              {
                "unit" : "second",
                "requests_per_unit" : 10
              }
            }
          ]
        }
---

`RouteFilter` to configure per-route rate-limit directive

---
apiVersion: enroute.saaras.io/v1beta1
kind: RouteFilter
metadata:
  labels:
    app: httpbin
  name: rl2
  namespace: enroute-gw-k8s
spec:
  name: rl2
  type: route_filter_ratelimit
  routeFilterConfig:
    config: |
          {
            "descriptors" :
            [
              {
                "generic_key":
                {
                  "descriptor_value":"default"
                }
              }
            ]
          }
---

`HttpFilter` to configure lua code attached to `GatewayHost`

---
apiVersion: enroute.saaras.io/v1beta1
kind: HttpFilter
metadata:
  labels:
    app: httpbin
  name: luatestfilter
  namespace: enroute-gw-k8s
spec:
  name: luatestfilter
  type: http_filter_lua
  httpFilterConfig:
    config: |
        function envoy_on_request(request_handle)
           request_handle:logInfo("Hello World request httpbin");
        end

        function envoy_on_response(response_handle)
           response_handle:logInfo("Hello World response httpbin");
        end
---

Running Enroute as a Kubernetes Ingress Gateway

Enroute leverages existing open source to function as an API gateway for microservices running inside a kubernetes cluster. Enroute uses filter configuration model to attach global filters and per-route filters to provide the flexible configuration. This configuration model is consistent with Enroute standalone gateway.

We’ll perform the following steps to run Enroute Kubernetes Ingress Gateway.

We’ll create a namespace and roles for running Enroute. Next we’ll create a deployment to run Enroute. We’ll create a LoadBalancer service which provides us with an external DNS name or IP as an entry-point into cluster.

We’ll then create a test service and configure it using CRDs. The CRDs let us attach additional global and route-level filter configuration with Enroute.

Once configured, we’ll send some traffic to it.

The next steps assumes you have a kubernetes cluster setup on a cloud that supports the LoadBalancer type of service. In a subsequent article we’ll describe how to run Enroute for cloud that do not support this service.

Configuration Steps

Create Namespace, Deployment, LoadBalancer, Custom Resource Definitions and run example service

All the enroute configuration will be hosted in a separate namespace. Some important config objects are also highlighted along with their links.

Create namespace and roles using -

bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/ns.yaml'

For this example, we run Enroute as a kubernetes deployment.

bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/deployment.yaml'

For this example, we’ll use a service of type LoadBalancer.

bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/lb.yaml'

When Enroute is used as an ingress gateway, the configuration can be provided using CRDs.

bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/crds.yaml'

We run example service httpbin to demonstrate Enroute’s capabilities

bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/httpbin.yaml'

Enroute filters

A global filter applies to all traffic going through Enroute. A per-route filter applies to traffic that matches that route. Enroute as of current version supports global lua filter and per-route advanced ratelimiting filter.

Filters are created using CRDs and got created in the step above.

Create global lua filter for all traffic

The HttpFilter.enroute.saaras.io/v1beta1 CRD is a way to specify a Global HTTP filter.

bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/httpbin.filter.lua.yaml'

This filter can then be attached at the global level.

bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/httpbin.filter.lua.attach.yaml'

The lua script invokes a function on the request and response path. This can be verified by sending a request and checking logs. Note below that 10.97.133.62 is the ClusterIP for the LoadBalancer service

Send traffic

Note that to match the domain name that we have programmed (demo.saaras.io), we need to use the --resolve functionality provided by curl

$ curl --resolve demo.saaras.io:80:10.97.133.62 http://demo.saaras.io/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Host": "demo.saaras.io",
    "User-Agent": "curl/7.58.0"
  },
  "origin": "192.168.42.192",
  "url": "http://demo.saaras.io/get"
}

Where 10.97.133.62 is the Service IP.

Using SSL

For SSL to work, we need a certificate installed with the appropriate domain name. As an example, we’ll use demo.saaras.io as the domain name. This domain name will be provided at three locations

When programming the Enroute Ingress Controller, the value of Fqdn will be set to this value.
A CNAME record that maps this domain to the public domain provided by LoadBalancer service by AWS. When passing traffic, we use this domain.

As an example for this demo, you can use script to create a certificate and key. The contents of the script are shown below. It creates the key, self-signed certificate and creates a kubernetes secret using them.

 
#!/bin/bash

if [ $# -gt 1 ]; then

    cn='localhost'
    cn=$1

    echo "Received cn as" $1
    openssl req -new -newkey rsa:$2 -days 36500 -nodes -x509 \
        -subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=$cn" \
        -keyout $1.key  -out $1.cert

    openssl x509 -in $cn.cert -text -noout

else
    echo "Script to generate self signed certificate with a provided CN name and keysize"
    echo "On successful execution, output is <cn_name.key> and <cn_name.cert>"
    echo "Usage:    ./self_signed_cert.sh <cn_name> <keysize>"
    echo "Example:  ./self_signed_cert.sh demo.saaras.io 4096"
    echo "Example:  ./self_signed_cert.sh demo.saaras.io 2048"
fi

kubectl -n enroute-gw-k8s create secret tls tls-secret-v0.4.0-httpbin-local --cert=$1.cert --key=$1.key

Send traffic

 
$ curl -k https://demo.saaras.io/get
GET /get HTTP/2
Host: demo.saaras.io
User-Agent: curl/7.54.0
Accept: */*

HTTP/2 200
server: envoy
date: Thu, 26 Mar 2020 00:07:15 GMT
content-type: application/json
content-length: 300
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 3

{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "0",
    "Host": "demo.saaras.io",
    "User-Agent": "curl/7.54.0",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Internal": "true"
  },
  "origin": "10.0.24.142",
  "url": "https://demo.saaras.io/get"
}
* Connection #0 to host demo.saaras.io left intact

Check logs

 
kubectl logs -n enroute-gw-k8s enroute-795679bf56-2jhg9 envoy
...
[2020-02-18 06:25:00.771][12][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:585] script log: Hello World request httpbin
[2020-02-18 06:25:00.771][12][debug][lua] [source/extensions/filters/common/lua/lua.cc:37] coroutine finished
...
2020-02-18 06:25:00.773][12][trace][lua] [bazel-out/k8-opt/bin/source/extensions/filters/common/lua/_virtual_includes/lua_lib/extensions/filters/common/lua/lua.h:122] creating N5Envoy10Extensions11HttpFilters3Lua19StreamHandleWrapperE at 0x41b07138
[2020-02-18 06:25:00.773][12][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:585] script log: Hello World response httpbin
[2020-02-18 06:25:00.773][12][debug][lua] [source/extensions/filters/common/lua/lua.cc:37] coroutine finished
...
':status', '200'
'server', 'envoy'
'date', 'Tue, 18 Feb 2020 06:25:00 GMT'
'content-type', 'application/json'
'content-length', '300'
'access-control-allow-origin', '*'
'access-control-allow-credentials', 'true'
'x-envoy-upstream-service-time', '1'

Create per-route rate-limiting filter for traffic matching that route.

The RouteFilter.enroute.saaras.io/v1beta1 and GlobalConfig.enroute.saaras.io/v1beta1 are CRDs to specify per-route rate-limit config.

 bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/httpbin.filter.rl.yaml'

This filter can then be attached on a per-route basis.

bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/httpbin.filter.lua.rl.attach.yaml'

The rate limit filter invokes the rate-limit service configured using the GlobalConfig.enroute.saaras.io/v1beta1. In the next article, we’ll cover more details on using the rate-limit filter and advanced configuration to run different use-cases. Note below that 10.99.185.36 is the ClusterIP for the LoadBalancer service

 
$ while true; do curl -vvv -k --resolve demo.saaras.io:80:10.99.185.36 http://demo.saaras.io/get; done

* Added demo.saaras.io:80:10.99.185.36 to DNS cache
* Hostname demo.saaras.io was found in DNS cache
*   Trying 10.99.185.36...
* TCP_NODELAY set
* Connected to demo.saaras.io (10.99.185.36) port 80 (#0)
> GET /get HTTP/1.1
> Host: demo.saaras.io
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 429 Too Many Requests
< x-envoy-ratelimited: true
< date: Tue, 18 Feb 2020 06:46:15 GMT
< server: envoy
< content-length: 0
<

Check logs to verify

...
[2020-02-18 06:43:45.654][12][debug][http] [source/common/http/conn_manager_impl.cc:600] [C8018][S13203667490673188384] request headers complete (end_stream=true):
':authority', 'demo.saaras.io'
':path', '/get'
':method', 'GET'
'user-agent', 'curl/7.58.0'
'accept', '*/*'

...
2020-02-18 06:43:45.654][12][debug][router] [source/common/router/router.cc:401] [C0][S10394085154332519343] cluster 'enroute_ratelimit' match for URL '/envoy.service.ratelimit.v2.RateLimitService/ShouldRateLimit'
[2020-02-18 06:43:45.654][12][debug][router] [source/common/router/router.cc:514] [C0][S10394085154332519343] router decoding headers:
':method', 'POST'
':path', '/envoy.service.ratelimit.v2.RateLimitService/ShouldRateLimit'
':status', '429'

...

'x-envoy-ratelimited', 'true'
'date', 'Tue, 18 Feb 2020 06:43:45 GMT'
'server', 'envoy'
...

Troubleshooting

Check Resources got created

$ kubectl get all -n enroute-gw-k8s
NAME                           READY   STATUS    RESTARTS   AGE
pod/enroute-795679bf56-2jhg9   3/3     Running   1          5h24m
pod/httpbin-5f69c4cf94-brknc   1/1     Running   0          5h20m

NAME              TYPE           CLUSTER-IP     EXTERNAL-               PORT(S)                      AGE
service/enroute   LoadBalancer   10.99.185.36   ....elb.amazonaws.com   80:30937/TCP,443:32323/TCP   5h21m
service/httpbin   ClusterIP      10.97.133.62   <none>                  80/TCP                       5h20m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/enroute   1/1     1            1           5h24m
deployment.apps/httpbin   1/1     1            1           5h20m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/enroute-795679bf56   1         1         1       5h24m
replicaset.apps/httpbin-5f69c4cf94   1         1         1       5h20m

Check Endpoint

$ kubectl get endpoints -n enroute-gw-k8s
NAME      ENDPOINTS                               AGE
enroute   192.168.200.5:8443,192.168.200.5:8080   5h24m
httpbin   192.168.200.6:80                        5h23m

Get a shell to enroute pod

$ kubectl -n enroute-gw-k8s exec enroute-795679bf56-2jhg9 --container=enroute -it -- /bin/bash
root@enroute-795679bf56-2jhg9:/bin#

Get a shell to envoy pod

$ kubectl -n enroute-gw-k8s exec enroute-795679bf56-2jhg9 --container=envoy -it -- /bin/bash
root@enroute-795679bf56-2jhg9:/#

Conclusion

We covered how the Enroute Kubernetes Ingress Gateway can be configured using a LoadBalancer service. Next we briefly walked through the filter architecture and how Enroute provides flexible global and per-route filters.

We went over the currently supported lua global filter and per-route rate-limit filter.

In subsequent articles, we’ll cover more advanced use-cases on using these filters. More documentation and use-cases can be found on https://getenroute.io

Enroute Kubernetes Ingress Gateway

Enroute Kubernetes Ingress Gateway

Enroute Universal Gateway

What this article covers

Enroute Config Model

Enroute Kubernetes Ingress Gateway in less than a minute

`GatewayHost` with `http_filter_lua` and `route_filter_ratelimit`

`GlobalConfig` to configure rate-limit engine

`RouteFilter` to configure per-route rate-limit directive

`HttpFilter` to configure lua code attached to `GatewayHost`

Running Enroute as a Kubernetes Ingress Gateway

Configuration Steps

Create Namespace, Deployment, LoadBalancer, Custom Resource Definitions and run example service

Enroute filters

Create global lua filter for all traffic

Send traffic

Using SSL

Create per-route rate-limiting filter for traffic matching that route.

Troubleshooting

Check Resources got created

Check Endpoint

Get a shell to enroute pod

Get a shell to envoy pod

Conclusion

Latest Posts

Are Shadow APIs breaking your security?

Can your API Gateway Tango to Open API Spec?

Why Every API Needs a Clock?

Categories

Tags

Enroute Kubernetes Ingress Gateway

Enroute Kubernetes Ingress Gateway

Enroute Universal Gateway

What this article covers

Enroute Config Model

Enroute Kubernetes Ingress Gateway in less than a minute

GatewayHost with http_filter_lua and route_filter_ratelimit

GlobalConfig to configure rate-limit engine

RouteFilter to configure per-route rate-limit directive

HttpFilter to configure lua code attached to GatewayHost

Running Enroute as a Kubernetes Ingress Gateway

Create Namespace, Deployment, LoadBalancer, Custom Resource Definitions and run example service

Create global lua filter for all traffic

Send traffic

Create per-route rate-limiting filter for traffic matching that route.

Check Resources got created

Check Endpoint

Get a shell to enroute pod

Get a shell to envoy pod

Latest Posts

Categories

Tags

`GatewayHost` with `http_filter_lua` and `route_filter_ratelimit`

`GlobalConfig` to configure rate-limit engine

`RouteFilter` to configure per-route rate-limit directive

`HttpFilter` to configure lua code attached to `GatewayHost`