Enroute Kubernetes Ingress Gateway
Enroute Kubernetes Ingress Gateway
Enroute Universal Gateway
Enroute Universal Gateway is a flexible API gateway built to support traditional and cloud-native use cases. It is designed to run either as a [Kubernetes Ingress Gateway, Standalone Gateway, Horizontally scaling L7 API gateway or a Mesh of Gateways] (/blog/enroute-topologies/) . Depending on the need of the user, the environment, the application, either one or many of these solutions can be deployed.
A consistent policy framework across all these network components makes the Enroute Universal Gateway a versatile and powerful solution.
What this article covers
This article covers how to get started with the Enroute Kubernetes Ingress Gateway. There are several options to running a kubernetes ingress gateway. This document only covers a cloud provider where service of type LoadBalancer is supported. Alternative mechanisms include using a NodePort which will be convered in another article.
To get a more detailed understanding of Enroute Universal Gateway and its architecture, refer to the article here To run Enroute outside of kubernetes as a standalone gateway, refer to the article on standalone gateway
Enroute also supports several other topologies including a Standalone Gateway and Gateway Mesh topology. One Enroute control plane can program multiple stateless Enroute data planes. Every Enroute data plane instance runs an instance of Envoy along with it. Only the Kubernetes Ingress topology is covered in this article.
Enroute Config Model
Enroute follows a configuration model similar to Envoy. It uses filters to extend functionality at the global Service level and per-route level. The config objects used for this are - GatewayHost, Route, HttpFilter, RouteFilter and Service as shown here for Kubernetes gateway.
Enroute Kubernetes Ingress Gateway in less than a minute
Use the following command to create all the configuration in one step. A step-by-step walkthrough is shown below -
bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/ic-all.yaml'While this step creates all CRDs, Roles and Enroute Objects, here are the important ones of interest -
GatewayHost with http_filter_lua and route_filter_ratelimit
---
apiVersion: enroute.saaras.io/v1beta1
kind: GatewayHost
metadata:
labels:
app: httpbin
name: httpbin
namespace: enroute-gw-k8s
spec:
virtualhost:
fqdn: demo.saaras.io
filters:
- name: luatestfilter
type: http_filter_lua
routes:
- match: /
services:
- name: httpbin
port: 80
filters:
- name: rl2
type: route_filter_ratelimit
---GlobalConfig to configure rate-limit engine
---
apiVersion: enroute.saaras.io/v1beta1
kind: GlobalConfig
metadata:
labels:
app: httpbin
name: rl-global-config
namespace: enroute-gw-k8s
spec:
name: rl-global-config
type: globalconfig_ratelimit
config: |
{
"domain": "enroute",
"descriptors" :
[
{
"key" : "generic_key",
"value" : "default",
"rate_limit" :
{
"unit" : "second",
"requests_per_unit" : 10
}
}
]
}
---RouteFilter to configure per-route rate-limit directive
---
apiVersion: enroute.saaras.io/v1beta1
kind: RouteFilter
metadata:
labels:
app: httpbin
name: rl2
namespace: enroute-gw-k8s
spec:
name: rl2
type: route_filter_ratelimit
routeFilterConfig:
config: |
{
"descriptors" :
[
{
"generic_key":
{
"descriptor_value":"default"
}
}
]
}
---HttpFilter to configure lua code attached to GatewayHost
---
apiVersion: enroute.saaras.io/v1beta1
kind: HttpFilter
metadata:
labels:
app: httpbin
name: luatestfilter
namespace: enroute-gw-k8s
spec:
name: luatestfilter
type: http_filter_lua
httpFilterConfig:
config: |
function envoy_on_request(request_handle)
request_handle:logInfo("Hello World request httpbin");
end
function envoy_on_response(response_handle)
response_handle:logInfo("Hello World response httpbin");
end
---Running Enroute as a Kubernetes Ingress Gateway
Enroute leverages existing open source to function as an API gateway for microservices running inside a kubernetes cluster. Enroute uses filter configuration model to attach global filters and per-route filters to provide the flexible configuration. This configuration model is consistent with Enroute standalone gateway.
We’ll perform the following steps to run Enroute Kubernetes Ingress Gateway.
We’ll create a namespace and roles for running Enroute. Next we’ll create a deployment to run Enroute. We’ll create a LoadBalancer service which provides us with an external DNS name or IP as an entry-point into cluster.
We’ll then create a test service and configure it using CRDs. The CRDs let us attach additional global and route-level filter configuration with Enroute.
Once configured, we’ll send some traffic to it.
The next steps assumes you have a kubernetes cluster setup on a cloud that supports the LoadBalancer type of service. In a subsequent article we’ll describe how to run Enroute for cloud that do not support this service.
Configuration Steps
Create Namespace, Deployment, LoadBalancer, Custom Resource Definitions and run example service
All the enroute configuration will be hosted in a separate namespace. Some important config objects are also highlighted along with their links.
Create namespace and roles using -
bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/ns.yaml'For this example, we run Enroute as a kubernetes deployment.
bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/deployment.yaml'For this example, we’ll use a service of type LoadBalancer.
bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/lb.yaml'When Enroute is used as an ingress gateway, the configuration can be provided using CRDs.
bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/crds.yaml'We run example service httpbin to demonstrate Enroute’s capabilities
bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/httpbin.yaml'Enroute filters
A global filter applies to all traffic going through Enroute. A per-route filter applies to traffic that matches that route. Enroute as of current version supports global lua filter and per-route advanced ratelimiting filter.
Filters are created using CRDs and got created in the step above.
Create global lua filter for all traffic
The HttpFilter.enroute.saaras.io/v1beta1 CRD is a way to specify a Global HTTP filter.
bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/httpbin.filter.lua.yaml'This filter can then be attached at the global level.
bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/httpbin.filter.lua.attach.yaml'The lua script invokes a function on the request and response path. This can be verified by sending a request and checking logs. Note below that 10.97.133.62 is the ClusterIP for the LoadBalancer service
Send traffic
Note that to match the domain name that we have programmed (demo.saaras.io), we need to use the --resolve functionality provided by curl
$ curl --resolve demo.saaras.io:80:10.97.133.62 http://demo.saaras.io/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Host": "demo.saaras.io",
"User-Agent": "curl/7.58.0"
},
"origin": "192.168.42.192",
"url": "http://demo.saaras.io/get"
}Where 10.97.133.62 is the Service IP.
Using SSL
For SSL to work, we need a certificate installed with the appropriate domain name. As an example, we’ll use demo.saaras.io as the domain name. This domain name will be provided at three locations
- When programming the Enroute Ingress Controller, the value of Fqdn will be set to this value.
- A CNAME record that maps this domain to the public domain provided by
LoadBalancerservice by AWS. When passing traffic, we use this domain.
As an example for this demo, you can use script to create a certificate and key. The contents of the script are shown below. It creates the key, self-signed certificate and creates a kubernetes secret using them.
#!/bin/bash
if [ $# -gt 1 ]; then
cn='localhost'
cn=$1
echo "Received cn as" $1
openssl req -new -newkey rsa:$2 -days 36500 -nodes -x509 \
-subj "/C=US/ST=Denial/L=Springfield/O=Dis/CN=$cn" \
-keyout $1.key -out $1.cert
openssl x509 -in $cn.cert -text -noout
else
echo "Script to generate self signed certificate with a provided CN name and keysize"
echo "On successful execution, output is <cn_name.key> and <cn_name.cert>"
echo "Usage: ./self_signed_cert.sh <cn_name> <keysize>"
echo "Example: ./self_signed_cert.sh demo.saaras.io 4096"
echo "Example: ./self_signed_cert.sh demo.saaras.io 2048"
fi
kubectl -n enroute-gw-k8s create secret tls tls-secret-v0.4.0-httpbin-local --cert=$1.cert --key=$1.keySend traffic
$ curl -k https://demo.saaras.io/get
GET /get HTTP/2
Host: demo.saaras.io
User-Agent: curl/7.54.0
Accept: */*
HTTP/2 200
server: envoy
date: Thu, 26 Mar 2020 00:07:15 GMT
content-type: application/json
content-length: 300
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 3
{
"args": {},
"headers": {
"Accept": "*/*",
"Content-Length": "0",
"Host": "demo.saaras.io",
"User-Agent": "curl/7.54.0",
"X-Envoy-Expected-Rq-Timeout-Ms": "15000",
"X-Envoy-Internal": "true"
},
"origin": "10.0.24.142",
"url": "https://demo.saaras.io/get"
}
* Connection #0 to host demo.saaras.io left intactCheck logs
kubectl logs -n enroute-gw-k8s enroute-795679bf56-2jhg9 envoy
...
[2020-02-18 06:25:00.771][12][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:585] script log: Hello World request httpbin
[2020-02-18 06:25:00.771][12][debug][lua] [source/extensions/filters/common/lua/lua.cc:37] coroutine finished
...
2020-02-18 06:25:00.773][12][trace][lua] [bazel-out/k8-opt/bin/source/extensions/filters/common/lua/_virtual_includes/lua_lib/extensions/filters/common/lua/lua.h:122] creating N5Envoy10Extensions11HttpFilters3Lua19StreamHandleWrapperE at 0x41b07138
[2020-02-18 06:25:00.773][12][info][lua] [source/extensions/filters/http/lua/lua_filter.cc:585] script log: Hello World response httpbin
[2020-02-18 06:25:00.773][12][debug][lua] [source/extensions/filters/common/lua/lua.cc:37] coroutine finished
...
':status', '200'
'server', 'envoy'
'date', 'Tue, 18 Feb 2020 06:25:00 GMT'
'content-type', 'application/json'
'content-length', '300'
'access-control-allow-origin', '*'
'access-control-allow-credentials', 'true'
'x-envoy-upstream-service-time', '1'Create per-route rate-limiting filter for traffic matching that route.
The RouteFilter.enroute.saaras.io/v1beta1 and GlobalConfig.enroute.saaras.io/v1beta1 are CRDs to specify per-route rate-limit config.
bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/httpbin.filter.rl.yaml'This filter can then be attached on a per-route basis.
bash -c 'kubectl apply -f https://getenroute.io/ic/v0.4.0/qs/httpbin.filter.lua.rl.attach.yaml'The rate limit filter invokes the rate-limit service configured using the GlobalConfig.enroute.saaras.io/v1beta1. In the next article, we’ll cover more details on using the rate-limit filter and advanced configuration to run different use-cases. Note below that 10.99.185.36 is the ClusterIP for the LoadBalancer service
$ while true; do curl -vvv -k --resolve demo.saaras.io:80:10.99.185.36 http://demo.saaras.io/get; done
* Added demo.saaras.io:80:10.99.185.36 to DNS cache
* Hostname demo.saaras.io was found in DNS cache
* Trying 10.99.185.36...
* TCP_NODELAY set
* Connected to demo.saaras.io (10.99.185.36) port 80 (#0)
> GET /get HTTP/1.1
> Host: demo.saaras.io
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 429 Too Many Requests
< x-envoy-ratelimited: true
< date: Tue, 18 Feb 2020 06:46:15 GMT
< server: envoy
< content-length: 0
<Check logs to verify
...
[2020-02-18 06:43:45.654][12][debug][http] [source/common/http/conn_manager_impl.cc:600] [C8018][S13203667490673188384] request headers complete (end_stream=true):
':authority', 'demo.saaras.io'
':path', '/get'
':method', 'GET'
'user-agent', 'curl/7.58.0'
'accept', '*/*'
...
2020-02-18 06:43:45.654][12][debug][router] [source/common/router/router.cc:401] [C0][S10394085154332519343] cluster 'enroute_ratelimit' match for URL '/envoy.service.ratelimit.v2.RateLimitService/ShouldRateLimit'
[2020-02-18 06:43:45.654][12][debug][router] [source/common/router/router.cc:514] [C0][S10394085154332519343] router decoding headers:
':method', 'POST'
':path', '/envoy.service.ratelimit.v2.RateLimitService/ShouldRateLimit'
':status', '429'
...
'x-envoy-ratelimited', 'true'
'date', 'Tue, 18 Feb 2020 06:43:45 GMT'
'server', 'envoy'
...Troubleshooting
Check Resources got created
$ kubectl get all -n enroute-gw-k8s
NAME READY STATUS RESTARTS AGE
pod/enroute-795679bf56-2jhg9 3/3 Running 1 5h24m
pod/httpbin-5f69c4cf94-brknc 1/1 Running 0 5h20m
NAME TYPE CLUSTER-IP EXTERNAL- PORT(S) AGE
service/enroute LoadBalancer 10.99.185.36 ....elb.amazonaws.com 80:30937/TCP,443:32323/TCP 5h21m
service/httpbin ClusterIP 10.97.133.62 <none> 80/TCP 5h20m
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/enroute 1/1 1 1 5h24m
deployment.apps/httpbin 1/1 1 1 5h20m
NAME DESIRED CURRENT READY AGE
replicaset.apps/enroute-795679bf56 1 1 1 5h24m
replicaset.apps/httpbin-5f69c4cf94 1 1 1 5h20mCheck Endpoint
$ kubectl get endpoints -n enroute-gw-k8s
NAME ENDPOINTS AGE
enroute 192.168.200.5:8443,192.168.200.5:8080 5h24m
httpbin 192.168.200.6:80 5h23mGet a shell to enroute pod
$ kubectl -n enroute-gw-k8s exec enroute-795679bf56-2jhg9 --container=enroute -it -- /bin/bash
root@enroute-795679bf56-2jhg9:/bin#Get a shell to envoy pod
$ kubectl -n enroute-gw-k8s exec enroute-795679bf56-2jhg9 --container=envoy -it -- /bin/bash
root@enroute-795679bf56-2jhg9:/#Conclusion
We covered how the Enroute Kubernetes Ingress Gateway can be configured using a LoadBalancer service. Next we briefly walked through the filter architecture and how Enroute provides flexible global and per-route filters.
We went over the currently supported lua global filter and per-route rate-limit filter.
In subsequent articles, we’ll cover more advanced use-cases on using these filters. More documentation and use-cases can be found on https://getenroute.io
Latest Posts
Are Shadow APIs breaking your security?
26 Aug 2020