Drive API Security at Kubernetes Ingress using Helm and Envoy

Drive API Security at Kubernetes Ingress using Helm and Envoy

Saaras Inc. April 19, 2021
Drive API Security at Kubernetes Ingress using Helm and Envoy

Introduction - EnRoute Helm Chart

Helm is a popular package manager choice for Kubernetes. Installation of software, managing versions, upgrading versions, and finding charts from the registry are key benefits of Helm.

EnRoute helm chart installs the EnRoute Ingress Controller and provides easy configuration options to define policy for a service. The helm chart provides fine-grained control to define L7 policies with its ability to enable/disable plugins for a service using configuration options that can be specified when helm is invoked.

For example, to enable the JWT plugin, simply invoke -

helm upgrade httpbin-service-policy saaras/service-policy       \
        --set service.namespace=demo-service                    \
        --set service.name=httpbin                              \
        --set service.port=80                                   \
        --set filters.jwt.enable=true                           \

This enables the JWT plugin for the service httpbin

EnRoute also supports plugins/filters to extend functionality and enforce policies. The features page lists the available plugins for the Gateway. More details about each of the plugins can also be found on plugin pages.

In the next few sections, we describe an extremely simple way to secure an application or a service in Kubernetes. Using the helm package manager and simple chart options, we describe how easy it is to secure the service using TLS and enforce L7 policy for an application.

Working with EnRoute Ingress API Gateway is organized in following steps

Kubernetes and Helm verison used

kubectl version
Client Version: version.Info{Major:"1", Minor:"20",
        GitVersion:"v1.20.1", GitCommit:"c4d752765b3bbac2237bf87cf0b1c2e307844666",
        GitTreeState:"clean", BuildDate:"2020-12-18T12:09:25Z",
        GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"18",
        GitVersion:"v1.18.10", GitCommit:"a84e568eeb56c4e3966314fc2d58374febd12ed7",
        GitTreeState:"clean", BuildDate:"2021-03-09T14:35:22Z",
        GoVersion:"go1.13.15 BoringCrypto", Compiler:"gc", Platform:"linux/amd64"}
helm version
version.BuildInfo{Version:"v3.4.2",
        GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}

Secure service using Helm, EnRoute, and Envoy

EnRoute Ingress Controller is a lightweight shim on Envoy proxy that works using filters/plugins. This plugin/filter model is similar to how Envoy proxy works and has a one-to-one correspondence between the filters on EnRoute and Envoy. The helm package for EnRoute has the ability to enable/disable these plugins for EnRoute gateway, which eventually enables/disables these plugins for Envoy.

EnRoute also supports plugins/filters to extend functionality and enforce policies. The features page lists the available plugins for the Gateway. More details about each of the plugins can also be found on plugin pages.

EnRoute is an actively maintained project with a modular approach to extending the gateway using plugins. Envoy functionality provided by filters can be enabled/disabled by enabling/disabling corresponding filters in EnRoute. For every service, filters can be enabled or disabled to control the L7 policy for it. Over time, plugins/filters are added on EnRoute to configure and control Envoy filters.

EnRoute also works in standalone mode without Kubernetes where simple REST APIs can be invoked to configure a docker-based gateway.

Enforce L7 Policy using helm

In the next few steps, we demonstrate how to apply L7 policy like RateLimit, JWT, CORS and enable Lua filter for a service. The next section demonstrates how to achive this for a service behind TLS where a certificate is automatically installed using Jetstacks Cert Manager

Add Saaras EnRoute helm repository

We first add the helm repository associated with EnRoute

helm repo add saaras https://getenroute.io

Check the list of added repositories.

helm repo list
NAME            URL
saaras          https://getenroute.io

Install EnRoute CRDs

EnRoute CRDs have a helm package that can be installed using helm

helm install saaras/enroute-crds --generate-name

Securly expose your own service

If you have a service running in Kubernetes, it can be securly exposed using the EnRoute helm chart. We demonstrate this by creating service httpbin and securly exposing it using the helm chart.

Install EnRoute API Ingress Controller
helm install enroute-service saaras/enroute \
  --set awsSettings.enable=true             \
  --set enrouteService.rbac.create=true

This creates the EnRoute service in enroutedemo namespace and programs the external load balancer using LoadBalancer service with a public IP

The above example sets annotations for external Load balancer for AWS. Similar annotations can be set for DigitalOcean (digitaloceanSettings.enable=true) and Oracle Cloud (ociSettings.enable=true)

kubectl get all -n enroutedemo
NAME                           READY   STATUS    RESTARTS   AGE
pod/enroute-7db7485b54-nxtgd   3/3     Running   1          3m52s

NAME              TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                      AGE
service/enroute   LoadBalancer   10.96.248.140   129.159.43.209   80:30219/TCP,443:31502/TCP   3m53s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/enroute   1/1     1            1           3m53s

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/enroute-7db7485b54   1         1         1       3m53s

The above command installs the EnRoute API Ingress Controller in the enroutedemo namespace.

Create demo service httpbin or use your own service

To demonstrate this workflow, we begin by creating httpbin service. If you have your own service running, you can use that instead of httpbin

Create namespace for demo service -

kubectl create namespace demo-service

Create httpbin as service -

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: httpbin
  namespace: demo-service
  labels:
    app: httpbin
spec:
  containers:
  - name: httpbin
    image: kennethreitz/httpbin
    ports:
      - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: httpbin
  namespace: demo-service
spec:
  selector:
    app: httpbin
  ports:
  - port: 80
EOF
Expose this service using EnRoute API Gateway Controller

The helm command below makes the httpbin service accessible externally. Note the port value and prefix specified here. These values are used to setup EnRoute CRDs to securly expose the service.

helm install httpbin-service-policy saaras/service-policy \
        --set service.name=httpbin                        \
        --set service.namespace=demo-service              \
        --set service.prefix=/get                         \
        --set service.port=80

The above command assumes that a service called httpbin is running on port 80. It maps the service to the route /get

kubectl get all -n enroutedemo
NAME                           READY   STATUS    RESTARTS   AGE
pod/enroute-7db7485b54-v8k66   3/3     Running   1          11m
pod/httpbin                    1/1     Running   0          4m

NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP        PORT(S)                      AGE
service/enroute        LoadBalancer   10.96.198.222   129.159.43.209     80:32417/TCP,443:30298/TCP   11m
service/externalauth   ExternalName   <none>          saaras.auth0.com   443/TCP                      11m
service/httpbin        ClusterIP      10.96.70.88     <none>             80/TCP                       4m

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/enroute   1/1     1            1           11m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/enroute-7db7485b54   1         1         1       11m
$ curl -v -s 129.159.43.209/get
*   Trying 129.159.43.209...
* Connected to 129.159.43.209 (129.159.43.209) port 80 (#0)
> GET /get HTTP/1.1
> Host: 129.159.43.209
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< server: envoy
< date: Tue, 13 Apr 2021 23:39:32 GMT
< content-type: application/json
< content-length: 299
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 2
< vary: Accept-Encoding
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "0",
    "Host": "129.159.43.209",
    "User-Agent": "curl/7.47.0",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Internal": "true"
  },
  "origin": "10.0.20.143",
  "url": "http://129.159.43.209/get"
}
* Connection #0 to host 129.159.43.209 left intact
Enable TLS

This section describes how easy it install a SSL certificate in front of your service. The SSL certificate is signed using Let’s Encrypt using the ACME protocol.


Need help with TLS/SSL?       Slack        Send Us a Note      

The following steps are involved in achieving this -

  • We first install Jetstacks’s Cert Manager
  • Create a DNS A that points the domain name to the External-IP address allocated by LoadBalancer service
  • Use the helm chart to sign a certificate using the Let’s Encrypt staging server
  • Use the helm chart to sign a certificate using the Let’s Encrypt production server
  • Install the certificate for the service
Install Jetstack’s Cert Manager

Jetstack’s Cert manager is a Kubernetes native project that can be installed on the cluster using helm chart and used to issue certificates from Let’s Encrypt.

Add the Jetstack’s cert manager repository to helm repos

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm search repo

Install Jetstack’s Cert manager

helm install                         \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager           \
  --version v1.2.0                   \
  --create-namespace                 \
  --set installCRDs=true

Check the cert manager installation

kubectl get all -n cert-manager
NAME                                           READY   STATUS    RESTARTS   AGE
pod/cert-manager-56f5c44b5d-wshwn              1/1     Running   0          27h
pod/cert-manager-cainjector-5f494679f9-q8jhq   1/1     Running   0          27h
pod/cert-manager-webhook-5f59d4c7bd-xhqgj      1/1     Running   0          27h

NAME                           TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/cert-manager           ClusterIP   10.96.82.166    <none>        9402/TCP   27h
service/cert-manager-webhook   ClusterIP   10.96.239.164   <none>        443/TCP    27h

NAME                                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/cert-manager              1/1     1            1           27h
deployment.apps/cert-manager-cainjector   1/1     1            1           27h
deployment.apps/cert-manager-webhook      1/1     1            1           27h

NAME                                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/cert-manager-56f5c44b5d              1         1         1       27h
replicaset.apps/cert-manager-cainjector-5f494679f9   1         1         1       27h
replicaset.apps/cert-manager-webhook-5f59d4c7bd      1         1         1       27h
Create a DNS A record that points to the domain name for the service

First query the value of external IP allocated by external load balancer

kubectl get service enroute -n enroutedemo -o jsonpath='{..status}'

{
  "loadBalancer": {
    "ingress": [
      {
        "ip": "129.159.43.209"
      }
    ]
  }
}

Next create a DNS A record for the domain name to be used for the service (or Common Name for the certificate). We’ll use the domain name httpbin.enroutedemo.com and associate it with 129.159.43.209

Once a DNS record has been created, ensure that it correctly points to the external IP.

ping httpbin.enroutedemo.com
PING httpbin.enroutedemo.com (129.159.43.209) 56(84) bytes of data.
Issue and Install Certificate using Let’s Encrypt
helm upgrade httpbin-service-policy saaras/service-policy       \
        --set service.namespace=demo-service                    \
        --set service.name=httpbin                              \
        --set service.prefix=/get                               \
        --set service.port=80                                   \
        --set service.enableTLS=true                            \
        --set autoTLS.certificateCN=httpbin.enroutedemo.com     \
        --set autoTLS.createIssuers=true                        \
        --set autoTLS.email=contact@example.com                 \
        --set autoTLS.enableProd=true                                 

We provide the Common Name for which we want to issue a certificate and the namespace in which the certificate will be placed. The autoTLS.enableProd switch uses the Let’s Encrypt production server. If this switch is not specified, the staging server is used.

kubectl get certificates.cert-manager.io --all-namespaces
NAMESPACE      NAME                      READY   SECRET                    AGE
demo-service   httpbin.enroutedemo.com   True    httpbin.enroutedemo.com   52s
Enable L7 filters/plugins to enforce policy

The helm chart supports enabling/disabling filters for the service

helm upgrade httpbin-service-policy saaras/service-policy       \
        --set service.namespace=demo-service                    \
        --set service.name=httpbin                              \
        --set service.prefix=/get                               \
        --set service.port=80                                   \
        --set service.enableTLS=true                            \
        --set autoTLS.certificateCN=httpbin.enroutedemo.com     \
        --set autoTLS.enableProd=true                           \
        --set autoTLS.createIssuers=true                        \
        --set autoTLS.email=contact@example.com                 \
        --set filters.lua.enable=true                           \
        --set filters.cors.enable=true                          \
        --set filters.jwt.enable=false                          \
        --set filters.ratelimit.enable=true                     

The above command issues a Certificate using Let’s Encrypt and installs it for the service. It also enables Lua, CORS and RateLimit filters.

The JWT filter is intially disabled. Let’s Encrypt uses HTTP01 verification and needs to access the /.well-known/acme-challenge/ url.

We can verify the cert installed by sending a request

curl -v https://httpbin.enroutedemo.com/get
*   Trying 129.159.43.209...
* Connected to httpbin.enroutedemo.com (129.159.43.209) port 443 (#0)
* found 129 certificates in /etc/ssl/certs/ca-certificates.crt
* found 516 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: httpbin.enroutedemo.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=httpbin.enroutedemo.com
*        start date: Fri, 16 Apr 2021 23:02:34 GMT
*        expire date: Thu, 15 Jul 2021 23:02:34 GMT
*        issuer: C=US,O=Let's Encrypt,CN=R3
*        compression: NULL
* ALPN, server accepted to use http/1.1
> GET /get HTTP/1.1
> Host: httpbin.enroutedemo.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< server: envoy
< date: Sat, 17 Apr 2021 00:07:27 GMT
< content-type: application/json
< content-length: 320
< access-control-allow-origin: *
< access-control-allow-credentials: true
< x-envoy-upstream-service-time: 11
< vary: Accept-Encoding
<
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "0",
    "Host": "httpbin.enroutedemo.com",
    "User-Agent": "curl/7.47.0",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-Internal": "true"
  },
  "origin": "10.0.20.103",
  "url": "https://httpbin.enroutedemo.com/get"
}

In the next section we explain the artifacts created using the above helm chart with a brief description.


Questions about filter/plugins?       Slack        Send Us a Note      

The above helm chart creates the following configuration objects.

GatewayHost for the service

GatewayHost is a high level construct to expose the service. It also specifies the filters applicable to the service. Both global and route filters are specified here.

The GatewayHost is created using values specified while exposing the service using the helm chart. For example, the path /get is set using the service.prefix option in helm chart.

The Secret httpbin.enroutedemo.com was created using Let’s Encrypt and assigned to the service along with the Fqdn which is set to CN of the certificate httpbin.enroutedemo.com

The naming follows the ServiceName-DomainName-Object convention. For example, the GatewayHost is named as httpbin-httpbin.enroutedemo.com-gatewayhost

kubectl describe gatewayhosts.enroute.saaras.io -n demo-service httpbin-httpbin.enroutedemo.com-gatewayhost
Name:         httpbin-httpbin.enroutedemo.com-gatewayhost
Namespace:    demo-service
Labels:       app=httpbin
              app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  enroute.saaras.io/v1beta1
Kind:         GatewayHost
Metadata:
  Creation Timestamp:  2021-04-17T00:00:27Z
  Generation:          2
...
    Operation:       Update
    Time:            2021-04-17T00:02:18Z
  Resource Version:  47879574
  Self Link:         /apis/enroute.saaras.io/v1beta1/namespaces/demo-service/gatewayhosts/httpbin-httpbin.enroutedemo.com-gatewayhost
  UID:               cccc2a6a-753c-47af-8e57-19cf5929b199
Spec:
  Routes:
    Conditions:
      Header:
        Exact:  GET
        Name:   :method
      Prefix:   /.well-known/acme-challenge/
    Services:
      Name:  acme-challenge-service
      Port:  80
    Conditions:
      Prefix:  /get
    Filters:
      Name:  httpbin-rl2
      Type:  route_filter_ratelimit
    Services:
      Name:  httpbin
      Port:  80
  Virtualhost:
    Filters:
      Name:  httpbin-luatestfilter
      Type:  http_filter_lua
      Name:  httpbin-corsfilter
      Type:  http_filter_cors
    Fqdn:    httpbin.enroutedemo.com
    Tls:
      Secret Name:  httpbin.enroutedemo.com
Events:             <none>

HttpFilter for the service

The helm script also installs HttpFilter for each filter configured. There is one installed for Lua, Cors and JWT.

kubectl describe -n demo-service httpfilters.enroute.saaras.io 
Name:         httpbin-corsfilter
Namespace:    demo-service
Labels:       app=httpbin-app
              app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  enroute.saaras.io/v1beta1
Kind:         HttpFilter
Metadata:
  Creation Timestamp:  2021-04-16T21:05:48Z
  Generation:          1
...
    Operation:       Update
    Time:            2021-04-16T21:05:48Z
  Resource Version:  47832791
  Self Link:         /apis/enroute.saaras.io/v1beta1/namespaces/demo-service/httpfilters/httpbin-corsfilter
  UID:               d9fe2628-3c61-47a2-ade9-b8151b107fb6
Spec:
  Http Filter Config:
    Config:  {
    "match_condition" : {
      "regex" : "\\*"
    },
    "access_control_allow_methods" : "GET, OPTIONS",
    "access_control_allow_headers" : "Content-Type",
    "access_control_expose_headers" : "*",
    "access_control_max_age" : "120"
}

  Name:  httpbin-corsfilter
  Type:  http_filter_cors
Events:  <none>

More description about the CORS filter can be found on the Cors filter page

The HttpFilter for JWT validates incoming JWT Tokens.

Name:         httpbin-jwtfilter
Namespace:    demo-service
Labels:       app=httpbin-app
              app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  enroute.saaras.io/v1beta1
Kind:         HttpFilter
Metadata:
  Creation Timestamp:  2021-04-16T21:05:48Z
  Generation:          1
  Managed Fields:
...
    Operation:       Update
    Time:            2021-04-16T21:05:48Z
  Resource Version:  47832790
  Self Link:         /apis/enroute.saaras.io/v1beta1/namespaces/demo-service/httpfilters/httpbin-jwtfilter
  UID:               ee11cf5a-3507-45ab-93fa-0e9d5ba9e972
Spec:
  Http Filter Config:
    Config:  {
  "name" : "auth0",
  "jwks_uri" : "https://saaras.auth0.com/.well-known/jwks.json",
  "audience" : "api-identifier",
  "issuer" : "https://saaras.auth0.com/",
  "route" : [{"prefix" : "/"}],
  "jwt_service_name" : "auth0",
  "jwt_service_port" : 443,
  "jwt_forward_header_name" : "x-jwt-token"
}

  Name:  httpbin-jwtfilter
  Services:
    Name:      httpbin-externalauth
    Port:      443
    Protocol:  tls
  Type:        http_filter_jwt
Events:        <none>

More description about the JWT filter can be found on the JWT filter page

The Lua HttpFilter invokes a Lua script on Request and Reponse path.

Name:         httpbin-luatestfilter
Namespace:    demo-service
Labels:       app=httpbin-app
              app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  enroute.saaras.io/v1beta1
Kind:         HttpFilter
Metadata:
  Creation Timestamp:  2021-04-16T21:05:48Z
  Generation:          1
...
    Operation:       Update
    Time:            2021-04-16T21:05:48Z
  Resource Version:  47832792
  Self Link:         /apis/enroute.saaras.io/v1beta1/namespaces/demo-service/httpfilters/httpbin-luatestfilter
  UID:               ccb62afe-0682-4b83-bf15-62a26c0cbe23
Spec:
  Http Filter Config:
    Config:  function envoy_on_request(request_handle)
   request_handle:logInfo("Hello World request");
end

function envoy_on_response(response_handle)
   response_handle:logInfo("Hello World response");
end

  Name:  httpbin-luatestfilter
  Type:  http_filter_lua
Events:  <none>

RouteFilter for the service

More description about the Lua filter can be found on the Lua filter page

Rate Limits are enforced per-route and configured using the RouteFilter

kubectl describe -n demo-service routefilters.enroute.saaras.io
Name:         httpbin-rl2
Namespace:    demo-service
Labels:       app=httpbin-app
              app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  enroute.saaras.io/v1beta1
Kind:         RouteFilter
Metadata:
  Creation Timestamp:  2021-04-16T21:05:48Z
  Generation:          1
...
    Operation:       Update
    Time:            2021-04-16T21:05:48Z
  Resource Version:  47832793
  Self Link:         /apis/enroute.saaras.io/v1beta1/namespaces/demo-service/routefilters/httpbin-rl2
  UID:               88ca9f0b-a37e-4232-9ffb-6b8a822008d4
Spec:
  Name:  httpbin-rl2
  Route Filter Config:
    Config:  {
  "descriptors" :
  [
    { "request_headers": { "header_name": "x-forwarded-for", "descriptor_key": "x-forwarded-for" } },
    { "request_headers": { "header_name": "x-forwarded-proto", "descriptor_key": "x-forwarded-proto" } },
    { "generic_key": { "descriptor_value" : "default_route" } }
  ]
}

  Type:  route_filter_ratelimit
Events:  <none>

More information about how to configure rate limits and the route rate limits can be found on the rate limit filter page

The GlobalConfig of type globalconfig_ratelimit is used to configure the global rate limit engine. The global rate limit engine takes incoming requests sent to it when every request is processed. During request processing, when a matching route has a RouteFilter of type route_filter_ratelimit, the corresponding L7 values are sent to the global rate limit engine to check for limits. The global rate limit engine, using its rate limit config determines the rate limits that should be applied to this bucket.

In the current example, for every x-forwarded-for, for protocol https, for default_route, a key of the kind enroute_x-forwarded-for_10.0.20.90_x-forwarded-proto_https_generic_key_default_route_1617919413 is created by the rate limit engine. These keys are counted using a redis cache and when the limit exceeds the configured value, a 429 is returned.

GlobalConfig for the system to configure the global rate-limit engine

kubectl describe -n demo-service globalconfigs.enroute.saaras.io
Name:         httpbin-rl-global-config
Namespace:    demo-service
Labels:       app=httpbin-app
              app.kubernetes.io/managed-by=Helm
Annotations:  meta.helm.sh/release-name: httpbin-service-policy
              meta.helm.sh/release-namespace: default
API Version:  enroute.saaras.io/v1beta1
Kind:         GlobalConfig
Metadata:
  Creation Timestamp:  2021-04-16T21:05:48Z
  Generation:          1
...
    Operation:       Update
    Time:            2021-04-16T21:05:48Z
  Resource Version:  47832789
  Self Link:         /apis/enroute.saaras.io/v1beta1/namespaces/demo-service/globalconfigs/httpbin-rl-global-config
  UID:               c3e89ce8-8513-45d6-86b3-33edc365e89b
Spec:
  Config:  {
  "domain": "enroute",
  "descriptors" :
  [
    {
      "key": "x-forwarded-for",
      "descriptors" :
      [
        {
          "key" : "x-forwarded-proto",
          "value" : "http",
          "descriptors" : [
           {
             "key" : "generic_key",
             "value" : "default_route",
             "rate_limit" : { "unit" : "second", "requests_per_unit" : 5 }
            }
          ]
        },
        {
          "key" : "x-forwarded-proto",
          "value" : "https",
          "descriptors" : [
           {
             "key" : "generic_key",
             "value" : "default_route",
             "rate_limit" : { "unit" : "second", "requests_per_unit" : 2 }
           }
          ]
        }
      ]
    }
  ]
}

  Name:  httpbin-rl-global-config
  Type:  globalconfig_ratelimit
Events:  <none>

In addition to the artificats created to control EnRoute API Gateway, additional artificats are created to install a certificate. More information about it can be found in the Cert Manager document

Helm switches to enable/disable features

EnRoute helm chart provides configuration options for multiple clouds and switches to enable/disable plugins or filters. These switches provide a fine-grained control over L7 policy.

The EnRoute Helm chart supports the following filter switches -

SwitchValid ValuesDescription
envoySettings.logLevelOne of -
["trace", "debug", "info", "error"]
Log level for underlying Envoy
EnRoute Ingress Controller Service Settings
enrouteService.enableOne of -
["true", "false"]
Disabled by default. Enable when installing EnRoute the first time.
Enables creation of EnRoute deployment and service
enrouteService.replicaCountNumber of replicas eg: 3By default, one instance of EnRoute pod is created
Change this value to increase/decrease count of replicas
Cloud Settings
awsSettings.enableOne of -
["true", "false"]
Disabled by default. Enable when running on AWS.
Enables propagation of client-IP to backend service
ociSettings.enableOne of -
["true", "false"]
Disabled by default. Enable when running on OCI.
Enables propagation of client-IP to backend service
digitaloceanSettings.enableOne of -
["true", "false"]
Disabled by default. Enable when running on DigitalOcean.
Enables propagation of client-IP to backend service
Filter Settings
filters.lua.enableOne of -
["true", "false"]
Enabled by default. Enable when you want to install a Lua filter for service.
filters.ratelimit.enableOne of -
["true", "false"]
Enabled by default. Enable when you want to install a RateLimit filter for service.
filters.cors.enableOne of -
["true", "false"]
Disabled by default. Enable when you want to install a CORS filter for service.
filters.jwt.enableOne of -
["true", "false"]
Disabled by default. Enable when you want to install a JWT filter for service.
Service Settings
service.nameA string representing service name in KubernetesDefault is "hello-enroute" service. Set it to your own service name
service.portPort on which to access this serviceDefault is 9090, the port for "hello-enroute" service. Set it to your own service port
service.enableTLSWhen securing the service using TLS, use this flag after cert-manager has been setup.Default is false. Enable this flag to automatically get a certificate for a service from Let's encrypt
service.namespaceThe namespace in which the service is runningThis setting is used to determine the namespace in which to create EnRoute Filters and ```GatewayHost```
service.prefixThe prefix on which the service is reachableThis installs the prefix route for this service
Auto-TLS Settings
autoTLS.issueCertOne of -
["true", "false"]
Enable this to issue a certificate signed using the ACME protocol (from Let's Encrypt)
autoTLS.certificateCNCommon Name for CertificateCommon Name to use for Certificate. This also gets installed as ```fqdn``` for the service
autoTLS.enableProdOne of -
["true", "false"]
Default is false.
When set to false, it uses the ACME staging server
https://acme-staging-v02.api.letsencrypt.org/directory .
When set to true, it uses the ACME production server
https://acme-v02.api.letsencrypt.org/directory
As filters/plugins are added to EnRoute, simple switches will allow enabling/disabling these features.

Questions about the helm chart?       Slack        Send Us a Note      

Advantages of Using EnRoute Ingress Controller

There are several distinct advantages of running an Ingress Controller and enforcing policies at Ingress.

  • Ingress provides a portable mechanism to enforce policy inside the Kubernetes Cluster. Policies enforced inside a cluster are easier to port across clouds.
  • Ingress can be scaled horizontally inside the Kubernetes Cluster. Elasticity of L7 fabric makes it easier to operate and scale it
  • L7 policies can be hosted along with services inside the cluster with cluster-native state storage
  • Keeping L7 policy closer to services simplifies policy enforcement and troubleshooting of services and APIs.

Plugins for fine-grained traffic control

EnRoute uses Envoy as the underlying proxy to provide the L7 ingress function. EnRoute has a modular architecture that closely reflects Envoy’s extensible model.

Plugins/Filters can be defined at the route level or service level to enforce L8 policies at Ingress. EnRoute provides advanced rate-limiting plugin in the community edition completely free without any limits. Clocking your APIs and micro-services using deep L7 state is a critical need and EnRoutes flexible rate-limiting funtion provides immense flexibility to match a variety of rate-limiting use-cases.

EnRoute Enterprise includes support and enterprise plugins that help secure traffic at Kubernetes Ingress.


Get questions answered       Slack        Send Us a Note      

Next Steps

Working with EnRoute Ingress API Gateway is organized in following steps