Canary releasing new software using Enroute API Gateway

Reducing risk of rolling out software

Canary release is a well known technique to reduce the risk in rolling out new software. Here we demonstrate how Enroute API gateway can be programmed for this. Note that the same thing can be done using non-GW enroute deployments when data plane is run separately.

Controlling traffic percentage to a service dynamically

Enroute architecture provides a way to control traffic sent to a service dynamically. The data plane component keeps itself in sync with the control plane by design. A REST call to update the traffic split on the control plane then automatically gets propagated to the data plane envoy.

In this example we demonstrate this. For this example, the gateway is run without the --net=host switch to use docker bridge network.

Note that Enroute uses Envoy proxy and has first class support for gRPC. The same canary deployments can also be achieved for gRPC services. For more information on gRPC load balancing, check out the gRPC article

Setup

In the next few steps, we show how easy it is to configure canary release a new version of software.

At a high level, we’ll perform the following steps -

  • Run the Enroute gateway (enroute-gw)
  • Create config on enroute-cp: Create Proxy gw, Service l, Route r, an Upstream u-stable
  • Create config on enroute-cp: Create another Upstream u-new
  • Wire up the objects: gw -> l -> r -> u-stable
  • Wire up the objects: r -> u-new
  • In the Enroute GW package, an instance of enroute-dp is already running with name gw
  • The above steps will configure the Envoy proxy with configuration provided on the Enroute control plane
  • Adjust traffic split between u-stable and u-new dynamically

Start the enroute gateway - enroute-gw

The controller is packaged as a docker image that can be run using the following command -

sudo docker run 					\
    -v db_data:/var/lib/postgresql/11/main 		\
    -p 80:8080 -p 1323:1323                             \
    saarasio/enroute-gw:latest

Create objects on controller using API

We use the Enroute API to perform the following tasks -

Create Proxy
$ curl -s -X POST "http://localhost:1323/proxy" -d 'Name=gw' | jq
{
    "name": "gw"
}


$ curl -s localhost:1323/proxy | jq
{
  "data": {
    "saaras_db_proxy": [
      {
        "proxy_id": 1,
        "proxy_name": "gw",
        "create_ts": "2019-11-16T19:36:54.601048+00:00",
        "update_ts": "2019-11-16T19:36:54.601048+00:00"
      }
    ]
  }
}
Create Service
$ curl -s -X POST "http://127.0.0.1:1323/service"    	\
    -d 'Service_Name=l'                                 \
    -d 'fqdn=127.0.0.1' | jq
{
    "data": {
        "insert_saaras_db_service": {
            "affected_rows": 1
        }
    }
}
Create Route
$ curl -s -X POST "http://127.0.0.1:1323/service/l/route"	\
    -d 'Route_Name=r'                         			\
    -d 'Route_prefix=/' | jq
{
    "data": {
        "insert_saaras_db_route": {
            "affected_rows": 2
        }
    }
}
Create Upstream u-stable
$ curl -s -X POST "http://127.0.0.1:1323/upstream"     	                \
    -d 'Upstream_name=u-stable'                       			\
    -d 'Upstream_ip=172.31.42.128'                     			\
    -d 'Upstream_port=8080'                     			\
    -d 'Upstream_hc_path=/'                     			\
    -d 'Upstream_weight=100' | jq
{
    "data": {
        "insert_saaras_db_upstream": {
            "affected_rows": 1
        }
    }
}

Note the upstream points to an IP address outside the enroute-gw docker container. Next we create the upstream u-new

$ curl -s -X POST "http://127.0.0.1:1323/upstream"     	                \
    -d 'Upstream_name=u-new'                         			\
    -d 'Upstream_ip=172.31.4.81'                     			\
    -d 'Upstream_port=8080'                     			\
    -d 'Upstream_hc_path=/'                     			\
    -d 'Upstream_weight=0' | jq
{
    "data": {
        "insert_saaras_db_upstream": {
            "affected_rows": 1
        }
    }
}

Note that the new upstream u-new is setup with an Upstream_weight of 0.

Wire up objects

Associate a Route to an upstream : Create Route -> Upstream association
$ curl -s -X POST "http://127.0.0.1:1323/service/l/route/r/upstream/u-stable" | jq
{
    "data": {
        "insert_saaras_db_route_upstream": {
            "affected_rows": 4
        }
    }
}
$ curl -s -X POST "http://127.0.0.1:1323/service/l/route/r/upstream/u-new" | jq
{
    "data": {
        "insert_saaras_db_route_upstream": {
            "affected_rows": 4
        }
    }
}
Associate a Service to a Proxy : Create Proxy -> Service association
$ curl -s -X POST "http://127.0.0.1:1323/proxy/gw/service/l" | jq
{
    "data": {
        "insert_saaras_db_proxy_service": {
            "affected_rows": 3
        }
    }
}
Dump proxy ‘gw’ config
$ curl -s 127.0.0.1:1323/proxy/dump/gw | jq
{
  "data": {
    "saaras_db_proxy": [
      {
        "proxy_id": 1,
        "proxy_name": "gw",
        "create_ts": "2019-11-16T23:38:43.72733+00:00",
        "update_ts": "2019-11-16T23:47:41.104007+00:00",
        "proxy_services": [
          {
            "service": {
              "service_id": 1,
              "service_name": "l",
              "fqdn": "127.0.0.1",
              "create_ts": "2019-11-16T23:39:01.325954+00:00",
              "update_ts": "2019-11-16T23:47:41.104007+00:00",
              "service_secrets": [],
              "routes": [
                {
                  "route_id": 1,
                  "route_name": "r",
                  "route_prefix": "/",
                  "create_ts": "2019-11-16T23:39:10.420272+00:00",
                  "update_ts": "2019-11-16T23:47:33.887194+00:00",
                  "route_upstreams": [
                    {
                      "upstream": {
                        "upstream_id": 1,
                        "upstream_name": "u-stable",
                        "upstream_ip": "172.31.42.128",
                        "upstream_port": 8080,
                        "upstream_hc_path": "/",
                        "upstream_hc_host": null,
                        "upstream_hc_intervalseconds": null,
                        "upstream_hc_timeoutseconds": null,
                        "upstream_hc_unhealthythresholdcount": null,
                        "upstream_hc_healthythresholdcount": null,
                        "upstream_strategy": null,
                        "upstream_validation_cacertificate": null,
                        "upstream_validation_subjectname": null,
                        "upstream_weight": 100,
                        "create_ts": "2019-11-16T23:46:40.122386+00:00",
                        "update_ts": "2019-11-16T23:47:23.369567+00:00"
                      }
                    },
                    {
                      "upstream": {
                        "upstream_id": 2,
                        "upstream_name": "u-new",
                        "upstream_ip": "172.31.4.81",
                        "upstream_port": 8080,
                        "upstream_hc_path": "/",
                        "upstream_hc_host": null,
                        "upstream_hc_intervalseconds": null,
                        "upstream_hc_timeoutseconds": null,
                        "upstream_hc_unhealthythresholdcount": null,
                        "upstream_hc_healthythresholdcount": null,
                        "upstream_strategy": null,
                        "upstream_validation_cacertificate": null,
                        "upstream_validation_subjectname": null,
                        "upstream_weight": 0,
                        "create_ts": "2019-11-16T23:47:10.649301+00:00",
                        "update_ts": "2019-11-16T23:47:33.887194+00:00"
                      }
                    }
                  ]
                }
              ]
            }
          }
        ]
      }
    ]
  }
}

Gradually send traffic to the new service u-new

Generate some traffic
while true; do curl 127.0.0.1; done
Gradually send some traffic to u-new
curl -s -X PATCH "http://127.0.0.1:1323/upstream/u-new" -d 'Upstream_weight=5' | jq
Check traffic on upstreams

Open a shell to the docker container -

$ docker ps
CONTAINER ID        IMAGE                        NAMES
75c70e6c1bb3        saarasio/enroute-gw:latest   cocky_moser

$ docker exec -it cocky_moser /bin/bash

Query the envoy admin interface for cluster stats -

curl -s localhost:9001/clusters | grep u- | grep rq_success
gw/u-stable/8080/42099b4af0::172.31.42.128:8080::rq_success::114
gw/u-new/8080/42099b4af0::172.31.4.81:8080::rq_success::5

Note above how the u-new service is getting a small fraction of traffic.