Run WASM at Ingress to Validate, Verify and Transform

Saaras Inc. April 26, 2022

Introduction

When developer get tools they like, they tend to be more productive. An extensibility in a language of choice, provides the convenience of extensibility without learning a new language. WebAssembly runtime executes bytecode, generated by compiling code written in high level languages - javascript, C, C++, Python, Rust or Golang.

We explore WebAssembly support in EnRoute gateway, and how we can load bytecode generated from custom logic written in a language of choice. into the WebAssembly runtime using EnRoute.

We start with a goal to verify a request, validate it’s schema and transform the request. We use custom code that is compiled and loaded into the WebAssembly runtime, to achieve this

WASM support is a part of EnRoute Community Edition which is free to use without any enterprise license.

What this article covers

The following steps walk through what’s needed to work with a request

Install EnRoute with WASM support
Install example workload
Program EnRoute to make the service externally available
Create a WebAssembly filter
Enable WebAssembly filter for service
Ensure request MAC is validated and transform request to redact sensitive information
Ensure JSON schema is validated

Next we go through the above mentioned steps in detail. We also look at how to build a WebAssembly image in compat variant which is then loaded into the Envoy WebAssembly runtime by EnRoute

Install EnRoute with WASM support

WASM support is provided in a separate EnRoute image. It can be installed using the following helm command after adding the helm repo and installing the enroute image with wasm tag

helm repo add saaras https://getenroute.io
helm install enroute-demo saaras/enroute \
	--set serviceAccount.create=true \
	--create-namespace \
	--namespace enroutedemo \
	--set images.enrouteService.tag=wasm

Install example workload

kubectl create namespace httpbin
kubectl apply -f https://raw.githubusercontent.com/saarasio/enroute/master/helm-chart/httpbin-bin-service.yaml

Program EnRoute to make the service externally available

helm install httpbin-service-policy saaras/service-policy --set service.name=httpbin --set service.prefix=/post --set service.port=80 --namespace httpbin

Send request to External-IP of LoadBalancer service to verify it works as expected. The getting started guide has details on setting this up.

curl -X POST -H 'Content-Type: application/json' -d '{}' 212.2.241.255/post

{
  "args": {},
  "data": "{}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "2",
    "Content-Type": "application/json",
    "Host": "212.2.241.255",
    "User-Agent": "curl/7.68.0",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-External-Address": "192.168.1.5"
  },
  "json": {},
  "origin": "152.70.114.65,192.168.1.5",
  "url": "http://212.2.241.255/post"
}

Create a WebAssembly filter

cat <<EOF | kubectl apply -f -
apiVersion: enroute.saaras.io/v1
kind: HttpFilter
metadata:
  name: httpbin-wasm-wasmfilter-oci
  namespace: httpbin
spec:
  httpFilterConfig:
    config: |
      {
              "url" : "oci://saarasio/vvx-json"
      }
  name: httpbin-wasm-wasmfilter-oci
  type: http_filter_wasm
EOF

Enable the WebAssembly filter for service

kubectl edit -n httpbin gatewayhosts.enroute.saaras.io httpbin-80-gatewayhost

  5 apiVersion: enroute.saaras.io/v1
  6 kind: GatewayHost
  7 metadata:
  8   annotations:
  9     meta.helm.sh/release-name: httpbin-service-policy
 10     meta.helm.sh/release-namespace: httpbin
 11   creationTimestamp: "2022-04-26T14:56:48Z"
 12   generation: 3
 13   labels:
 14     app: httpbin
 15     app.kubernetes.io/managed-by: Helm
 16   name: httpbin-80-gatewayhost
 17   namespace: httpbin
 18   resourceVersion: "3736"
 19   uid: b2e39054-a959-40c3-9754-365b518b8ee6
 20 spec:
 21   routes:
 22   - conditions:
 23     - prefix: /post
 24     filters:
 25     - name: httpbin-80-rl2
 26       type: route_filter_ratelimit
 27     services:
 28     - healthCheck:
 29         healthyThresholdCount: 3
 30         host: hc
 31         intervalSeconds: 5
 32         path: /
 33         timeoutSeconds: 3
 34         unhealthyThresholdCount: 3
 35       name: httpbin
 36       port: 80
 37   virtualhost:
 38     filters:
 39     - name: httpbin-80-luatestfilter
 40       type: http_filter_lua
 41     - name: httpbin-wasm-wasmfilter-oci
 42       type: http_filter_wasm
 43     fqdn: '*'

Ensure MAC is verified and request is transformed with personal information redacted

We first create a payload JSON that we send to the server using a POST request. Here is the json payload in a file called payload.json

{
  "data": [{
    "type": "articles",
    "id": "1",
    "attributes": {
      "title": "JSON:API paints my bikeshed!",
      "body": "The shortest article. Ever.",
      "created": "2006-01-02 15:04",
      "updated": "2011-01-19 22:15",
      "dob": "2022-01-19 22:15"
    },
    "relationships": {
      "author": {
        "data": {"id": "42", "type": "people"}
      }
    }
  }],
  "included": [
    {
      "type": "people",
      "id": "42",
      "attributes": {
        "name": "John",
        "age": 80,
        "gender": "male"
      }
    }
  ]
}

For verification, we calculate the SHA256 of the payload and compare it to the value we compute and send in the Digest header

cat payload.json | tr -d "\n" | openssl dgst -sha256 -binary | base64

h3wEb7jrjmwD8O7TDvOD3WGG23lfnzsfmODcbwLmdlk=

To send a request

curl -vv 212.2.241.255/post -H 'Content-Type: application/json' -H "Digest: sha256=h3wEb7jrjmwD8O7TDvOD3WGG23lfnzsfmODcbwLmdlk="  --data @/home/ubuntu/payload.json

{
  "args": {},
  "data": "{  \"data\": [{    \"type\": \"articles\",    \"id\": \"1\",    \"attributes\": {      \"title\": \"JSON:API paints my bikeshed!\",      \"body\": \"The shortest article. Ever.\",      \"created\": \"2006-01-02 15:04\",      \"updated\": \"2011-01-19 22:15\",      \"dob\": \"0000-00-00 00:00\"    },    \"relationships\": {      \"author\": {        \"data\": {\"id\": \"42\", \"type\": \"people\"}      }    }  }],  \"included\": [    {      \"type\": \"people\",      \"id\": \"42\",      \"attributes\": {        \"name\": \"John\",        \"age\": 80,        \"gender\": \"male\"      }    }  ]}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Content-Length": "532",
    "Content-Type": "application/json",
    "Digest": "sha256=h3wEb7jrjmwD8O7TDvOD3WGG23lfnzsfmODcbwLmdlk=",
    "Host": "212.2.241.255",
    "User-Agent": "curl/7.68.0",
    "X-Envoy-Expected-Rq-Timeout-Ms": "15000",
    "X-Envoy-External-Address": "192.168.1.5"
  },
  "json": {
    "data": [
      {
        "attributes": {
          "body": "The shortest article. Ever.",
          "created": "2006-01-02 15:04",
          "dob": "0000-00-00 00:00",
          "title": "JSON:API paints my bikeshed!",
          "updated": "2011-01-19 22:15"
        },
        "id": "1",
        "relationships": {
          "author": {
            "data": {
              "id": "42",
              "type": "people"
            }
          }
        },
        "type": "articles"
      }
    ],
    "included": [
      {
        "attributes": {
          "age": 80,
          "gender": "male",
          "name": "John"
        },
        "id": "42",
        "type": "people"
      }
    ]
  },
  "origin": "152.70.114.65,192.168.1.5",
  "url": "http://212.2.241.255/post"
}

Ensure JSON body schema is validated

The JSON validation checks for presence of a few JSON elements in the payload and the format of certain fields. In the above payload, it checks for type, id, title, body and verifies the format of created and updated files to ensure valid dates are provided.

Watch The Demo

Code

Here is the code and it’s flow to achieve validate, verify, transform functions. The code can also be found on the saarasio github repo

Accumulate Request body and call validate, verify and transform functions

220 func (ctx *validateVerifyTransformContext) OnHttpRequestBody(bodySize int, endOfStream bool) types.Action {
221         proxywasm.LogInfo("OnHttpRequestBody from Go!")
222         ctx.totalRequestBodySize += bodySize
223         if !endOfStream {
224                 // Wait until we see the entire body to replace.
225                 return types.ActionPause
226         }
227
228         jsonBody, err := proxywasm.GetHttpRequestBody(0, ctx.totalRequestBodySize)
229         if err != nil {
230                 proxywasm.LogErrorf("failed to get request body: %v", err)
231                 return types.ActionContinue
232         }
233
234         if ctx.digestValueRead {
235                 secretKey := ""
236                 vererr, received, computed := Verify(jsonBody, ctx.digestValue, secretKey)
237
238                 proxywasm.LogInfof("WASM:OnHttpRequestBody() Received Digest: [%v] Computed Digest [%v]", received, computed)
239
240                 if vererr != nil {
241                         if err := proxywasm.SendHttpResponse(400, nil, []byte(vererr.Error()), -1); err != nil {
242                                 panic(err)
243                         }
244                         return types.ActionPause
245                 }
246
247                 if received != computed {
248                         if err := proxywasm.SendHttpResponse(400, nil, []byte("Received digest doesn't match computed digest\n"), -1); err != nil {
249                                 panic(err)
250                         }
251                         return types.ActionPause
252                 }
253         }
254
255         valerr := ValidateJSON(&jsonBody)
256
257         if valerr != nil {
258                 if err := proxywasm.SendHttpResponse(400, nil, []byte(valerr.Error()), -1); err != nil {
259                         panic(err)
260                 }
261                 return types.ActionPause
262
263         }
264
265         TransformJSON(&jsonBody)
266
267         return types.ActionContinue
268 }

Verify HMAC

103 //////// Verify Digest ////////
104
105 func Verify(bytesIn []byte, encodedHash string, secretKey string) (error, string, string) {
106         var valerr error
107
108         var payload string
109         var prefix string
110
111         if strings.HasPrefix(encodedHash, "sha1=") {
112                 payload = strings.TrimPrefix(encodedHash, "sha1=")
113                 prefix = "sha1"
114         } else if strings.HasPrefix(encodedHash, "sha256=") {
115                 payload = strings.TrimPrefix(encodedHash, "sha256=")
116                 prefix = "sha256"
117         } else {
118                 valerr = fmt.Errorf("Unknown prefix in hash: [%v]\n", encodedHash)
119                 return valerr, "", ""
120         }
121
122         messageMAC := payload
123
124         messageMACBuf2, _ := b64.StdEncoding.DecodeString(messageMAC)
125
126         var receivedHash, computedHash string
127
128         if prefix == "sha1" {
129                 sum := sha1.Sum(bytesIn)
130                 receivedHash = b64.StdEncoding.EncodeToString(messageMACBuf2[:])
131                 computedHash = b64.StdEncoding.EncodeToString(sum[:])
132         } else if prefix == "sha256" {
133                 sum := sha256.Sum256(bytesIn)
134                 receivedHash = b64.StdEncoding.EncodeToString(messageMACBuf2[:])
135                 computedHash = b64.StdEncoding.EncodeToString(sum[:])
136         } else {
137                 valerr = fmt.Errorf("Unknown prefix in hash: [%v]\n", encodedHash)
138         }
139
140         return valerr, receivedHash, computedHash
141 }

Validate JSON

 51 func ValidateJSON(jsonBody *[]byte) error {
 52
 53         // Verify elements are present
 54         paths := [][]string{
 55                 []string{"data", "[0]", "type"},
 56                 []string{"data", "[0]", "id"},
 57                 []string{"data", "[0]", "attributes", "title"},
 58                 []string{"data", "[0]", "attributes", "body"},
 59                 []string{"data", "[0]", "attributes", "created"},
 60                 []string{"data", "[0]", "attributes", "updated"},
 61         }
 62         match := 0
 63         var created, updated []byte
 64         jsonparser.EachKey(*jsonBody, func(idx int, value []byte, vt jsonparser.ValueType, err error) {
 65                 switch idx {
 66                 case 0: //
 67                         match += 1
 68                 case 1: //
 69                         match += 1
 70                 case 2: //
 71                         match += 1
 72                 case 3: //
 73                         match += 1
 74                 case 4: //
 75                         match += 1
 76                         created = value
 77                 case 5: //
 78                         match += 1
 79                         updated = value
 80                 }
 81         }, paths...)
 82
 83         proxywasm.LogInfof("JSON Schema Validation Success, Match Count [%d]\n", match)
 84
 85         var ret error
 86
 87         // Verify created and updated are in date format
 88         _, err := time.Parse(string(created), string(updated))
 89         if err != nil {
 90                 proxywasm.LogError("Failed to parse created, updated")
 91                 ret = fmt.Errorf("Failed to parse created, updated")
 92
 93         }
 94
 95         if match != 6 {
 96                 ret = fmt.Errorf("Failed to validate json schema")
 97         }
 98
 99         proxywasm.LogInfo("JSON Field Validation Success, Validated Created, Updated \n")
100         return ret
101 }

Transform JSON Body

143 //////// Transform JSON ////////
144
145 func TransformJSON(jsonBody *[]byte) {
146
147         paths := [][]string{
148                 []string{"data", "[0]", "attributes", "dob"},
149         }
150         jsonparser.EachKey(*jsonBody, func(idx int, value []byte, vt jsonparser.ValueType, err error) {
151                 switch idx {
152                 case 0: //
153                         *jsonBody, _ = jsonparser.Set(*jsonBody, []byte("\"0000-00-00 00:00\""), "data", "[0]", "attributes", "dob")
154                         proxywasm.ReplaceHttpRequestBody(*jsonBody)
155                 }
156         }, paths...)
157
158 }

Conclusion

WebAssembly provides a flexible extension mechanism to customize request verification, validation and transformation. WebAssembly in a proxy implements the proxy-wasm abi spec, that is implemented by SDKs. The SDKs in different languages provide a mechanism to interface with WebAssembly runtime. We use the go sdk that implements the proxy-wasm abi spec to achieve this.

The code generated in the above step is compiled to WASM using the WASI libc when compiling using tinygo The bytecode generated after the compilation is packaged into a WASM artifact in compat variant that is then loaded into the Envoy WebAssembly runtime.

In the next article we cover how to build a compat variant of an image that can be loaded by EnRoute and details on proxy-wasm ABI that let’s us recieve callbacks for request processing.

Run WASM at Ingress to Validate, Verify and Transform