Extensibility

• 1: Build a Wasm image
• 2: Envoy Patch Policy
• 3: Envoy Gateway Extension Server
• 4: External Processing
• 5: Lua Extensions
• 6: Wasm Extensions

1 - Build a Wasm image

Envoy Gateway supports two types of Wasm extensions within the EnvoyExtensionPolicy API: HTTP Wasm Extensions and Image Wasm Extensions. Packaging a Wasm extension as an OCI image is beneficial because it simplifies versioning and distribution for users. Additionally, users can leverage existing image toolchain to build and manage Wasm images.

This document describes how to build OCI images which are consumable by Envoy Gateway.

Wasm Image Formats

There are two types of images that are supported by Envoy Gateway. One is in the Docker format, and another is the standard OCI specification compliant format. Please note that both of them are supported by any OCI registries. You can choose either format depending on your preference, and both types of images are consumable by Envoy Gateway EnvoyExtensionPolicy API.

Build Wasm Docker image

We assume that you have a valid Wasm binary named plugin.wasm. Then you can build a Wasm Docker image with the Docker CLI.

1. First, we prepare the following Dockerfile:

$ cat Dockerfile
FROM scratch

COPY plugin.wasm ./

Note: you must have exactly one COPY instruction in the Dockerfile in order to end up having only one layer in produced images.

2. Then, build your image via docker build command

$ docker build . -t my-registry/mywasm:0.1.0

3. Finally, push the image to your registry via docker push command

$ docker push my-registry/mywasm:0.1.0

Build Wasm OCI image

We assume that you have a valid Wasm binary named plugin.wasm, and you have buildah installed on your machine. Then you can build a Wasm OCI image with the buildah CLI.

1. First, we create a working container from scratch base image with buildah from command.

$ buildah --name mywasm from scratch
mywasm

2. Then copy the Wasm binary into that base image by buildah copy command to create the layer.

$ buildah copy mywasm plugin.wasm ./
af82a227630327c24026d7c6d3057c3d5478b14426b74c547df011ca5f23d271

Note: you must execute buildah copy exactly once in order to end up having only one layer in produced images

4. Now, you can build an OCI image and push it to your registry via buildah commit command

$ buildah commit mywasm docker://my-remote-registry/mywasm:0.1.0

2 - Envoy Patch Policy

This task explains the usage of the EnvoyPatchPolicy API. Note: This API is meant for users extremely familiar with Envoy xDS semantics. Also before considering this API for production use cases, please be aware that this API is unstable and the outcome may change across versions. Use at your own risk.

Introduction

The EnvoyPatchPolicy API allows user to modify the output xDS configuration generated by Envoy Gateway intended for EnvoyProxy, using JSON Patch semantics.

Motivation

This API was introduced to allow advanced users to be able to leverage Envoy Proxy functionality not exposed by Envoy Gateway APIs today.

Quickstart

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Enable EnvoyPatchPolicy

• By default EnvoyPatchPolicy is disabled. Lets enable it in the EnvoyGateway startup configuration

• The default installation of Envoy Gateway installs a default EnvoyGateway configuration and attaches it using a ConfigMap. In the next step, we will update this resource to enable EnvoyPatchPolicy.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true
EOF

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true

After updating the ConfigMap, you will need to wait the configuration kicks in.
You can force the configuration to be reloaded by restarting the envoy-gateway deployment.

kubectl rollout restart deployment envoy-gateway -n envoy-gateway-system

XDS Name Scheme V2

Starting from v1.5, Envoy Gateway uses version 2 of the xDS name scheme when generating xDS resources. Because EnvoyPatchPolicy relies on specific xDS resource names, it’s important to use the correct naming format when authoring a patch policy.

This change is gated by the XDSNameSchemeV2 runtime flag. The flag is disabled by default in v1.5 and will be enabled by default starting in v1.6.

We recommend users begin migrating their EnvoyPatchPolicy resources to use the version 2 naming scheme before upgrading to v1.6.

To opt in to the new naming scheme early, add theXDSNameSchemeV2 runtime flag to the runtimeFlags.enabled field in your EnvoyGateway configuration.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true
    runtimeFlags:
      enabled:
      - XDSNameSchemeV2
EOF

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true
    runtimeFlags:
      enabled:
      - XDSNameSchemeV2

Testing

Customize Response

• Use EnvoyProxy’s Local Reply Modification feature to return a custom response back to the client when the status code is 404

• Apply the configuration

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"

When mergeGateways is enabled, there will be one Envoy deployment for all Gateways in the cluster. Then the EnvoyPatchPolicy should target a specific GatewayClass.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: GatewayClass
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: GatewayClass
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"

• Edit the HTTPRoute resource from the Quickstart to only match on paths with value /get

kubectl patch httproute backend --type=json --patch '
  - op: add
    path: /spec/rules/0/matches/0/path/value
    value: /get
  '

• Test it out by specifying a path apart from /get

$ curl --header "Host: www.example.com" http://$GATEWAY_HOST/find
Handling connection for 8888
could not find what you are looking for

Customize VirtualHost by name

• Use EnvoyProxy’s include_attempt_count_in_response feature to include the attempt count as header in the downstream response.
• Apply the configuration

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: include-attempts
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.route.v3.RouteConfiguration"
      # The RouteConfiguration name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        # Every virtual_host that ends with 'www_example_com' (using RegEx Filter)
        jsonPath: "..virtual_hosts[?match(@.name, '.*www_example_com')]"
        # If the property does not exists, it can not be selected with jsonPath
        # Therefore the new property must be set in path
        path: "include_attempt_count_in_response"
        value: true
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: include-attempts
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.route.v3.RouteConfiguration"
      # The RouteConfiguration name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        # Every virtual_host that ends with 'www_example_com' (using RegEx Filter)
        jsonPath: "..virtual_hosts[?match(@.name, '.*www_example_com')]"
        # If the property does not exists, it can not be selected with jsonPath
        # Therefore the new property must be set in path
        path: "include_attempt_count_in_response"
        value: true

• Test it out by looking at the response headers

$ curl -v --header "Host: www.example.com" http://localhost:8888/
...
< x-envoy-attempt-count: 1
...

Debugging

Runtime

• The Status subresource should have information about the status of the resource. Make sure Accepted=True and Programmed=True conditions are set to ensure that the policy has been applied to Envoy Proxy.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.envoyproxy.io/v1alpha1","kind":"EnvoyPatchPolicy","metadata":{"annotations":{},"name":"custom-response-patch-policy","namespace":"default"},"spec":{"jsonPatches":[{"name":"default/eg/http","operation":{"op":"add","path":"/default_filter_chain/filters/0/typed_config/local_reply_config","value":{"mappers":[{"body":{"inline_string":"could not find what you are looking for"},"filter":{"status_code_filter":{"comparison":{"op":"EQ","value":{"default_value":404}}}}}]}},"type":"type.googleapis.com/envoy.config.listener.v3.Listener"}],"priority":0,"targetRef":{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"eg","namespace":"default"},"type":"JSONPatch"}}
  creationTimestamp: "2023-07-31T21:47:53Z"
  generation: 1
  name: custom-response-patch-policy
  namespace: default
  resourceVersion: "10265"
  uid: a35bda6e-a0cc-46d7-a63a-cee765174bc3
spec:
  jsonPatches:
  - name: default/eg/http
    operation:
      op: add
      path: /default_filter_chain/filters/0/typed_config/local_reply_config
      value:
        mappers:
        - body:
            inline_string: could not find what you are looking for
          filter:
            status_code_filter:
              comparison:
                op: EQ
                value:
                  default_value: 404
    type: type.googleapis.com/envoy.config.listener.v3.Listener
  priority: 0
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
status:
  conditions:
  - lastTransitionTime: "2023-07-31T21:48:19Z"
    message: EnvoyPatchPolicy has been accepted.
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2023-07-31T21:48:19Z"
    message: successfully applied patches.
    reason: Programmed
    status: "True"
    type: Programmed

Offline

• You can use egctl x translate to validate the translated xds output.

Caveats

This API will always be an unstable API and the same outcome cannot be guaranteed across versions for these reasons

• The Envoy Proxy API might deprecate and remove API fields
• Envoy Gateway might alter the xDS translation creating a different xDS output such as changing the name field of resources.

3 - Envoy Gateway Extension Server

This task explains how to extend Envoy Gateway using an Extension Server. Envoy Gateway can be configured to call an external server over gRPC with the xDS configuration before it is sent to Envoy Proxy. The external server can modify the provided configuration programmatically using any semantics supported by the xDS API.

Using an extension server allows vendors to add xDS configuration that Envoy Gateway itself doesn’t support with a very high level of control over the generated xDS configuration.

Note: Modifying the xDS configuration generated by Envoy Gateway may break functionality configured by native Envoy Gateway means. Like other cases where the xDS configuration is modified outside of Envoy Gateway’s control, this is risky and should be tested thoroughly, especially when using the same extension server across different Envoy Gateway versions.

Introduction

One of the Envoy Gateway project goals is to “provide a common foundation for vendors to build value-added products without having to re-engineer fundamental interactions”. The Envoy Gateway Extension Server provides a mechanism where Envoy Gateway tracks all provider resources and then calls a set of hooks that allow the generated xDS configuration to be modified before it is sent to Envoy Proxy. See the design documentation for full details.

Extension Hooks Overview

Envoy Gateway provides several extension hooks that are called at different stages of the xDS translation process. These hooks allow extensions to modify various aspects of the generated xDS configuration:

Available Hooks

Hook Execution Order

The hooks are executed in the following order during xDS translation:

1. Route Processing Phase

   • Route hook (PostRouteModifyHook): Called for each route that has extension filters attached
   • Cluster hook (PostClusterModifyHook): Called for each cluster generated from custom backend references

2. Virtual Host Processing Phase

   • VirtualHost hook (PostVirtualHostModifyHook): Called for each virtual host after all routes are processed

3. Listener Processing Phase

   • HTTPListener hook (PostHTTPListenerModifyHook): Called for each HTTP listener after virtual hosts are configured

4. Final Translation Phase

   • Translation hook (PostTranslateModifyHook): Called once with all generated clusters, secrets, listeners, and routes

Hook Details

Route Hook (PostRouteModifyHook)

• When called: After each individual route is generated from an HTTPRoute with extension filters
• Input: Single route, hostnames, and extension resources from the HTTPRoute
• Output: Modified route
• Use cases: Add route-specific filters, modify route configuration, add typed per-filter config

Cluster Hook (PostClusterModifyHook)

• When called: During cluster generation for custom backend references only
• Input: Single cluster and extension resources from the backend reference
• Output: Modified cluster
• Use cases: Configure custom backend clusters, add health checks, modify load balancing

VirtualHost Hook (PostVirtualHostModifyHook)

• When called: After all routes for a virtual host are processed
• Input: Complete virtual host with all its routes
• Output: Modified virtual host (can add/remove/modify routes)
• Use cases: Add virtual host-level configuration, inject additional routes

HTTPListener Hook (PostHTTPListenerModifyHook)

• When called: After listener is fully configured with all virtual hosts
• Input: Complete HTTP listener and associated extension policies
• Output: Modified listener
• Use cases: Add listener filters, modify listener configuration, add authentication

Translation Hook (PostTranslateModifyHook)

• When called: After all individual resources are generated and processed
• Input: All clusters, secrets, listeners, and routes, plus extension policies
• Output: Complete set of modified resources
• Use cases: Global resource injection, cross-resource modifications, cleanup operations

Configuration

Extensions must specify which hooks they want to use in the extensionManager.hooks configuration:

extensionManager:
  hooks:
    xdsTranslator:
      post:
        - Route          # Enable route modification hook
        - VirtualHost    # Enable virtual host modification hook
        - HTTPListener   # Enable HTTP listener modification hook
        - Cluster        # Enable cluster modification hook
        - Translation    # Enable global translation hook
      # Configure which resources to include in PostTranslateModifyHook
      # Default: true for clusters and secrets (for backward compatibility)
      # Default: false for listeners and routes (for backward compatibility)
      translation:
        listener:
          includeAll: true
        route:
          includeAll: true

This task sets up an example extension server that adds the Envoy Proxy Basic Authentication HTTP filter to all the listeners generated by Envoy Gateway. The example extension server includes its own CRD which allows defining username/password pairs that will be accepted by the Envoy Proxy.

Note: Envoy Gateway supports adding Basic Authentication to routes using a SecurityPolicy. See this task for the preferred way to configure Basic Authentication.

Quickstart

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Build and run the example Extension Server

Build and deploy the example extension server in the examples/extension-server folder into the cluster running Envoy Gateway.

• Build the extension server image

  Note: The provided Makefile builds an image with the name extension-server:latest. You may need to create a different tag for it in order to allow Kubernetes to pull it correctly.

  make image
  

• Publish the extension server image in your docker repository

  • local kind server
  • other Kubernetes server

  kind load docker-image --name envoy-gateway extension-server:latest
  

  docker tag extension-server:latest $YOUR_DOCKER_REPO
  docker push $YOUR_DOCKER_REPO
  

• Deploy the extension server in your cluster

  If you are using your own docker image repository, make sure to update the values.yaml with the correct image name and tag.

  helm install -n envoy-gateway-system extension-server ./examples/extension-server/charts/extension-server
  

Configure Envoy Gateway

• Grant Envoy Gateway’s ServiceAccount permission to access the extension server’s CRD

  kubectl create clusterrole listener-context-example-status-update \
             --verb=update  \
             --resource=ListenerContextExample/status
  
  kubectl create clusterrole listener-context-example-viewer \
             --verb=get,list,watch  \
             --resource=ListenerContextExample
  
  kubectl create clusterrolebinding envoy-gateway-listener-context \
             --clusterrole=listener-context-example-viewer \
             --serviceaccount=envoy-gateway-system:envoy-gateway
  
  kubectl create clusterrolebinding envoy-gateway-listener-context-status \
             --clusterrole=listener-context-example-status-update \
             --serviceaccount=envoy-gateway-system:envoy-gateway
  

• Configure Envoy Gateway to use the Extension Server

  Add the following fragment to Envoy Gateway’s configmap:

  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: envoy-gateway-config
    namespace: envoy-gateway-system
  data:
    envoy-gateway.yaml: |
      apiVersion: gateway.envoyproxy.io/v1alpha1
      kind: EnvoyGateway
      provider:
        type: Kubernetes
      gateway:
        controllerName: gateway.envoyproxy.io/gatewayclass-controller
      extensionManager:
        # Envoy Gateway will watch these resource kinds and use them as extension policies
        # which can be attached to Gateway resources.
        policyResources:
        - group: example.extensions.io
          version: v1alpha1
          kind: ListenerContextExample
        hooks:
          # The type of hooks that should be invoked
          xdsTranslator:
            post:
            - HTTPListener
        service:
          # The service that is hosting the extension server
          fqdn:
            hostname: extension-server.envoy-gateway-system.svc.cluster.local
            port: 5005
  EOF
  

  After updating Envoy Gateway’s configmap, restart Envoy Gateway.

Testing

Get the Gateway’s address:

export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')

The extension server adds the Basic Authentication HTTP filter to all listeners configured by Envoy Gateway. Initially there are no valid user/password combinations available. Accessing the example backend should fail with a 401 status:

$ curl -v --header "Host: www.example.com" "http://${GATEWAY_HOST}/example"
...
> GET /example HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< www-authenticate: Basic realm="http://www.example.com/example"
< content-length: 58
< content-type: text/plain
< date: Mon, 08 Jul 2024 10:53:11 GMT
< 
...
User authentication failed. Missing username and password.
...

Add a new Username/Password combination using the example extension server’s CRD:

kubectl apply -f - << EOF 
apiVersion: example.extensions.io/v1alpha1
kind: ListenerContextExample
metadata:
  name: listeneruser
spec:
  targetRefs:
  - kind: Gateway
    name: eg
    group: gateway.networking.k8s.io
  username: user
  password: p@ssw0rd
EOF

Authenticating with this user/password combination will now work.

$ curl -v http://${GATEWAY_HOST}/example  -H "Host: www.example.com"   --user 'user:p@ssw0rd'
...
> GET /example HTTP/1.1
> Host: www.example.com
> Authorization: Basic dXNlcm5hbWU6cEBzc3cwcmQ=
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Mon, 08 Jul 2024 10:56:17 GMT
< content-length: 559
< 
...
 "headers": {
  "Authorization": [
   "Basic dXNlcm5hbWU6cEBzc3cwcmQ="
  ],
  "X-Example-Ext": [
   "user"
  ],
...

4 - External Processing

This task provides instructions for configuring external processing.

External processing calls an external gRPC service to process HTTP requests and responses. The external processing service can inspect and mutate requests and responses.

Envoy Gateway introduces a new CRD called EnvoyExtensionPolicy that allows the user to configure external processing. This instantiated resource can be linked to a Gateway and HTTPRoute resource.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

GRPC External Processing Service

Installation

Install a demo GRPC service that will be used as the external processing service:

kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/ext-proc-grpc-service.yaml

Create a new HTTPRoute resource to route traffic on the path /myapp to the backend service.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp
spec:
  parentRefs:
  - name: eg
  hostnames:
  - "www.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /myapp
    backendRefs:
    - name: backend
      port: 3000   
EOF

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp
spec:
  parentRefs:
  - name: eg
  hostnames:
  - "www.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /myapp
    backendRefs:
    - name: backend
      port: 3000   

Verify the HTTPRoute status:

kubectl get httproute/myapp -o yaml

Configuration

Create a new EnvoyExtensionPolicy resource to configure the external processing service. This EnvoyExtensionPolicy targets the HTTPRoute “myApp” created in the previous step. It calls the GRPC external processing service “grpc-ext-proc” on port 9002 for processing.

By default, requests and responses are not sent to the external processor. The processingMode struct is used to define what should be sent to the external processor. In this example, we configure the following processing modes:

• The empty request field configures envoy to send request headers to the external processor.
• The response field includes configuration for body processing. As a result, response headers are sent to the external processor. Additionally, the response body is streamed to the external processor.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: ext-proc-example
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: myapp
  extProc:
  - backendRefs:
    - name: grpc-ext-proc
      port: 9002
    processingMode:
      request: {}
      response: 
        body: Streamed 
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: ext-proc-example
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: myapp
  extProc:
    - backendRefs:
        - name: grpc-ext-proc
          port: 9002
      processingMode:
        request: {}
        response: 
          body: Streamed

Verify the Envoy Extension Policy configuration:

kubectl get envoyextensionpolicy/ext-proc-example -o yaml

Because the gRPC external processing service is enabled with TLS, a BackendTLSPolicy needs to be created to configure the communication between the Envoy proxy and the gRPC auth service.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: grpc-ext-proc-btls
spec:
  targetRefs:
  - group: ''
    kind: Service
    name: grpc-ext-proc
  validation:
    caCertificateRefs:
    - name: grpc-ext-proc-ca
      group: ''
      kind: ConfigMap
    hostname: grpc-ext-proc.envoygateway
EOF

---
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: grpc-ext-proc-btls
spec:
  targetRefs:
    - group: ''
      kind: Service
      name: grpc-ext-proc
  validation:
    caCertificateRefs:
      - name: grpc-ext-proc-ca
        group: ''
        kind: ConfigMap
    hostname: grpc-ext-proc.envoygateway

Verify the BackendTLSPolicy configuration:

kubectl get backendtlspolicy/grpc-ext-proc-btls -o yaml

Testing

Ensure the GATEWAY_HOST environment variable from the Quickstart is set. If not, follow the Quickstart instructions to set the variable.

echo $GATEWAY_HOST

Send a request to the backend service without Authentication header:

curl -v -H "Host: www.example.com" "http://${GATEWAY_HOST}/myapp"

You should see that the external processor added headers:

• x-request-ext-processed - this header was added before the request was forwarded to the backend
• x-response-ext-processed- this header was added before the response was returned to the client

curl -v -H "Host: www.example.com"  http://localhost:10080/myapp
[...]
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Fri, 14 Jun 2024 19:30:40 GMT
< content-length: 502
< x-response-ext-processed: true
<
{
 "path": "/myapp",
 "host": "www.example.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
[...] 
  "X-Request-Ext-Processed": [
   "true"
  ],
[...]
 }

Clean-Up

Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.

Delete the demo auth services, HTTPRoute, EnvoyExtensionPolicy and BackendTLSPolicy:

kubectl delete -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/ext-proc-grpc-service.yaml
kubectl delete httproute/myapp
kubectl delete envoyextensionpolicy/ext-proc-example
kubectl delete backendtlspolicy/grpc-ext-proc-btls

Next Steps

Checkout the Developer Guide to get involved in the project.

5 - Lua Extensions

This task provides instructions for extending Envoy Gateway with Lua extensions.

Lua extensions allow you to extend the functionality of Envoy Gateway by running custom code against HTTP requests and responses, without modifying the Envoy Gateway binary. These comparatively light-weight extensions are written in the Lua scripting language using APIs defined here.

Envoy Gateway allows the user to configure Lua extensions using the EnvoyExtensionPolicy CRD. This instantiated resource can be linked to a Gateway or HTTPRoute resource. If linked to both, the resource linked to the route takes precedence over those linked to Gateway.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Configuration

Envoy Gateway supports Lua in EnvoyExtensionPolicy in two modes:

• Inline Lua: The extension defines the Lua script as an inline string.
• ValueRef Lua: The extension points to an in-cluster ConfigMap resource that contains the Lua script in it’s data.

The following example demonstrates how to configure an EnvoyExtensionPolicy to attach a Lua extension to a HTTPRoute. This Lua extension adds a custom header x-lua-custom: FOO to the response.

Lua Extension - Inline

This EnvoyExtensionPolicy configuration defines the Lua script as an inline string.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-inline-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  lua:
    - type: Inline
      inline: |
        function envoy_on_response(response_handle)
          response_handle:headers():add("x-lua-custom", "FOO")
        end
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-inline-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  lua:
    - type: Inline
      inline: |
        function envoy_on_response(response_handle)
          response_handle:headers():add("x-lua-custom", "FOO")
        end

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/lua-inline-test -o yaml

Lua Extension - ValueRef

This EnvoyExtensionPolicy configuration defines the Lua extension in a ConfigMap resource.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-valueref-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  lua:
    - type: ValueRef
      valueRef:
        name: cm-lua-valueref
        kind: ConfigMap
        group: v1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-lua-valueref
data:
  lua: |
    function envoy_on_response(response_handle)
      response_handle:headers():add("x-lua-custom", "FOO")
    end
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-valueref-test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: backend
  lua:
    - type: ValueRef
      valueRef:
        name: cm-lua-valueref
        kind: ConfigMap
        group: v1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-lua-valueref
data:
  lua: |
    function envoy_on_response(response_handle)
      response_handle:headers():add("x-lua-custom", "FOO")
    end

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/lua-valueref-test -o yaml

Testing

Ensure the GATEWAY_HOST environment variable from the Quickstart is set. If not, follow the Quickstart instructions to set the variable.

echo $GATEWAY_HOST

Send a request to the backend service:

curl -i -H "Host: www.example.com" "http://${GATEWAY_HOST}"

You should see that the lua extension has added this header to the response:

x-lua-custom: FOO

Clean-Up

Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.

Delete the EnvoyExtensionPolicy:

kubectl delete envoyextensionpolicy/lua-inline-test
kubectl delete envoyextensionpolicy/lua-valueref-test
kubectl delete configmap/cm-lua-valueref

Next Steps

Checkout the Developer Guide to get involved in the project.

6 - Wasm Extensions

This task provides instructions for extending Envoy Gateway with WebAssembly (Wasm) extensions.

Wasm extensions allow you to extend the functionality of Envoy Gateway by running custom code against HTTP requests and responses, without modifying the Envoy Gateway binary. These extensions can be written in any language that compiles to Wasm, such as C++, Rust, AssemblyScript, or TinyGo.

Envoy Gateway introduces a new CRD called EnvoyExtensionPolicy that allows the user to configure Wasm extensions. This instantiated resource can be linked to a Gateway and HTTPRoute resource.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Configuration

Envoy Gateway supports two types of Wasm extensions:

• HTTP Wasm Extension: The Wasm extension is fetched from a remote URL.
• Image Wasm Extension: The Wasm extension is packaged as an OCI image and fetched from an image registry.

The following example demonstrates how to configure an EnvoyExtensionPolicy to attach a Wasm extension to an EnvoyExtensionPolicy . This Wasm extension adds a custom header x-wasm-custom: FOO to the response.

HTTP Wasm Extension

This EnvoyExtensionPolicy configuration fetches the Wasm extension from an HTTP URL.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  wasm:
  - name: wasm-filter
    rootID: my_root_id
    code:
      type: HTTP
      http:
        url: https://raw.githubusercontent.com/envoyproxy/examples/main/wasm-cc/lib/envoy_filter_http_wasm_example.wasm
        sha256: 79c9f85128bb0177b6511afa85d587224efded376ac0ef76df56595f1e6315c0
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  wasm:
  - name: wasm-filter
    rootID: my_root_id
    code:
      type: HTTP
      http:
        url: https://raw.githubusercontent.com/envoyproxy/examples/main/wasm-cc/lib/envoy_filter_http_wasm_example.wasm
        sha256: 79c9f85128bb0177b6511afa85d587224efded376ac0ef76df56595f1e6315c0

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/http-wasm-source-test -o yaml

Image Wasm Extension

This EnvoyExtensionPolicy configuration fetches the Wasm extension from an OCI image.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  wasm:
  - name: wasm-filter
    rootID: my_root_id
    code:
      type: Image
      image:
        url: zhaohuabing/testwasm:v0.0.1
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  wasm:
  - name: wasm-filter
    rootID: my_root_id
    code:
      type: Image
      image:
        url: zhaohuabing/testwasm:v0.0.1

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/http-wasm-source-test -o yaml

Testing

Ensure the GATEWAY_HOST environment variable from the Quickstart is set. If not, follow the Quickstart instructions to set the variable.

echo $GATEWAY_HOST

Send a request to the backend service:

curl -i -H "Host: www.example.com" "http://${GATEWAY_HOST}"

You should see that the wasm extension has added this header to the response:

x-wasm-custom: FOO

Clean-Up

Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.

Delete the EnvoyExtensionPolicy:

kubectl delete envoyextensionpolicy/wasm-test

Next Steps

Checkout the Developer Guide to get involved in the project.