Extensibility

• 1: Build a Wasm image
• 2: Envoy Patch Policy
• 3: Envoy Gateway Extension Server
• 4: External Processing
• 5: Lua Extensions
• 6: OPA Sidecar with Unix Domain Socket
• 7: Wasm Extensions

1 - Build a Wasm image

Envoy Gateway supports two types of Wasm extensions within the EnvoyExtensionPolicy API: HTTP Wasm Extensions and Image Wasm Extensions. Packaging a Wasm extension as an OCI image is beneficial because it simplifies versioning and distribution for users. Additionally, users can leverage existing image toolchain to build and manage Wasm images.

This document describes how to build OCI images which are consumable by Envoy Gateway.

Wasm Image Formats

There are two types of images that are supported by Envoy Gateway. One is in the Docker format, and another is the standard OCI specification compliant format. Please note that both of them are supported by any OCI registries. You can choose either format depending on your preference, and both types of images are consumable by Envoy Gateway EnvoyExtensionPolicy API.

Build Wasm Docker image

We assume that you have a valid Wasm binary named plugin.wasm. Then you can build a Wasm Docker image with the Docker CLI.

1. First, we prepare the following Dockerfile:

$ cat Dockerfile
FROM scratch

COPY plugin.wasm ./

Note: you must have exactly one COPY instruction in the Dockerfile in order to end up having only one layer in produced images.

2. Then, build your image via docker build command

$ docker build . -t my-registry/mywasm:0.1.0

3. Finally, push the image to your registry via docker push command

$ docker push my-registry/mywasm:0.1.0

Build Wasm OCI image

We assume that you have a valid Wasm binary named plugin.wasm, and you have buildah installed on your machine. Then you can build a Wasm OCI image with the buildah CLI.

1. First, we create a working container from scratch base image with buildah from command.

$ buildah --name mywasm from scratch
mywasm

2. Then copy the Wasm binary into that base image by buildah copy command to create the layer.

$ buildah copy mywasm plugin.wasm ./
af82a227630327c24026d7c6d3057c3d5478b14426b74c547df011ca5f23d271

Note: you must execute buildah copy exactly once in order to end up having only one layer in produced images

4. Now, you can build an OCI image and push it to your registry via buildah commit command

$ buildah commit mywasm docker://my-remote-registry/mywasm:0.1.0

2 - Envoy Patch Policy

This task explains the usage of the EnvoyPatchPolicy API. Note: This API is meant for users extremely familiar with Envoy xDS semantics. Also before considering this API for production use cases, please be aware that this API is unstable and the outcome may change across versions. Use at your own risk.

Introduction

The EnvoyPatchPolicy API allows user to modify the output xDS configuration generated by Envoy Gateway intended for EnvoyProxy, using JSON Patch semantics.

Motivation

This API was introduced to allow advanced users to be able to leverage Envoy Proxy functionality not exposed by Envoy Gateway APIs today.

Quickstart

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Enable EnvoyPatchPolicy

• By default EnvoyPatchPolicy is disabled. Lets enable it in the EnvoyGateway startup configuration

• The default installation of Envoy Gateway installs a default EnvoyGateway configuration and attaches it using a ConfigMap. In the next step, we will update this resource to enable EnvoyPatchPolicy.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true
EOF

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true

After updating the ConfigMap, you will need to wait the configuration kicks in.
You can force the configuration to be reloaded by restarting the envoy-gateway deployment.

kubectl rollout restart deployment envoy-gateway -n envoy-gateway-system

XDS Name Scheme V2

Starting from v1.5, Envoy Gateway uses version 2 of the xDS name scheme when generating xDS resources. Because EnvoyPatchPolicy relies on specific xDS resource names, it’s important to use the correct naming format when authoring a patch policy.

This change is gated by the XDSNameSchemeV2 runtime flag. The flag is disabled by default in v1.5 and will be enabled by default starting in v1.10.

We recommend users begin migrating their EnvoyPatchPolicy resources to use the version 2 naming scheme before upgrading to v1.10.

To opt in to the new naming scheme early, add theXDSNameSchemeV2 runtime flag to the runtimeFlags.enabled field in your EnvoyGateway configuration.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true
    runtimeFlags:
      enabled:
      - XDSNameSchemeV2
EOF

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: envoy-gateway-config
  namespace: envoy-gateway-system
data:
  envoy-gateway.yaml: |
    apiVersion: gateway.envoyproxy.io/v1alpha1
    kind: EnvoyGateway
    provider:
      type: Kubernetes
    gateway:
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
    extensionApis:
      enableEnvoyPatchPolicy: true
    runtimeFlags:
      enabled:
      - XDSNameSchemeV2

Testing

Customize Response

• Use EnvoyProxy’s Local Reply Modification feature to return a custom response back to the client when the status code is 404

• Apply the configuration

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"

When mergeGateways is enabled, there will be one Envoy deployment for all Gateways in the cluster. Then the EnvoyPatchPolicy should target a specific GatewayClass.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: GatewayClass
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: custom-response-patch-policy
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: GatewayClass
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.listener.v3.Listener"
      # The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
        value:
          mappers:
          - filter:
              status_code_filter:
                comparison:
                  op: EQ
                  value:
                    default_value: 404
                    runtime_key: key_b
            status_code: 406
            body:
              inline_string: "could not find what you are looking for"

• Edit the HTTPRoute resource from the Quickstart to only match on paths with value /get

kubectl patch httproute backend --type=json --patch '
  - op: add
    path: /spec/rules/0/matches/0/path/value
    value: /get
  '

• Test it out by specifying a path apart from /get

$ curl --header "Host: www.example.com" http://$GATEWAY_HOST/find
Handling connection for 8888
could not find what you are looking for

Customize VirtualHost by name

• Use EnvoyProxy’s include_attempt_count_in_response feature to include the attempt count as header in the downstream response.
• Apply the configuration

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: include-attempts
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.route.v3.RouteConfiguration"
      # The RouteConfiguration name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        # Every virtual_host that ends with 'www_example_com' (using RegEx Filter)
        jsonPath: "..virtual_hosts[?match(@.name, '.*www_example_com')]"
        # If the property does not exists, it can not be selected with jsonPath
        # Therefore the new property must be set in path
        path: "include_attempt_count_in_response"
        value: true
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  name: include-attempts
  namespace: default
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
  jsonPatches:
    - type: "type.googleapis.com/envoy.config.route.v3.RouteConfiguration"
      # The RouteConfiguration name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
      name: default/eg/http
      operation:
        op: add
        # Every virtual_host that ends with 'www_example_com' (using RegEx Filter)
        jsonPath: "..virtual_hosts[?match(@.name, '.*www_example_com')]"
        # If the property does not exists, it can not be selected with jsonPath
        # Therefore the new property must be set in path
        path: "include_attempt_count_in_response"
        value: true

• Test it out by looking at the response headers

$ curl -v --header "Host: www.example.com" http://localhost:8888/
...
< x-envoy-attempt-count: 1
...

Debugging

Runtime

• The Status subresource should have information about the status of the resource. Make sure Accepted=True and Programmed=True conditions are set to ensure that the policy has been applied to Envoy Proxy.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.envoyproxy.io/v1alpha1","kind":"EnvoyPatchPolicy","metadata":{"annotations":{},"name":"custom-response-patch-policy","namespace":"default"},"spec":{"jsonPatches":[{"name":"default/eg/http","operation":{"op":"add","path":"/default_filter_chain/filters/0/typed_config/local_reply_config","value":{"mappers":[{"body":{"inline_string":"could not find what you are looking for"},"filter":{"status_code_filter":{"comparison":{"op":"EQ","value":{"default_value":404}}}}}]}},"type":"type.googleapis.com/envoy.config.listener.v3.Listener"}],"priority":0,"targetRef":{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"eg","namespace":"default"},"type":"JSONPatch"}}
  creationTimestamp: "2023-07-31T21:47:53Z"
  generation: 1
  name: custom-response-patch-policy
  namespace: default
  resourceVersion: "10265"
  uid: a35bda6e-a0cc-46d7-a63a-cee765174bc3
spec:
  jsonPatches:
  - name: default/eg/http
    operation:
      op: add
      path: /default_filter_chain/filters/0/typed_config/local_reply_config
      value:
        mappers:
        - body:
            inline_string: could not find what you are looking for
          filter:
            status_code_filter:
              comparison:
                op: EQ
                value:
                  default_value: 404
    type: type.googleapis.com/envoy.config.listener.v3.Listener
  priority: 0
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  type: JSONPatch
status:
  conditions:
  - lastTransitionTime: "2023-07-31T21:48:19Z"
    message: EnvoyPatchPolicy has been accepted.
    observedGeneration: 1
    reason: Accepted
    status: "True"
    type: Accepted
  - lastTransitionTime: "2023-07-31T21:48:19Z"
    message: successfully applied patches.
    reason: Programmed
    status: "True"
    type: Programmed

Offline

• You can use egctl x translate to validate the translated xds output.

Caveats

This API will always be an unstable API and the same outcome cannot be guaranteed across versions for these reasons

• The Envoy Proxy API might deprecate and remove API fields
• Envoy Gateway might alter the xDS translation creating a different xDS output such as changing the name field of resources.

3 - Envoy Gateway Extension Server

This task explains how to extend Envoy Gateway using an Extension Server. Envoy Gateway can be configured to call an external server over gRPC with the xDS configuration before it is sent to Envoy Proxy. The external server can modify the provided configuration programmatically using any semantics supported by the xDS API.

Using an extension server allows vendors to add xDS configuration that Envoy Gateway itself doesn’t support with a very high level of control over the generated xDS configuration.

Note: Modifying the xDS configuration generated by Envoy Gateway may break functionality configured by native Envoy Gateway means. Like other cases where the xDS configuration is modified outside of Envoy Gateway’s control, this is risky and should be tested thoroughly, especially when using the same extension server across different Envoy Gateway versions.

Introduction

One of the Envoy Gateway project goals is to “provide a common foundation for vendors to build value-added products without having to re-engineer fundamental interactions”. The Envoy Gateway Extension Server provides a mechanism where Envoy Gateway tracks all provider resources and then calls a set of hooks that allow the generated xDS configuration to be modified before it is sent to Envoy Proxy. See the design documentation for full details.

Extension Hooks Overview

Envoy Gateway provides several extension hooks that are called at different stages of the xDS translation process. These hooks allow extensions to modify various aspects of the generated xDS configuration:

Available Hooks

Hook Execution Order

The hooks are executed in the following order during xDS translation:

1. Route Processing Phase

   • Route hook (PostRouteModifyHook): Called for each route that has extension filters attached
   • Cluster hook (PostClusterModifyHook): Called for each cluster generated from custom backend references

2. Virtual Host Processing Phase

   • VirtualHost hook (PostVirtualHostModifyHook): Called for each virtual host after all routes are processed

3. Listener Processing Phase

   • HTTPListener hook (PostHTTPListenerModifyHook): Called for each HTTP listener after virtual hosts are configured

4. Final Translation Phase

   • Translation hook (PostTranslateModifyHook): Called once with all generated clusters, secrets, listeners, and routes

Hook Details

Route Hook (PostRouteModifyHook)

• When called: After each individual route is generated from an HTTPRoute with extension filters
• Input: Single route, hostnames, and extension resources from the HTTPRoute
• Output: Modified route
• Use cases: Add route-specific filters, modify route configuration, add typed per-filter config

Cluster Hook (PostClusterModifyHook)

• When called: During cluster generation for custom backend references only
• Input: Single cluster and extension resources from the backend reference
• Output: Modified cluster
• Use cases: Configure custom backend clusters, add health checks, modify load balancing

VirtualHost Hook (PostVirtualHostModifyHook)

• When called: After all routes for a virtual host are processed
• Input: Complete virtual host with all its routes
• Output: Modified virtual host (can add/remove/modify routes)
• Use cases: Add virtual host-level configuration, inject additional routes

HTTPListener Hook (PostHTTPListenerModifyHook)

• When called: After listener is fully configured with all virtual hosts
• Input: Complete HTTP listener and associated extension policies
• Output: Modified listener
• Use cases: Add listener filters, modify listener configuration, add authentication

Translation Hook (PostTranslateModifyHook)

• When called: After all individual resources are generated and processed
• Input: All clusters, secrets, listeners, and routes, plus extension policies
• Output: Complete set of modified resources
• Use cases: Global resource injection, cross-resource modifications, cleanup operations

Configuration

Extensions must specify which hooks they want to use in the extensionManager.hooks configuration:

extensionManager:
  hooks:
    xdsTranslator:
      post:
        - Route          # Enable route modification hook
        - VirtualHost    # Enable virtual host modification hook
        - HTTPListener   # Enable HTTP listener modification hook
        - Cluster        # Enable cluster modification hook
        - Translation    # Enable global translation hook
      # Configure which resources to include in PostTranslateModifyHook
      # Default: true for clusters and secrets (for backward compatibility)
      # Default: false for listeners and routes (for backward compatibility)
      translation:
        listener:
          includeAll: true
        route:
          includeAll: true

This task sets up an example extension server that adds the Envoy Proxy Basic Authentication HTTP filter to all the listeners generated by Envoy Gateway. The example extension server includes its own CRD which allows defining username/password pairs that will be accepted by the Envoy Proxy.

Note: Envoy Gateway supports adding Basic Authentication to routes using a SecurityPolicy. See this task for the preferred way to configure Basic Authentication.

Quickstart

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Build and run the example Extension Server

Build and deploy the example extension server in the examples/extension-server folder into the cluster running Envoy Gateway.

• Build the extension server image

  Note: The provided Makefile builds an image with the name extension-server:latest. You may need to create a different tag for it in order to allow Kubernetes to pull it correctly.

  make image
  

• Publish the extension server image in your docker repository

  • local kind server
  • other Kubernetes server

  kind load docker-image --name envoy-gateway extension-server:latest
  

  docker tag extension-server:latest $YOUR_DOCKER_REPO
  docker push $YOUR_DOCKER_REPO
  

• Deploy the extension server in your cluster

  If you are using your own docker image repository, make sure to update the values.yaml with the correct image name and tag.

  helm install -n envoy-gateway-system extension-server ./examples/extension-server/charts/extension-server
  

Configure Envoy Gateway

• Grant Envoy Gateway’s ServiceAccount permission to access the extension server’s CRD

  kubectl create clusterrole listener-context-example-status-update \
             --verb=update  \
             --resource=ListenerContextExample/status
  
  kubectl create clusterrole listener-context-example-viewer \
             --verb=get,list,watch  \
             --resource=ListenerContextExample
  
  kubectl create clusterrolebinding envoy-gateway-listener-context \
             --clusterrole=listener-context-example-viewer \
             --serviceaccount=envoy-gateway-system:envoy-gateway
  
  kubectl create clusterrolebinding envoy-gateway-listener-context-status \
             --clusterrole=listener-context-example-status-update \
             --serviceaccount=envoy-gateway-system:envoy-gateway
  

• Configure Envoy Gateway to use the Extension Server

  Add the following fragment to Envoy Gateway’s configmap:

  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: envoy-gateway-config
    namespace: envoy-gateway-system
  data:
    envoy-gateway.yaml: |
      apiVersion: gateway.envoyproxy.io/v1alpha1
      kind: EnvoyGateway
      provider:
        type: Kubernetes
      gateway:
        controllerName: gateway.envoyproxy.io/gatewayclass-controller
      extensionManager:
        # Envoy Gateway will watch these resource kinds and use them as extension policies
        # which can be attached to Gateway resources.
        policyResources:
        - group: example.extensions.io
          version: v1alpha1
          kind: ListenerContextExample
        hooks:
          # The type of hooks that should be invoked
          xdsTranslator:
            post:
            - HTTPListener
        service:
          # The service that is hosting the extension server
          fqdn:
            hostname: extension-server.envoy-gateway-system.svc.cluster.local
            port: 5005
  EOF
  

  After updating Envoy Gateway’s configmap, restart Envoy Gateway.

Testing

Get the Gateway’s address:

export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')

The extension server adds the Basic Authentication HTTP filter to all listeners configured by Envoy Gateway. Initially there are no valid user/password combinations available. Accessing the example backend should fail with a 401 status:

$ curl -v --header "Host: www.example.com" "http://${GATEWAY_HOST}/example"
...
> GET /example HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< www-authenticate: Basic realm="http://www.example.com/example"
< content-length: 58
< content-type: text/plain
< date: Mon, 08 Jul 2024 10:53:11 GMT
< 
...
User authentication failed. Missing username and password.
...

Add a new Username/Password combination using the example extension server’s CRD:

kubectl apply -f - << EOF 
apiVersion: example.extensions.io/v1alpha1
kind: ListenerContextExample
metadata:
  name: listeneruser
spec:
  targetRefs:
  - kind: Gateway
    name: eg
    group: gateway.networking.k8s.io
  username: user
  password: p@ssw0rd
EOF

Authenticating with this user/password combination will now work.

$ curl -v http://${GATEWAY_HOST}/example  -H "Host: www.example.com"   --user 'user:p@ssw0rd'
...
> GET /example HTTP/1.1
> Host: www.example.com
> Authorization: Basic dXNlcm5hbWU6cEBzc3cwcmQ=
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Mon, 08 Jul 2024 10:56:17 GMT
< content-length: 559
< 
...
 "headers": {
  "Authorization": [
   "Basic dXNlcm5hbWU6cEBzc3cwcmQ="
  ],
  "X-Example-Ext": [
   "user"
  ],
...

4 - External Processing

This task provides instructions for configuring external processing.

External processing calls an external gRPC service to process HTTP requests and responses. The external processing service can inspect and mutate requests and responses.

Envoy Gateway introduces a new CRD called EnvoyExtensionPolicy that allows the user to configure external processing. This instantiated resource can be linked to a Gateway and HTTPRoute resource.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

GRPC External Processing Service

Installation

Install a demo GRPC service that will be used as the external processing service:

kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/ext-proc-grpc-service.yaml

Create a new HTTPRoute resource to route traffic on the path /myapp to the backend service.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp
spec:
  parentRefs:
  - name: eg
  hostnames:
  - "www.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /myapp
    backendRefs:
    - name: backend
      port: 3000   
EOF

---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: myapp
spec:
  parentRefs:
  - name: eg
  hostnames:
  - "www.example.com"
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /myapp
    backendRefs:
    - name: backend
      port: 3000   

Verify the HTTPRoute status:

kubectl get httproute/myapp -o yaml

Configuration

Create a new EnvoyExtensionPolicy resource to configure the external processing service. This EnvoyExtensionPolicy targets the HTTPRoute “myApp” created in the previous step. It calls the GRPC external processing service “grpc-ext-proc” on port 9002 for processing.

By default, requests and responses are not sent to the external processor. The processingMode struct is used to define what should be sent to the external processor. In this example, we configure the following processing modes:

• The empty request field configures envoy to send request headers to the external processor.
• The response field includes configuration for body processing. As a result, response headers are sent to the external processor. Additionally, the response body is streamed to the external processor.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: ext-proc-example
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: myapp
  extProc:
  - backendRefs:
    - name: grpc-ext-proc
      port: 9002
    processingMode:
      request: {}
      response: 
        body: Streamed 
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: ext-proc-example
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: myapp
  extProc:
    - backendRefs:
        - name: grpc-ext-proc
          port: 9002
      processingMode:
        request: {}
        response: 
          body: Streamed

Verify the Envoy Extension Policy configuration:

kubectl get envoyextensionpolicy/ext-proc-example -o yaml

Because the gRPC external processing service is enabled with TLS, a BackendTLSPolicy needs to be created to configure the communication between the Envoy proxy and the gRPC auth service.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: grpc-ext-proc-btls
spec:
  targetRefs:
  - group: ''
    kind: Service
    name: grpc-ext-proc
  validation:
    caCertificateRefs:
    - name: grpc-ext-proc-ca
      group: ''
      kind: ConfigMap
    hostname: grpc-ext-proc.envoygateway
EOF

---
apiVersion: gateway.networking.k8s.io/v1alpha3
kind: BackendTLSPolicy
metadata:
  name: grpc-ext-proc-btls
spec:
  targetRefs:
    - group: ''
      kind: Service
      name: grpc-ext-proc
  validation:
    caCertificateRefs:
      - name: grpc-ext-proc-ca
        group: ''
        kind: ConfigMap
    hostname: grpc-ext-proc.envoygateway

Verify the BackendTLSPolicy configuration:

kubectl get backendtlspolicy/grpc-ext-proc-btls -o yaml

Testing

Ensure the GATEWAY_HOST environment variable from the Quickstart is set. If not, follow the Quickstart instructions to set the variable.

echo $GATEWAY_HOST

Send a request to the backend service without Authentication header:

curl -v -H "Host: www.example.com" "http://${GATEWAY_HOST}/myapp"

You should see that the external processor added headers:

• x-request-ext-processed - this header was added before the request was forwarded to the backend
• x-response-ext-processed- this header was added before the response was returned to the client

curl -v -H "Host: www.example.com"  http://localhost:10080/myapp
[...]
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Fri, 14 Jun 2024 19:30:40 GMT
< content-length: 502
< x-response-ext-processed: true
<
{
 "path": "/myapp",
 "host": "www.example.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
[...] 
  "X-Request-Ext-Processed": [
   "true"
  ],
[...]
 }

Clean-Up

Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.

Delete the demo auth services, HTTPRoute, EnvoyExtensionPolicy and BackendTLSPolicy:

kubectl delete -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/ext-proc-grpc-service.yaml
kubectl delete httproute/myapp
kubectl delete envoyextensionpolicy/ext-proc-example
kubectl delete backendtlspolicy/grpc-ext-proc-btls

Next Steps

Checkout the Developer Guide to get involved in the project.

5 - Lua Extensions

This task provides instructions for extending Envoy Gateway with Lua extensions.

Lua extensions allow you to extend the functionality of Envoy Gateway by running custom code against HTTP requests and responses, without modifying the Envoy Gateway binary. These comparatively light-weight extensions are written in the Lua scripting language using APIs defined here.

Envoy Gateway allows the user to configure Lua extensions using the EnvoyExtensionPolicy CRD. This instantiated resource can be linked to a Gateway or HTTPRoute resource. If linked to both, the resource linked to the route takes precedence over those linked to Gateway.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Configuration

Envoy Gateway supports Lua in EnvoyExtensionPolicy in two modes:

• Inline Lua: The extension defines the Lua script as an inline string.
• ValueRef Lua: The extension points to an in-cluster ConfigMap resource that contains the Lua script in it’s data.

The following example demonstrates how to configure an EnvoyExtensionPolicy to attach a Lua extension to a HTTPRoute. This Lua extension adds a custom header x-lua-custom: FOO to the response.

Lua Extension - Inline

This EnvoyExtensionPolicy configuration defines the Lua script as an inline string.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-inline-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  lua:
    - type: Inline
      inline: |
        function envoy_on_response(response_handle)
          response_handle:headers():add("x-lua-custom", "FOO")
        end
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-inline-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  lua:
    - type: Inline
      inline: |
        function envoy_on_response(response_handle)
          response_handle:headers():add("x-lua-custom", "FOO")
        end

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/lua-inline-test -o yaml

Lua Extension - ValueRef

This EnvoyExtensionPolicy configuration defines the Lua extension in a ConfigMap resource.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-valueref-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  lua:
    - type: ValueRef
      valueRef:
        name: cm-lua-valueref
        kind: ConfigMap
        group: v1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-lua-valueref
data:
  lua: |
    function envoy_on_response(response_handle)
      response_handle:headers():add("x-lua-custom", "FOO")
    end
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: lua-valueref-test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: backend
  lua:
    - type: ValueRef
      valueRef:
        name: cm-lua-valueref
        kind: ConfigMap
        group: v1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cm-lua-valueref
data:
  lua: |
    function envoy_on_response(response_handle)
      response_handle:headers():add("x-lua-custom", "FOO")
    end

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/lua-valueref-test -o yaml

Testing

Ensure the GATEWAY_HOST environment variable from the Quickstart is set. If not, follow the Quickstart instructions to set the variable.

echo $GATEWAY_HOST

Send a request to the backend service:

curl -i -H "Host: www.example.com" "http://${GATEWAY_HOST}"

You should see that the lua extension has added this header to the response:

x-lua-custom: FOO

Clean-Up

Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.

Delete the EnvoyExtensionPolicy:

kubectl delete envoyextensionpolicy/lua-inline-test
kubectl delete envoyextensionpolicy/lua-valueref-test
kubectl delete configmap/cm-lua-valueref

Next Steps

Checkout the Developer Guide to get involved in the project.

6 - OPA Sidecar with Unix Domain Socket

This task demonstrates how to deploy an Open Policy Agent (OPA) sidecar inside Envoy Proxy pods created by Envoy Gateway, with Envoy Proxy communicating with OPA over a Unix Domain Socket.

The setup provides these advantages over running OPA as its own Kubernetes service:

• Using a sidecar ensures that the OPA is always on the same worker node as Envoy Proxy. This reduces latency and simplifies access control.
• Using a Unix Domain Socket avoids TCP/TLS overhead while improving security by avoiding network exposure and reduces latency.

This production-ready approach uses Envoy Gateway’s extension APIs to configure Envoy Proxy’s external authorization and standard Kubernetes API objects to configure OPA’s Envoy External Authorization gRPC plugin.

graph TB
    subgraph K8sPod ["⎈ Envoy Proxy Pod"]
        subgraph EnvoyContainer ["Envoy Proxy Container"]
            ExtAuthz("🔐 ext_authz Filter")
        end

        UnixSocket[("Unix Domain Socket<br/>/shared/opa/opa.sock")]

        subgraph OPAContainer ["OPA Container"]
            RegoPolicy("⚙️ Rego Policy")
        end
    end

    %% Flow Definitions
    ExtAuthz <-->|"Authz check"| UnixSocket
    UnixSocket <-->|"Authz check"| RegoPolicy

    style UnixSocket fill:#D0D0D0
    style EnvoyContainer fill:#2962FF,color:#FFFFFF
    style OPAContainer fill:#2962FF,color:#FFFFFF

Note: In this diagram, the Unix Domain Socket is shown outside of the Proxy and OPA containers, but it is actually mounted to both containers.

The example flow on this page will use Envoy to authenticate JWTs and OPA to validate the ISS claim on the JWT. If the ISS matches a configured env var value then requests will be allowed.

Note that the OPA sidecar with Unix Domain Sockets pattern does not require that JWTs be used. Also, simple ISS claim validation could be done more easily using the “authentication” property of the SecurityPolicy spec, but using OPA allows for more complex logic to be applied. See JWT Claim-Based Authorization to see how to use the SecurityPolicy “authentication” property.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Steps overview

1. Enable the Backend extension API so Envoy can reference a Unix Domain Socket.
2. Create an OPA policy in a ConfigMap and mount it into an OPA sidecar in the Envoy pods.
3. Configure a SecurityPolicy that delegates authorization to OPA via the Unix Domain Socket Backend. For demonstration purposes we’ll also configure JWT authentication.
4. Create an HTTPRoute that requires protection and verify traffic flow.

Step 1: Enable the Backend extension API

Enable the Backend API in your Envoy Gateway ConfigMap to allow Envoy to reference Unix Domain Sockets by setting the “enableBackend” property to “true” in the “envoy-gateway-config” ConfigMap:

data:
  envoy-gateway.yaml: |
    ...
    extensionApis:
      enableBackend: true

Run these commands to update the ConfigMap:

OLD_CONFIG=$(kubectl get cm envoy-gateway-config -n envoy-gateway-system -o jsonpath='{.data.envoy-gateway\.yaml}')
NEW_CONFIG=$(echo "$OLD_CONFIG" | sed "s/extensionApis: {}/extensionApis:\n  enableBackend: true/")
kubectl create configmap envoy-gateway-config -n envoy-gateway-system \
  --from-literal=envoy-gateway.yaml="$NEW_CONFIG" \
  --dry-run=client -o yaml | kubectl apply -f -

Ignore the kubectl warning. It happens because kubectl apply expects a “receipt” (the last-applied-configuration annotation) to be present on the resource so it can track changes over time. If the ConfigMap was originally created using kubectl create (without --save-config) or by a Helm chart, that receipt is missing. kubectl is simply telling you that it’s adding that receipt now.

Restart the Envoy Gateway deployment to apply the updated configuration:

kubectl -n envoy-gateway-system rollout restart deploy/envoy-gateway

The Envoy Proxy pod will be restarted later in this guide.

Step 2: Create the OPA policy

Create a ConfigMap with a Rego policy that OPA will load. This example reads the “ALLOWED_ISS” environment variable and compares it to the JWT ISS claim, allowing the request only if they match.

apiVersion: v1
kind: ConfigMap
metadata:
  name: opa-policy
  namespace: envoy-gateway-system
data:
  policy.rego: |
    package envoy.authz

    import rego.v1

    default allow := false

    opa_env := opa.runtime().env
    allowed_iss := opa_env.ALLOWED_ISS
    jwt_iss := input.attributes.request.http.headers["x-jwt-iss"]

    allow if {
        jwt_iss == allowed_iss
    }

On MacOS you can apply that by copying it and running pbpaste | kubectl apply -f -.

In this policy we extract a x-jwt-iss header from the request. That header is not automatically available on ext_authz filter requests. We will configure Envoy to add that header from the decoded JWT. That configuration happens in the SecurityPolicy in step 4.

Step 3: Configure the Envoy Proxy pods with an OPA sidecar

Create an EnvoyProxy resource that mounts the OPA policy ConfigMap, shares a Unix Domain Socket between Envoy and OPA, and configures the “ext_authz” filter order so OPA receives JWT headers from the JWT authentication filter.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: eg
  namespace: envoy-gateway-system
spec:
  # Ensure ext_authz runs after jwt_authn so that OPA sees the JWT-derived headers
  # configured in the SecurityPolicy in step 4
  filterOrder:
    - name: envoy.filters.http.ext_authz
      after: envoy.filters.http.jwt_authn
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        patch:
          type: StrategicMerge
          value:
            spec:
              template:
                spec:
                  volumes:
                    # Shared Unix Domain Socket volume for communication between Envoy
                    # and OPA. This will be mounted in both containers.
                    - name: opa-unix-socket
                      emptyDir: {}
                    # OPA Rego policy file from ConfigMap. This will be mounted
                    # in the OPA container.
                    - name: opa-policy
                      configMap:
                        name: opa-policy
                  containers:
                    # Add the Unox domain socket volume to the existing envoy
                    # proxy container named "envoy".
                    - name: envoy
                      volumeMounts:
                        # Mount the Unix Domain Socket volume in the Envoy container
                        - name: opa-unix-socket
                          mountPath: /shared/opa
                    # Create the OPA sidecar container
                    - name: opa
                      # OPA image tag format:
                      # - "x.y.z": OPA version
                      # - "envoy": includes OPA's Envoy External Authorization gRPC plugin
                      # - "5": plugin version
                      # - "static": static build (no glibc dependency, smaller image)
                      image: openpolicyagent/opa:1.10.1-envoy-5-static
                      args:
                        - "run"
                        - "--server"
                        # In addition to listening on the Unix Domain Socket, we'll listen
                        # on port 8181 for k8s probes
                        - "--addr=0.0.0.0:8181"
                        # Configure OPA's Envoy external authorization plugin to listen on a Unix Domain Socket
                        - "--set=plugins.envoy_ext_authz_grpc.addr=unix:///shared/opa/opa.sock"
                        # In the OPA policy we defined the policy as being in a
                        # "envoy.authz" package and set the "allow" variable
                        # based on the policy decision. Here we tell the plugin
                        # to check that value.
                        - "--set=plugins.envoy_ext_authz_grpc.query=data.envoy.authz.allow"
                        - "--ignore=.*"
                        # You can set log-level to "debug" for detailed request logging
                        - "--log-level=error"
                        # Tell OPA where to find the policy file
                        - "/policy/policy.rego"
                      env:
                        # Environment variable used by the OPA policy to validate the JWT issuer
                        - name: ALLOWED_ISS
                          value: "https://example.com"
                      ports:
                        # Port for health checks only; gRPC server listens on the Unix Domain Socket
                        - name: http
                          containerPort: 8181
                          protocol: TCP
                      volumeMounts:
                        # Mount the Unix Domain Socket volume in the OPA container
                        - name: opa-unix-socket
                          mountPath: /shared/opa
                        # Mount the OPA Rego policy from ConfigMap
                        - name: opa-policy
                          mountPath: /policy
                          readOnly: true
                      livenessProbe:
                        httpGet:
                          path: /health
                          port: 8181
                        initialDelaySeconds: 5
                        periodSeconds: 10
                        timeoutSeconds: 3
                        failureThreshold: 3
                      readinessProbe:
                        httpGet:
                          path: /health?bundle=true
                          port: 8181
                        initialDelaySeconds: 5
                        periodSeconds: 5
                        timeoutSeconds: 3
                        failureThreshold: 3
                      securityContext:
                        # Use the same user ID as the Envoy container (65532) so both containers can access the socket
                        # The user ID was found by running "docker pull docker.io/envoyproxy/gateway:v<version> && docker inspect docker.io/envoyproxy/gateway:v<version>"
                        runAsUser: 65532
                        allowPrivilegeEscalation: false
                      resources:
                        # Adjust as necessary
                        requests:
                          cpu: 100m
                          memory: 128Mi
                        limits:
                          memory: 512Mi

This patch adds an OPA sidecar container to the Envoy pods, creates a shared Unix Domain Socket at /shared/opa/opa.sock, and configures OPA to listen only on this socket instead of TCP. It sets an ALLOWED_ISS property that OPA will use to validate the request. We purposefully set the wrong value here so we can see the response when the wrong ISS is used. We’ll set it to the correct value later.

Tell Envoy Gateway to use this EnvoyProxy spec by patching the GatewayClass.

kubectl patch gatewayclass eg --type merge -p '
{
  "spec": {
    "parametersRef": {
      "group": "gateway.envoyproxy.io",
      "kind": "EnvoyProxy",
      "name": "eg",
      "namespace": "envoy-gateway-system"
    }
  }
}'

This will cause the Envoy Proxy pod to be replaced with the new config.

Step 4: Connect Envoy to OPA with a SecurityPolicy and Backend

Create a Unix Domain Socket SecurityPolicy and a Backend that authenticates JWTs, maps the ISS claims to the x-jwt-iss header, and delegates authorization to OPA.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
  name: opa-jwt-authz
  # The SecurityPolicy must be in the same namespace as the HTTPRoutes that it
  # will apply to
  namespace: default
spec:
  targetSelectors:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      matchLabels:
        auth: opa-jwt
  jwt:
    providers:
      - name: my-jwt-validator
        # Verify that the JWT is signed with the correct key
        remoteJWKS:
          uri: https://raw.githubusercontent.com/envoyproxy/gateway/refs/heads/main/examples/kubernetes/jwt/jwks.json
        # Add the JWT ISS claim to request to the headers sent to OPA. Note: this only works
        # for string claims, not arrays. For example, the "aud" claim cannot be passed
        # this way because the JWT spec defines it as an array.
        claimToHeaders:
          - header: x-jwt-iss
            claim: iss
  # Delegate authorization to an external program after JWT verification
  extAuth:
    grpc:
      backendRefs:
        # Reference the OPA Unix Domain Socket backend, which is the next API object in this code block
        - group: gateway.envoyproxy.io
          kind: Backend
          name: opa-unix-socket
    # Headers to send to OPA for authorization decisions. This should match the
    # "claimToHeaders" values above.
    headersToExtAuth:
      - x-jwt-iss
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
  name: opa-unix-socket
  # The Backend must be in the same namespace as the SecurityPolicy
  namespace: default
spec:
  endpoints:
    # This configures Envoy proxy to send external auth requests to the Unix Domain Socket
    - unix:
        path: /shared/opa/opa.sock

Step 5: Configure route to use the OPA authorization

Attach the policy by labeling routes that require authorization. This HTTPRoute uses the backend service installed in the Quick Start:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: test-opa
  namespace: default
  labels:
    auth: opa-jwt
spec:
  hostnames:
  - foo.bar.com
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: eg
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /test-opa
    backendRefs:
    - name: backend
      port: 3000

Step 5: Test the configuration

Prepare some env vars:

export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
export VALID_TOKEN="eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6ImI1MjBiM2MyYzRiZDc1YTEwZTljZWJjOTU3NjkzM2RjIn0.eyJpc3MiOiJodHRwczovL2Zvby5iYXIuY29tIiwic3ViIjoiMTIzNDU2Nzg5MCIsInVzZXIiOnsibmFtZSI6IkpvaG4gRG9lIiwiZW1haWwiOiJqb2huLmRvZUBleGFtcGxlLmNvbSIsInJvbGVzIjpbImFkbWluIiwiZWRpdG9yIl19LCJwcmVtaXVtX3VzZXIiOnRydWUsImlhdCI6MTUxNjIzOTAyMiwic2NvcGUiOiJyZWFkIGFkZCBkZWxldGUgbW9kaWZ5In0.P36iAlmiRCC79OiB3vstF5Q_9OqUYAMGF3a3H492GlojbV6DcuOz8YIEYGsRSWc-BNJaBKlyvUKsKsGVPtYbbF8ajwZTs64wyO-zhd2R8riPkg_HsW7iwGswV12f5iVRpfQ4AG2owmdOToIaoch0aym89He1ZzEjcShr9olgqlAbbmhnk-namd1rP-xpzPnWhhIVI3mCz5hYYgDTMcM7qbokM5FzFttTRXAn5_Luor23U1062Ct_K53QArwxBvwJ-QYiqcBycHf-hh6sMx_941cUswrZucCpa-EwA3piATf9PKAyeeWHfHV9X-y8ipGOFg3mYMMVBuUZ1lBkJCik9f9kboRY6QzpOISARQj9PKMXfxZdIPNuGmA7msSNAXQgqkvbx04jMwb9U7eCEdGZztH4C8LhlRjgj0ZdD7eNbRjeH2F6zrWyMUpGWaWyq6rMuP98W2DWM5ZflK6qvT1c7FuFsWPvWLkgxQwTWQKrHdKwdbsu32Sj8VtUBJ0-ddEb"

If Envoy is not running behind a load balancer, then you’ll have to start port forwarding whenever the Envoy Proxy pod is reconfigured and the proxy pods are replaced:

kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8888:80 &

Test without a JWT

Requests without a valid JWT are denied:

curl -v -H "Host: foo.bar.com" "http://${GATEWAY_HOST}/test-opa/get"

Expected result: HTTP 401 Unauthorized

Test with a valid JWT that has the wrong ISS claim

Earlier, we configured the “ALLOWED_ISS” env var on the proxy pods to be “https://example.com”, but the JWT in the “VALID_TOKEN” env var has an ISS claim of “https://foo.bar.com”.

Requests with an incorrect ISS are denied:

curl -v -H "Host: foo.bar.com" -H "Authorization: Bearer $VALID_TOKEN" "http://${GATEWAY_HOST}/test-opa/get"

Expected result: HTTP 403 Forbidden

Test with a valid JWT and ISS claim

We must update the EnvoyProxy spec to use “https://foo.bar.com” for the “ALLOWED_ISS” so that it matches the ISS in “$VALID_TOKEN”.

...
                  containers:
                    - name: opa
                      env:
                        - name: ALLOWED_ISS
                          value: "https://foo.bar.com" # Change to this
...

The quickest way to do that for this tutorial is via kubectl -n envoy-gateway-system edit envoyproxy eg.

Wait for the Envoy Proxy pod to be replaced.

When the JWT issuer matches your configured provider and the OPA policy allows it, the request is forwarded to the backend:

curl -v -H "Host: foo.bar.com" -H "Authorization: Bearer $VALID_TOKEN" "http://${GATEWAY_HOST}/test-opa/get"

Expected result: HTTP 200 and a response from the backend service.

Clean up

Delete all resources created by this task:

kubectl delete httproute/test-opa securitypolicy/opa-jwt-authz backend/opa-unix-socket
kubectl delete envoyproxy/eg configmap/opa-policy -n envoy-gateway-system
kubectl delete deploy/$(kubectl get deploy -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}') -n envoy-gateway-system

7 - Wasm Extensions

This task provides instructions for extending Envoy Gateway with WebAssembly (Wasm) extensions.

Wasm extensions allow you to extend the functionality of Envoy Gateway by running custom code against HTTP requests and responses, without modifying the Envoy Gateway binary. These extensions can be written in any language that compiles to Wasm, such as C++, Rust, AssemblyScript, or TinyGo.

Envoy Gateway introduces a new CRD called EnvoyExtensionPolicy that allows the user to configure Wasm extensions. This instantiated resource can be linked to a Gateway and HTTPRoute resource.

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Configuration

Envoy Gateway supports two types of Wasm extensions:

• HTTP Wasm Extension: The Wasm extension is fetched from a remote URL.
• Image Wasm Extension: The Wasm extension is packaged as an OCI image and fetched from an image registry.

The following example demonstrates how to configure an EnvoyExtensionPolicy to attach a Wasm extension to an EnvoyExtensionPolicy . This Wasm extension adds a custom header x-wasm-custom: FOO to the response.

HTTP Wasm Extension

This EnvoyExtensionPolicy configuration fetches the Wasm extension from an HTTP URL.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  wasm:
  - name: wasm-filter
    rootID: my_root_id
    code:
      type: HTTP
      http:
        url: https://raw.githubusercontent.com/envoyproxy/examples/main/wasm-cc/lib/envoy_filter_http_wasm_example.wasm
        sha256: 79c9f85128bb0177b6511afa85d587224efded376ac0ef76df56595f1e6315c0
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  wasm:
  - name: wasm-filter
    rootID: my_root_id
    code:
      type: HTTP
      http:
        url: https://raw.githubusercontent.com/envoyproxy/examples/main/wasm-cc/lib/envoy_filter_http_wasm_example.wasm
        sha256: 79c9f85128bb0177b6511afa85d587224efded376ac0ef76df56595f1e6315c0

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/wasm-test -o yaml

Image Wasm Extension

This EnvoyExtensionPolicy configuration fetches the Wasm extension from an OCI image.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  wasm:
  - name: wasm-filter
    rootID: my_root_id
    code:
      type: Image
      image:
        url: zhaohuabing/testwasm:v0.0.1
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyExtensionPolicy
metadata:
  name: wasm-test
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: backend
  wasm:
  - name: wasm-filter
    rootID: my_root_id
    code:
      type: Image
      image:
        url: zhaohuabing/testwasm:v0.0.1

Verify the EnvoyExtensionPolicy status:

kubectl get envoyextensionpolicy/wasm-test -o yaml

Testing

Ensure the GATEWAY_HOST environment variable from the Quickstart is set. If not, follow the Quickstart instructions to set the variable.

echo $GATEWAY_HOST

Send a request to the backend service:

curl -i -H "Host: www.example.com" "http://${GATEWAY_HOST}"

You should see that the wasm extension has added this header to the response:

x-wasm-custom: FOO

Clean-Up

Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.

Delete the EnvoyExtensionPolicy:

kubectl delete envoyextensionpolicy/wasm-test

Next Steps

Checkout the Developer Guide to get involved in the project.