Envoy Gateway Extension Server

This task explains how to extend Envoy Gateway using an Extension Server. Envoy Gateway can be configured to call an external server over gRPC with the xDS configuration before it is sent to Envoy Proxy. The external server can modify the provided configuration programmatically using any semantics supported by the xDS API.

Using an extension server allows vendors to add xDS configuration that Envoy Gateway itself doesn’t support with a very high level of control over the generated xDS configuration.

Note: Modifying the xDS configuration generated by Envoy Gateway may break functionality configured by native Envoy Gateway means. Like other cases where the xDS configuration is modified outside of Envoy Gateway’s control, this is risky and should be tested thoroughly, especially when using the same extension server across different Envoy Gateway versions.

Introduction

One of the Envoy Gateway project goals is to “provide a common foundation for vendors to build value-added products without having to re-engineer fundamental interactions”. The Envoy Gateway Extension Server provides a mechanism where Envoy Gateway tracks all provider resources and then calls a set of hooks that allow the generated xDS configuration to be modified before it is sent to Envoy Proxy. See the design documentation for full details.

Extension Hooks Overview

Envoy Gateway provides several extension hooks that are called at different stages of the xDS translation process. These hooks allow extensions to modify various aspects of the generated xDS configuration:

Available Hooks

Hook Execution Order

The hooks are executed in the following order during xDS translation:

1. Route Processing Phase

   • Route hook (PostRouteModifyHook): Called for each route that has extension filters attached
   • Cluster hook (PostClusterModifyHook): Called for each cluster generated from custom backend references

2. Virtual Host Processing Phase

   • VirtualHost hook (PostVirtualHostModifyHook): Called for each virtual host after all routes are processed

3. Listener Processing Phase

   • HTTPListener hook (PostHTTPListenerModifyHook): Called for each HTTP listener after virtual hosts are configured

4. Final Translation Phase

   • Translation hook (PostTranslateModifyHook): Called once with all generated clusters, secrets, listeners, and routes

Hook Details

Route Hook (PostRouteModifyHook)

• When called: After each individual route is generated from an HTTPRoute with extension filters
• Input: Single route, hostnames, and extension resources from the HTTPRoute
• Output: Modified route
• Use cases: Add route-specific filters, modify route configuration, add typed per-filter config

Cluster Hook (PostClusterModifyHook)

• When called: During cluster generation for custom backend references only
• Input: Single cluster and extension resources from the backend reference
• Output: Modified cluster
• Use cases: Configure custom backend clusters, add health checks, modify load balancing

VirtualHost Hook (PostVirtualHostModifyHook)

• When called: After all routes for a virtual host are processed
• Input: Complete virtual host with all its routes
• Output: Modified virtual host (can add/remove/modify routes)
• Use cases: Add virtual host-level configuration, inject additional routes

HTTPListener Hook (PostHTTPListenerModifyHook)

• When called: After listener is fully configured with all virtual hosts
• Input: Complete HTTP listener and associated extension policies
• Output: Modified listener
• Use cases: Add listener filters, modify listener configuration, add authentication

Translation Hook (PostTranslateModifyHook)

• When called: After all individual resources are generated and processed
• Input: All clusters, secrets, listeners, and routes, plus extension policies
• Output: Complete set of modified resources
• Use cases: Global resource injection, cross-resource modifications, cleanup operations

Configuration

Extensions must specify which hooks they want to use in the extensionManager.hooks configuration:

extensionManager:
  hooks:
    xdsTranslator:
      post:
        - Route          # Enable route modification hook
        - VirtualHost    # Enable virtual host modification hook
        - HTTPListener   # Enable HTTP listener modification hook
        - Cluster        # Enable cluster modification hook
        - Translation    # Enable global translation hook
      # Configure which resources to include in PostTranslateModifyHook
      # Default: true for clusters and secrets (for backward compatibility)
      # Default: false for listeners and routes (for backward compatibility)
      translation:
        listener:
          includeAll: true
        route:
          includeAll: true

This task sets up an example extension server that adds the Envoy Proxy Basic Authentication HTTP filter to all the listeners generated by Envoy Gateway. The example extension server includes its own CRD which allows defining username/password pairs that will be accepted by the Envoy Proxy.

Note: Envoy Gateway supports adding Basic Authentication to routes using a SecurityPolicy. See this task for the preferred way to configure Basic Authentication.

Quickstart

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Build and run the example Extension Server

Build and deploy the example extension server in the examples/extension-server folder into the cluster running Envoy Gateway.

• Build the extension server image

  Note: The provided Makefile builds an image with the name extension-server:latest. You may need to create a different tag for it in order to allow Kubernetes to pull it correctly.

  make image
  

• Publish the extension server image in your docker repository

  • local kind server
  • other Kubernetes server

  kind load docker-image --name envoy-gateway extension-server:latest
  

  docker tag extension-server:latest $YOUR_DOCKER_REPO
  docker push $YOUR_DOCKER_REPO
  

• Deploy the extension server in your cluster

  If you are using your own docker image repository, make sure to update the values.yaml with the correct image name and tag.

  helm install -n envoy-gateway-system extension-server ./examples/extension-server/charts/extension-server
  

Configure Envoy Gateway

• Grant Envoy Gateway’s ServiceAccount permission to access the extension server’s CRD

  kubectl create clusterrole listener-context-example-status-update \
             --verb=update  \
             --resource=ListenerContextExample/status
  
  kubectl create clusterrole listener-context-example-viewer \
             --verb=get,list,watch  \
             --resource=ListenerContextExample
  
  kubectl create clusterrolebinding envoy-gateway-listener-context \
             --clusterrole=listener-context-example-viewer \
             --serviceaccount=envoy-gateway-system:envoy-gateway
  
  kubectl create clusterrolebinding envoy-gateway-listener-context-status \
             --clusterrole=listener-context-example-status-update \
             --serviceaccount=envoy-gateway-system:envoy-gateway
  

• Configure Envoy Gateway to use the Extension Server

  Add the following fragment to Envoy Gateway’s configmap:

  cat <<EOF | kubectl apply -f -
  apiVersion: v1
  kind: ConfigMap
  metadata:
    name: envoy-gateway-config
    namespace: envoy-gateway-system
  data:
    envoy-gateway.yaml: |
      apiVersion: gateway.envoyproxy.io/v1alpha1
      kind: EnvoyGateway
      provider:
        type: Kubernetes
      gateway:
        controllerName: gateway.envoyproxy.io/gatewayclass-controller
      extensionManager:
        # Envoy Gateway will watch these resource kinds and use them as extension policies
        # which can be attached to Gateway resources.
        policyResources:
        - group: example.extensions.io
          version: v1alpha1
          kind: ListenerContextExample
        hooks:
          # The type of hooks that should be invoked
          xdsTranslator:
            post:
            - HTTPListener
        service:
          # The service that is hosting the extension server
          fqdn:
            hostname: extension-server.envoy-gateway-system.svc.cluster.local
            port: 5005
  EOF
  

  After updating Envoy Gateway’s configmap, restart Envoy Gateway.

Testing

Get the Gateway’s address:

export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')

The extension server adds the Basic Authentication HTTP filter to all listeners configured by Envoy Gateway. Initially there are no valid user/password combinations available. Accessing the example backend should fail with a 401 status:

$ curl -v --header "Host: www.example.com" "http://${GATEWAY_HOST}/example"
...
> GET /example HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< www-authenticate: Basic realm="http://www.example.com/example"
< content-length: 58
< content-type: text/plain
< date: Mon, 08 Jul 2024 10:53:11 GMT
< 
...
User authentication failed. Missing username and password.
...

Add a new Username/Password combination using the example extension server’s CRD:

kubectl apply -f - << EOF 
apiVersion: example.extensions.io/v1alpha1
kind: ListenerContextExample
metadata:
  name: listeneruser
spec:
  targetRefs:
  - kind: Gateway
    name: eg
    group: gateway.networking.k8s.io
  username: user
  password: p@ssw0rd
EOF

Authenticating with this user/password combination will now work.

$ curl -v http://${GATEWAY_HOST}/example  -H "Host: www.example.com"   --user 'user:p@ssw0rd'
...
> GET /example HTTP/1.1
> Host: www.example.com
> Authorization: Basic dXNlcm5hbWU6cEBzc3cwcmQ=
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Mon, 08 Jul 2024 10:56:17 GMT
< content-length: 559
< 
...
 "headers": {
  "Authorization": [
   "Basic dXNlcm5hbWU6cEBzc3cwcmQ="
  ],
  "X-Example-Ext": [
   "user"
  ],
...