Notice:
This is the "latest" release of Envoy Gateway, which contains the most recent commits from the main branch.
This release might not be stable.
Please refer to the /docs documentation for the most current information.

Envoy Gateway Extension Server

This task explains how to extend Envoy Gateway using an Extension Server. Envoy Gateway can be configured to call an external server over gRPC with the xDS configuration before it is sent to Envoy Proxy. The external server can modify the provided configuration programmatically using any semantics supported by the xDS API.

Using an extension server allows vendors to add xDS configuration that Envoy Gateway itself doesn’t support with a very high level of control over the generated xDS configuration.

Note: Modifying the xDS configuration generated by Envoy Gateway may break functionality configured by native Envoy Gateway means. Like other cases where the xDS configuration is modified outside of Envoy Gateway’s control, this is risky and should be tested thoroughly, especially when using the same extension server across different Envoy Gateway versions.

Introduction

One of the Envoy Gateway project goals is to “provide a common foundation for vendors to build value-added products without having to re-engineer fundamental interactions”. The Envoy Gateway Extension Server provides a mechanism where Envoy Gateway tracks all provider resources and then calls a set of hooks that allow the generated xDS configuration to be modified before it is sent to Envoy Proxy. See the design documentation for full details.

Extension Hooks Overview

Envoy Gateway provides several extension hooks that are called at different stages of the xDS translation process. These hooks allow extensions to modify various aspects of the generated xDS configuration:

Available Hooks

Hook TypeHook Method NamePurposeResources ModifiedExecution Context
RoutePostRouteModifyHookModify individual routesSingle envoy.config.route.v3.RoutePer-route, only for routes with extension filters
ClusterPostClusterModifyHookModify clusters for custom backendsSingle envoy.config.cluster.v3.ClusterPer-cluster, only for custom backend clusters
VirtualHostPostVirtualHostModifyHookModify virtual hosts and add custom routesSingle envoy.config.route.v3.VirtualHostPer-virtual-host
HTTPListenerPostHTTPListenerModifyHookModify HTTP listenersSingle envoy.config.listener.v3.ListenerPer-listener
TranslationPostTranslateModifyHookGlobal modification of all xDS resourcesAll clusters, secrets, listeners, and routesOnce per translation cycle

Hook Execution Order

The hooks are executed in the following order during xDS translation:

  1. Route Processing Phase

    • Route hook (PostRouteModifyHook): Called for each route that has extension filters attached
    • Cluster hook (PostClusterModifyHook): Called for each cluster generated from custom backend references
  2. Virtual Host Processing Phase

    • VirtualHost hook (PostVirtualHostModifyHook): Called for each virtual host after all routes are processed
  3. Listener Processing Phase

    • HTTPListener hook (PostHTTPListenerModifyHook): Called for each HTTP listener after virtual hosts are configured
  4. Final Translation Phase

    • Translation hook (PostTranslateModifyHook): Called once with all generated clusters, secrets, listeners, and routes

Hook Details

Route Hook (PostRouteModifyHook)

  • When called: After each individual route is generated from an HTTPRoute with extension filters
  • Input: Single route, hostnames, and extension resources from the HTTPRoute
  • Output: Modified route
  • Use cases: Add route-specific filters, modify route configuration, add typed per-filter config

Cluster Hook (PostClusterModifyHook)

  • When called: During cluster generation for custom backend references only
  • Input: Single cluster and extension resources from the backend reference
  • Output: Modified cluster
  • Use cases: Configure custom backend clusters, add health checks, modify load balancing

VirtualHost Hook (PostVirtualHostModifyHook)

  • When called: After all routes for a virtual host are processed
  • Input: Complete virtual host with all its routes
  • Output: Modified virtual host (can add/remove/modify routes)
  • Use cases: Add virtual host-level configuration, inject additional routes

HTTPListener Hook (PostHTTPListenerModifyHook)

  • When called: After listener is fully configured with all virtual hosts
  • Input: Complete HTTP listener and associated extension policies
  • Output: Modified listener
  • Use cases: Add listener filters, modify listener configuration, add authentication

Translation Hook (PostTranslateModifyHook)

  • When called: After all individual resources are generated and processed
  • Input: All clusters, secrets, listeners, and routes, plus extension policies
  • Output: Complete set of modified resources
  • Use cases: Global resource injection, cross-resource modifications, cleanup operations

Configuration

Extensions must specify which hooks they want to use in the extensionManager.hooks configuration:

extensionManager:
  hooks:
    xdsTranslator:
      post:
        - Route          # Enable route modification hook
        - VirtualHost    # Enable virtual host modification hook
        - HTTPListener   # Enable HTTP listener modification hook
        - Cluster        # Enable cluster modification hook
        - Translation    # Enable global translation hook
      # Configure which resources to include in PostTranslateModifyHook
      # Default: true for clusters and secrets (for backward compatibility)
      # Default: false for listeners and routes (for backward compatibility)
      translation:
        listener:
          includeAll: true
        route:
          includeAll: true

This task sets up an example extension server that adds the Envoy Proxy Basic Authentication HTTP filter to all the listeners generated by Envoy Gateway. The example extension server includes its own CRD which allows defining username/password pairs that will be accepted by the Envoy Proxy.

Note: Envoy Gateway supports adding Basic Authentication to routes using a SecurityPolicy. See this task for the preferred way to configure Basic Authentication.

Quickstart

Prerequisites

Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Expand for instructions
  1. Install the Gateway API CRDs and Envoy Gateway using Helm:

    helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.0.0-latest -n envoy-gateway-system --create-namespace
    
  2. Install the GatewayClass, Gateway, HTTPRoute and example app:

    kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default
    
  3. Verify Connectivity:

    Get the External IP of the Gateway:

    export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
       

    Curl the example app through Envoy proxy:

    curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/get
       

    The above command should succeed with status code 200.

    Get the name of the Envoy service created the by the example Gateway:

    export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
       

    Get the deployment of the Envoy service created the by the example Gateway:

    export ENVOY_DEPLOYMENT=$(kubectl get deploy -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
       

    Port forward to the Envoy service:

    kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8888:80 &
       

    Curl the example app through Envoy proxy:

    curl --verbose --header "Host: www.example.com" http://localhost:8888/get
       

    The above command should succeed with status code 200.

Build and run the example Extension Server

Build and deploy the example extension server in the examples/extension-server folder into the cluster running Envoy Gateway.

  • Build the extension server image

    Note: The provided Makefile builds an image with the name extension-server:latest. You may need to create a different tag for it in order to allow Kubernetes to pull it correctly.

    make image
    
  • Publish the extension server image in your docker repository

    kind load docker-image --name envoy-gateway extension-server:latest
    
    docker tag extension-server:latest $YOUR_DOCKER_REPO
    docker push $YOUR_DOCKER_REPO
    
  • Deploy the extension server in your cluster

    If you are using your own docker image repository, make sure to update the values.yaml with the correct image name and tag.

    helm install -n envoy-gateway-system extension-server ./examples/extension-server/charts/extension-server
    

Configure Envoy Gateway

  • Grant Envoy Gateway’s ServiceAccount permission to access the extension server’s CRD

    kubectl create clusterrole listener-context-example-status-update \
               --verb=update  \
               --resource=ListenerContextExample/status
    
    kubectl create clusterrole listener-context-example-viewer \
               --verb=get,list,watch  \
               --resource=ListenerContextExample
    
    kubectl create clusterrolebinding envoy-gateway-listener-context \
               --clusterrole=listener-context-example-viewer \
               --serviceaccount=envoy-gateway-system:envoy-gateway
    
    kubectl create clusterrolebinding envoy-gateway-listener-context-status \
               --clusterrole=listener-context-example-status-update \
               --serviceaccount=envoy-gateway-system:envoy-gateway
    
  • Configure Envoy Gateway to use the Extension Server

    Add the following fragment to Envoy Gateway’s configmap:

    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: ConfigMap
    metadata:
      name: envoy-gateway-config
      namespace: envoy-gateway-system
    data:
      envoy-gateway.yaml: |
        apiVersion: gateway.envoyproxy.io/v1alpha1
        kind: EnvoyGateway
        provider:
          type: Kubernetes
        gateway:
          controllerName: gateway.envoyproxy.io/gatewayclass-controller
        extensionManager:
          # Envoy Gateway will watch these resource kinds and use them as extension policies
          # which can be attached to Gateway resources.
          policyResources:
          - group: example.extensions.io
            version: v1alpha1
            kind: ListenerContextExample
          hooks:
            # The type of hooks that should be invoked
            xdsTranslator:
              post:
              - HTTPListener
          service:
            # The service that is hosting the extension server
            fqdn:
              hostname: extension-server.envoy-gateway-system.svc.cluster.local
              port: 5005
    EOF
    

    After updating Envoy Gateway’s configmap, restart Envoy Gateway.

Testing

Get the Gateway’s address:

export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')

The extension server adds the Basic Authentication HTTP filter to all listeners configured by Envoy Gateway. Initially there are no valid user/password combinations available. Accessing the example backend should fail with a 401 status:

$ curl -v --header "Host: www.example.com" "http://${GATEWAY_HOST}/example"
...
> GET /example HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< www-authenticate: Basic realm="http://www.example.com/example"
< content-length: 58
< content-type: text/plain
< date: Mon, 08 Jul 2024 10:53:11 GMT
< 
...
User authentication failed. Missing username and password.
...

Add a new Username/Password combination using the example extension server’s CRD:

kubectl apply -f - << EOF 
apiVersion: example.extensions.io/v1alpha1
kind: ListenerContextExample
metadata:
  name: listeneruser
spec:
  targetRefs:
  - kind: Gateway
    name: eg
    group: gateway.networking.k8s.io
  username: user
  password: p@ssw0rd
EOF

Authenticating with this user/password combination will now work.

$ curl -v http://${GATEWAY_HOST}/example  -H "Host: www.example.com"   --user 'user:p@ssw0rd'
...
> GET /example HTTP/1.1
> Host: www.example.com
> Authorization: Basic dXNlcm5hbWU6cEBzc3cwcmQ=
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Mon, 08 Jul 2024 10:56:17 GMT
< content-length: 559
< 
...
 "headers": {
  "Authorization": [
   "Basic dXNlcm5hbWU6cEBzc3cwcmQ="
  ],
  "X-Example-Ext": [
   "user"
  ],
...