Envoy Gateway Extension Server

This task explains how to extend Envoy Gateway using an Extension Server. Envoy Gateway can be configured to call an external server over gRPC with the xDS configuration before it is sent to Envoy Proxy. The external server can modify the provided configuration programmatically using any semantics supported by the xDS API.

Using an extension server allows vendors to add xDS configuration that Envoy Gateway itself doesn’t support with a very high level of control over the generated xDS configuration.

Note: Modifying the xDS configuration generated by Envoy Gateway may break functionality configured by native Envoy Gateway means. Like other cases where the xDS configuration is modified outside of Envoy Gateway’s control, this is risky and should be tested thoroughly, especially when using the same extension server across different Envoy Gateway versions.

Introduction

One of the Envoy Gateway project goals is to “provide a common foundation for vendors to build value-added products without having to re-engineer fundamental interactions”. The Envoy Gateway Extension Server provides a mechanism where Envoy Gateway tracks all provider resources and then calls a set of hooks that allow the generated xDS configuration to be modified before it is sent to Envoy Proxy. See the design documentation for full details.

This task sets up an example extension server that adds the Envoy Proxy Basic Authentication HTTP filter to all the listeners generated by Envoy Gateway. The example extension server includes its own CRD which allows defining username/password pairs that will be accepted by the Envoy Proxy.

Note: Envoy Gateway supports adding Basic Authentication to routes using a SecurityPolicy. See this task for the preferred way to configure Basic Authentication.

Quickstart

Prerequisites

Follow the steps from the Quickstart task to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Verify the Gateway status:

• kubectl
• egctl (experimental)

kubectl get gateway/eg -o yaml

egctl x status gateway -v

Build and run the example Extension Server

Build and deploy the example extension server in the examples/extension-server folder into the cluster running Envoy Gateway.

• Build the extension server image

  Note: The provided Makefile builds an image with the name extension-server:latest. You may need to create a different tag for it in order to allow Kubernetes to pull it correctly.

  make image
  

• Publish the extension server image in your docker repository

  • local kind server
  • other Kubernetes server

  kind load docker-image --name envoy-gateway extension-server:latest
  

  docker tag extension-server:latest $YOUR_DOCKER_REPO
  docker push $YOUR_DOCKER_REPO
  

• Deploy the extension server in your cluster

  If you are using your own docker image repository, make sure to update the values.yaml with the correct image name and tag.

  helm install -n envoy-gateway-system extension-server ./examples/extension-server/charts/extension-server
  

Configure Envoy Gateway

• Grant Envoy Gateway’s ServiceAccount permission to access the extension server’s CRD

  kubectl create clusterrole listener-context-example-status-update \
             --verb=update  \
             --resource=ListenerContextExample/status
  
  kubectl create clusterrole listener-context-example-viewer \
             --verb=get,list,watch  \
             --resource=ListenerContextExample
  
  kubectl create clusterrolebinding envoy-gateway-listener-context \
             --clusterrole=listener-context-example-viewer \
             --serviceaccount=envoy-gateway-system:envoy-gateway
  
  kubectl create clusterrolebinding envoy-gateway-listener-context-status \
             --clusterrole=listener-context-example-status-update \
             --serviceaccount=envoy-gateway-system:envoy-gateway
  

• Configure Envoy Gateway to use the Extension Server

  Add the following fragment to Envoy Gateway’s configuration file:

  extensionManager:
    # Envoy Gateway will watch these resource kinds and use them as extension policies
    # which can be attached to Gateway resources.
    policyResources:
    - group: example.extensions.io
      version: v1alpha1
      kind: ListenerContextExample
    hooks:
      # The type of hooks that should be invoked
      xdsTranslator:
        post:
        - HTTPListener
    service:
      # The service that is hosting the extension server
      fqdn: 
        hostname: extension-server.envoy-gateway-system.svc.cluster.local
        port: 5005
  

  After updating Envoy Gateway’s configuration file, restart Envoy Gateway.

Testing

Get the Gateway’s address:

export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')

The extension server adds the Basic Authentication HTTP filter to all listeners configured by Envoy Gateway. Initially there are no valid user/password combinations available. Accessing the example backend should fail with a 401 status:

$ curl -v --header "Host: www.example.com" "http://${GATEWAY_HOST}/example"
...
> GET /example HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< www-authenticate: Basic realm="http://www.example.com/example"
< content-length: 58
< content-type: text/plain
< date: Mon, 08 Jul 2024 10:53:11 GMT
< 
...
User authentication failed. Missing username and password.
...

Add a new Username/Password combination using the example extension server’s CRD:

kubectl apply -f - << EOF 
apiVersion: example.extensions.io/v1alpha1
kind: ListenerContextExample
metadata:
  name: listeneruser
spec:
  targetRefs:
  - kind: Gateway
    name: eg
    group: gateway.networking.k8s.io
  username: user
  password: p@ssw0rd
EOF

Authenticating with this user/password combination will now work.

$ curl -v http://${GATEWAY_HOST}/example  -H "Host: www.example.com"   --user 'user:p@ssw0rd'
...
> GET /example HTTP/1.1
> Host: www.example.com
> Authorization: Basic dXNlcm5hbWU6cEBzc3cwcmQ=
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Mon, 08 Jul 2024 10:56:17 GMT
< content-length: 559
< 
...
 "headers": {
  "Authorization": [
   "Basic dXNlcm5hbWU6cEBzc3cwcmQ="
  ],
  "X-Example-Ext": [
   "user"
  ],
...