This is the "latest" release of Envoy Gateway, which contains the most recent commits from the main branch.
This release might not be stable.
Please refer to the /docs documentation for the most current information.
Envoy Gateway Extension Server
7 minute read
This task explains how to extend Envoy Gateway using an Extension Server. Envoy Gateway can be configured to call an external server over gRPC with the xDS configuration before it is sent to Envoy Proxy. The external server can modify the provided configuration programmatically using any semantics supported by the xDS API.
Using an extension server allows vendors to add xDS configuration that Envoy Gateway itself doesn’t support with a very high level of control over the generated xDS configuration.
Note: Modifying the xDS configuration generated by Envoy Gateway may break functionality configured by native Envoy Gateway means. Like other cases where the xDS configuration is modified outside of Envoy Gateway’s control, this is risky and should be tested thoroughly, especially when using the same extension server across different Envoy Gateway versions.
Introduction
One of the Envoy Gateway project goals is to “provide a common foundation for vendors to build value-added products without having to re-engineer fundamental interactions”. The Envoy Gateway Extension Server provides a mechanism where Envoy Gateway tracks all provider resources and then calls a set of hooks that allow the generated xDS configuration to be modified before it is sent to Envoy Proxy. See the design documentation for full details.
Extension Hooks Overview
Envoy Gateway provides several extension hooks that are called at different stages of the xDS translation process. These hooks allow extensions to modify various aspects of the generated xDS configuration:
Available Hooks
Hook Type | Hook Method Name | Purpose | Resources Modified | Execution Context |
---|---|---|---|---|
Route | PostRouteModifyHook | Modify individual routes | Single envoy.config.route.v3.Route | Per-route, only for routes with extension filters |
Cluster | PostClusterModifyHook | Modify clusters for custom backends | Single envoy.config.cluster.v3.Cluster | Per-cluster, only for custom backend clusters |
VirtualHost | PostVirtualHostModifyHook | Modify virtual hosts and add custom routes | Single envoy.config.route.v3.VirtualHost | Per-virtual-host |
HTTPListener | PostHTTPListenerModifyHook | Modify HTTP listeners | Single envoy.config.listener.v3.Listener | Per-listener |
Translation | PostTranslateModifyHook | Global modification of all xDS resources | All clusters, secrets, listeners, and routes | Once per translation cycle |
Hook Execution Order
The hooks are executed in the following order during xDS translation:
Route Processing Phase
Route
hook (PostRouteModifyHook
): Called for each route that has extension filters attachedCluster
hook (PostClusterModifyHook
): Called for each cluster generated from custom backend references
Virtual Host Processing Phase
VirtualHost
hook (PostVirtualHostModifyHook
): Called for each virtual host after all routes are processed
Listener Processing Phase
HTTPListener
hook (PostHTTPListenerModifyHook
): Called for each HTTP listener after virtual hosts are configured
Final Translation Phase
Translation
hook (PostTranslateModifyHook
): Called once with all generated clusters, secrets, listeners, and routes
Hook Details
Route Hook (PostRouteModifyHook
)
- When called: After each individual route is generated from an HTTPRoute with extension filters
- Input: Single route, hostnames, and extension resources from the HTTPRoute
- Output: Modified route
- Use cases: Add route-specific filters, modify route configuration, add typed per-filter config
Cluster Hook (PostClusterModifyHook
)
- When called: During cluster generation for custom backend references only
- Input: Single cluster and extension resources from the backend reference
- Output: Modified cluster
- Use cases: Configure custom backend clusters, add health checks, modify load balancing
VirtualHost Hook (PostVirtualHostModifyHook
)
- When called: After all routes for a virtual host are processed
- Input: Complete virtual host with all its routes
- Output: Modified virtual host (can add/remove/modify routes)
- Use cases: Add virtual host-level configuration, inject additional routes
HTTPListener Hook (PostHTTPListenerModifyHook
)
- When called: After listener is fully configured with all virtual hosts
- Input: Complete HTTP listener and associated extension policies
- Output: Modified listener
- Use cases: Add listener filters, modify listener configuration, add authentication
Translation Hook (PostTranslateModifyHook
)
- When called: After all individual resources are generated and processed
- Input: All clusters, secrets, listeners, and routes, plus extension policies
- Output: Complete set of modified resources
- Use cases: Global resource injection, cross-resource modifications, cleanup operations
Configuration
Extensions must specify which hooks they want to use in the extensionManager.hooks
configuration:
extensionManager:
hooks:
xdsTranslator:
post:
- Route # Enable route modification hook
- VirtualHost # Enable virtual host modification hook
- HTTPListener # Enable HTTP listener modification hook
- Cluster # Enable cluster modification hook
- Translation # Enable global translation hook
# Configure which resources to include in PostTranslateModifyHook
# Default: true for clusters and secrets (for backward compatibility)
# Default: false for listeners and routes (for backward compatibility)
translation:
listener:
includeAll: true
route:
includeAll: true
This task sets up an example extension server that adds the Envoy Proxy Basic Authentication HTTP filter to all the listeners generated by Envoy Gateway. The example extension server includes its own CRD which allows defining username/password pairs that will be accepted by the Envoy Proxy.
Note: Envoy Gateway supports adding Basic Authentication to routes using a SecurityPolicy. See this task for the preferred way to configure Basic Authentication.
Quickstart
Prerequisites
Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Expand for instructions
Install the Gateway API CRDs and Envoy Gateway using Helm:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.0.0-latest -n envoy-gateway-system --create-namespace
Install the GatewayClass, Gateway, HTTPRoute and example app:
kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default
Verify Connectivity:
Get the External IP of the Gateway:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Curl the example app through Envoy proxy:
curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/get
The above command should succeed with status code 200.
Get the name of the Envoy service created the by the example Gateway:
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
Get the deployment of the Envoy service created the by the example Gateway:
export ENVOY_DEPLOYMENT=$(kubectl get deploy -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
Port forward to the Envoy service:
kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8888:80 &
Curl the example app through Envoy proxy:
curl --verbose --header "Host: www.example.com" http://localhost:8888/get
The above command should succeed with status code 200.
Build and run the example Extension Server
Build and deploy the example extension server in the examples/extension-server
folder into the cluster
running Envoy Gateway.
Build the extension server image
Note: The provided
Makefile
builds an image with the nameextension-server:latest
. You may need to create a different tag for it in order to allow Kubernetes to pull it correctly.make image
Publish the extension server image in your docker repository
kind load docker-image --name envoy-gateway extension-server:latest
docker tag extension-server:latest $YOUR_DOCKER_REPO docker push $YOUR_DOCKER_REPO
Deploy the extension server in your cluster
If you are using your own docker image repository, make sure to update the
values.yaml
with the correct image name and tag.helm install -n envoy-gateway-system extension-server ./examples/extension-server/charts/extension-server
Configure Envoy Gateway
Grant Envoy Gateway’s
ServiceAccount
permission to access the extension server’s CRDkubectl create clusterrole listener-context-example-status-update \ --verb=update \ --resource=ListenerContextExample/status kubectl create clusterrole listener-context-example-viewer \ --verb=get,list,watch \ --resource=ListenerContextExample kubectl create clusterrolebinding envoy-gateway-listener-context \ --clusterrole=listener-context-example-viewer \ --serviceaccount=envoy-gateway-system:envoy-gateway kubectl create clusterrolebinding envoy-gateway-listener-context-status \ --clusterrole=listener-context-example-status-update \ --serviceaccount=envoy-gateway-system:envoy-gateway
Configure Envoy Gateway to use the Extension Server
Add the following fragment to Envoy Gateway’s configmap:
cat <<EOF | kubectl apply -f - apiVersion: v1 kind: ConfigMap metadata: name: envoy-gateway-config namespace: envoy-gateway-system data: envoy-gateway.yaml: | apiVersion: gateway.envoyproxy.io/v1alpha1 kind: EnvoyGateway provider: type: Kubernetes gateway: controllerName: gateway.envoyproxy.io/gatewayclass-controller extensionManager: # Envoy Gateway will watch these resource kinds and use them as extension policies # which can be attached to Gateway resources. policyResources: - group: example.extensions.io version: v1alpha1 kind: ListenerContextExample hooks: # The type of hooks that should be invoked xdsTranslator: post: - HTTPListener service: # The service that is hosting the extension server fqdn: hostname: extension-server.envoy-gateway-system.svc.cluster.local port: 5005 EOF
After updating Envoy Gateway’s configmap, restart Envoy Gateway.
Testing
Get the Gateway’s address:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
The extension server adds the Basic Authentication HTTP filter to all listeners configured by Envoy Gateway. Initially there are no valid user/password combinations available. Accessing the example backend should fail with a 401 status:
$ curl -v --header "Host: www.example.com" "http://${GATEWAY_HOST}/example"
...
> GET /example HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< www-authenticate: Basic realm="http://www.example.com/example"
< content-length: 58
< content-type: text/plain
< date: Mon, 08 Jul 2024 10:53:11 GMT
<
...
User authentication failed. Missing username and password.
...
Add a new Username/Password combination using the example extension server’s CRD:
kubectl apply -f - << EOF
apiVersion: example.extensions.io/v1alpha1
kind: ListenerContextExample
metadata:
name: listeneruser
spec:
targetRefs:
- kind: Gateway
name: eg
group: gateway.networking.k8s.io
username: user
password: p@ssw0rd
EOF
Authenticating with this user/password combination will now work.
$ curl -v http://${GATEWAY_HOST}/example -H "Host: www.example.com" --user 'user:p@ssw0rd'
...
> GET /example HTTP/1.1
> Host: www.example.com
> Authorization: Basic dXNlcm5hbWU6cEBzc3cwcmQ=
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Mon, 08 Jul 2024 10:56:17 GMT
< content-length: 559
<
...
"headers": {
"Authorization": [
"Basic dXNlcm5hbWU6cEBzc3cwcmQ="
],
"X-Example-Ext": [
"user"
],
...
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.