Zone Aware Routing

EnvoyGateway makes use of Envoy Zone Aware Routing to keep network traffic in the originating zone. Preferring same-zone traffic between Pods in your cluster can help with reliability, performance (network latency and throughput), or cost.

Zone-aware routing may be enabled in one of two ways:

1. Configuring a BackendTrafficPolicy with the loadbalancer.zoneAware field
2. Configuring a backendRef Kubernetes Service with Traffic Distribution or Topology Aware Routing

When both a backendRef and a BackendTrafficPolicy include a configuration for zone awareness, the BackendTrafficPolicy takes precedence.

Prerequisites

• The Kubernetes cluster’s nodes must indicate topology information via the topology.kubernetes.io/zone well-known label.
• There must be at least two valid topology zones for scheduling.

• Follow the steps below to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

  Expand for instructions

  1. Install the Gateway API CRDs and Envoy Gateway using Helm:

     helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.0.0-latest -n envoy-gateway-system --create-namespace
     

  2. Install the GatewayClass, Gateway, HTTPRoute and example app:

     kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/latest/quickstart.yaml -n default
     

  3. Verify Connectivity:

     • With External LoadBalancer Support
     • Without LoadBalancer Support

     Get the External IP of the Gateway:

     export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
        

     Curl the example app through Envoy proxy:

     curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/get
        

     The above command should succeed with status code 200.

     Get the name of the Envoy service created the by the example Gateway:

     export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
        

     Get the deployment of the Envoy service created the by the example Gateway:

     export ENVOY_DEPLOYMENT=$(kubectl get deploy -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
        

     Port forward to the Envoy service:

     kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8888:80 &
        

     Curl the example app through Envoy proxy:

     curl --verbose --header "Host: www.example.com" http://localhost:8888/get
        

     The above command should succeed with status code 200.

Configuration

Choose one of the following configuration options.

Option 1: Kubernetes Service

Create the example Kubernetes Service with either topology aware routing or traffic distribution enabled.

Topology Aware Routing

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.kubernetes.io/topology-mode: Auto
  name: zone-aware-routing-backend
  labels:
    app: zone-aware-routing-backend
    service: zone-aware-routing-backend
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: zone-aware-routing-backend
EOF

---
apiVersion: v1
kind: Service
metadata:
  annotations:
    service.kubernetes.io/topology-mode: Auto
  name: zone-aware-routing-backend
  labels:
    app: zone-aware-routing-backend
    service: zone-aware-routing-backend
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: zone-aware-routing-backend

Traffic Distribution

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
  name: zone-aware-routing-backend
  labels:
    app: zone-aware-routing-backend
    service: zone-aware-routing-backend
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: zone-aware-routing-backend
  trafficDistribution: PreferClose
EOF

---
apiVersion: v1
kind: Service
metadata:
  name: zone-aware-routing-backend
  labels:
    app: zone-aware-routing-backend
    service: zone-aware-routing-backend
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: zone-aware-routing-backend
  trafficDistribution: PreferClose

Option 2: BackendTrafficPolicy

Zone aware routing can also be enabled directly with a BackendTrafficPolicy. The example below configures similar behavior to Kubernetes Traffic Distribution and forces all traffic to the local zone via the force field instead of Envoy’s default behavior which prefers routing locally as much as possible while still achieving overall equal request distribution across all endpoints.

• Apply from stdin
• Apply from file

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: zone-aware-routing
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: zone-aware-routing
  loadBalancer:
    type: RoundRobin
    zoneAware:
      preferLocal: # Enables Envoy to prefer local zone endpoints while maintaining overall traffic balance across zones
        minEndpointsThreshold: 1 # Zone-aware routing is disabled if total number of endpoints is less than this threshold
        force: # Forces all traffic to stay within the local zone, regardless of upstream endpoint distribution between zones
          minEndpointsInZoneThreshold: 1  # If fewer local endpoints exist than this threshold, fallback to standard zone-aware routing behavior
EOF

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
  name: zone-aware-routing
spec:
  targetRefs:
  - group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: zone-aware-routing
  loadBalancer:
    type: RoundRobin
    zoneAware:
      preferLocal: # Enables Envoy to prefer local zone endpoints while maintaining overall traffic balance across zones
        minEndpointsThreshold: 1 # Zone-aware routing is disabled if total number of endpoints is less than this threshold
        force: # Forces all traffic to stay within the local zone, regardless of upstream endpoint distribution between zones
          minEndpointsInZoneThreshold: 1  # If fewer local endpoints exist than this threshold, fallback to standard zone-aware routing behavior

Example deployments and HTTPRoute

Next apply the example manifests to create two Deployments and an HTTPRoute. For the test configuration one Deployment (zone-aware-routing-backend-local) includes affinity for EnvoyProxy Pods to ensure its Pods are scheduled in the same zone and the second Deployment (zone-aware-routing-backend-nonlocal) uses anti-affinity to ensure its Pods don’t schedule to the same zone in order to demonstrate functionality.

kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/zone-aware-routing.yaml -n default

Ensure that both example Deployments are marked as ready and produces the following output:

kubectl get deployment/zone-aware-routing-backend-local deployment/zone-aware-routing-backend-nonlocal -n default
NAME                            READY   UP-TO-DATE   AVAILABLE   AGE
zone-aware-routing-backend-local      3/3     3            3           9m1s
zone-aware-routing-backend-nonlocal   3/3     3            3           9m1s

An HTTPRoute resource is created for the /zone-aware-routing path prefix along with two example Deployment resources that are respectively configured with pod affinity/anti-affinity targeting the Envoy Proxy Pods for testing.

Verify the HTTPRoute configuration and status:

kubectl get httproute/zone-aware-routing -o yaml

If used during configuration, verify the BackendTrafficPolicy:

kubectl get backendtrafficpolicy/zone-aware-routing -o yaml

Testing

Ensure the GATEWAY_HOST environment variable from the Quickstart is set. If not, follow the Quickstart instructions to set the variable.

echo $GATEWAY_HOST

HTTPRoute

We will now send a request and expect to be routed to backend in the same local zone as the Envoy Proxy Pods.

curl -H "Host: www.example.com" "http://${GATEWAY_HOST}/zone-aware-routing" -XPOST -d '{}'

We will see the following output. The pod header identifies which backend received the request and should have a name prefix of zone-aware-routing-backend-local-. We should see this behavior every time and the nonlocal backend should not receive requests.

{
  "path": "/zone-aware-routing",
  "host": "www.example.com",
  "method": "GET",
  "proto": "HTTP/1.1",
  "headers": {
    "Accept": [
      "*/*"
    ],
    "User-Agent": [
      "curl/8.7.1"
    ],
    "X-Envoy-External-Address": [
      "127.0.0.1"
    ],
    "X-Forwarded-For": [
      "10.244.1.5"
    ],
    "X-Forwarded-Proto": [
      "http"
    ],
    "X-Request-Id": [
      "0f4c5d28-52d0-4727-881f-abf2ac12b3b7"
    ]
  },
  "namespace": "default",
  "ingress": "",
  "service": "",
  "pod": "zone-aware-routing-backend-local-5586cd668d-md8sf"
}

Clean-Up

Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.

Delete the zone-aware-routing example resources and HTTPRoute:

kubectl delete service/zone-aware-routing-backend 
kubectl delete deployment/zone-aware-routing-backend-local
kubectl delete deployment/zone-aware-routing-backend-nonlocal
kubectl delete httproute/zone-aware-routing
kubectl delete backendtrafficpolicy/zone-aware-routing