This is the multi-page printable view of this section. Click here to print.
User Guides
- 1: Quickstart
- 2: CORS
- 3: Customize EnvoyProxy
- 4: Deployment Mode
- 5: Envoy Patch Policy
- 6: Gateway Address
- 7: Gateway API Metrics
- 8: Gateway API Support
- 9: GRPC Routing
- 10: HTTP Redirects
- 11: HTTP Request Headers
- 12: HTTP Response Headers
- 13: HTTP Routing
- 14: HTTP URL Rewrite
- 15: HTTPRoute Request Mirroring
- 16: HTTPRoute Traffic Splitting
- 17: JWT Authentication
- 18: Multicluster Service Routing
- 19: Proxy Observability
- 20: Rate Limit
- 21: Secure Gateways
- 22: TCP Routing
- 23: TLS Passthrough
- 24: TLS Termination for TCP
- 25: UDP Routing
- 26: Use egctl
- 27: Using cert-manager For TLS Termination
- 28: Visualising metrics using Grafana
1 - Quickstart
This guide will help you get started with Envoy Gateway in a few simple steps.
Prerequisites
A Kubernetes cluster.
Note: Refer to the Compatibility Matrix for supported Kubernetes versions.
Note: In case your Kubernetes cluster, does not have a LoadBalancer implementation, we recommend installing one
so the Gateway resource has an Address associated with it. We recommend using MetalLB.
Installation
Install the Gateway API CRDs and Envoy Gateway:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.6.0 -n envoy-gateway-system --create-namespace
Wait for Envoy Gateway to become available:
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
Install the GatewayClass, Gateway, HTTPRoute and example app:
kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/v0.6.0/quickstart.yaml -n default
Note: quickstart.yaml defines that Envoy Gateway will listen for
traffic on port 80 on its globally-routable IP address, to make it easy to use
browsers to test Envoy Gateway. When Envoy Gateway sees that its Listener is
using a privileged port (<1024), it will map this internally to an
unprivileged port, so that Envoy Gateway doesn’t need additional privileges.
It’s important to be aware of this mapping, since you may need to take it into
consideration when debugging.
Testing the Configuration
Get the name of the Envoy service created the by the example Gateway:
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
Port forward to the Envoy service:
kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8888:80 &
Curl the example app through Envoy proxy:
curl --verbose --header "Host: www.example.com" http://localhost:8888/get
External LoadBalancer Support
You can also test the same functionality by sending traffic to the External IP. To get the external IP of the Envoy service, run:
export GATEWAY_HOST=$(kubectl get svc/${ENVOY_SERVICE} -n envoy-gateway-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
In certain environments, the load balancer may be exposed using a hostname, instead of an IP address. If so, replace
ip in the above command with hostname.
Curl the example app through Envoy proxy:
curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/get
Clean-Up
Use the steps in this section to uninstall everything from the quickstart guide.
Delete the GatewayClass, Gateway, HTTPRoute and Example App:
kubectl delete -f https://github.com/envoyproxy/gateway/releases/download/v0.6.0/quickstart.yaml --ignore-not-found=true
Delete the Gateway API CRDs and Envoy Gateway:
helm uninstall eg -n envoy-gateway-system
Next Steps
Checkout the Developer Guide to get involved in the project.
2 - CORS
This guide provides instructions for configuring Cross-Origin Resource Sharing (CORS) on Envoy Gateway. CORS defines a way for client web applications that are loaded in one domain to interact with resources in a different domain.
Envoy Gateway introduces a new CRD called SecurityPolicy that allows the user to configure CORS. This instantiated resource can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Prerequisites
Follow the steps from the Quickstart guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Configuration
The below example defines a SecurityPolicy that allows CORS requests from www.foo.com.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
name: cors-example
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: backend
cors:
allowOrigins:
- type: Exact
value: "www.foo.com"
allowMethods:
- GET
- POST
allowHeaders:
- "x-header-1"
- "x-header-2"
exposeHeaders:
- "x-header-3"
- "x-header-4"
EOF
Verify the SecurityPolicy configuration:
kubectl get securitypolicy/cors-example -o yaml
Testing
Ensure the GATEWAY_HOST environment variable from the Quickstart guide is set. If not, follow the
Quickstart instructions to set the variable.
echo $GATEWAY_HOST
Verify that the CORS headers are present in the response of the OPTIONS request from http://www.foo.com:
curl -H "Origin: http://www.foo.com" \
-H "Host: www.example.com" \
-H "Access-Control-Request-Method: GET" \
-X OPTIONS -v -s \
http://$GATEWAY_HOST \
1> /dev/null
You should see the below response, indicating that the request from http://www.foo.com is allowed:
< access-control-allow-origin: http://www.foo.com
< access-control-allow-methods: GET, POST
< access-control-allow-headers: x-header-1, x-header-2
< access-control-max-age: 86400
< access-control-expose-headers: x-header-3, x-header-4
If you try to send a request from http://www.bar.com, you should see the below response:
curl -H "Origin: http://www.bar.com" \
-H "Host: www.example.com" \
-H "Access-Control-Request-Method: GET" \
-X OPTIONS -v -s \
http://$GATEWAY_HOST \
1> /dev/null
You won’t see any CORS headers in the response, indicating that the request from http://www.bar.com was not allowed.
Note: CORS specification requires that the browsers to send a preflight request to the server to ask if it’s allowed to access the limited resource in another domains. The browsers are supposed to follow the response from the server to determine whether to send the actual request or not. The CORS filter only response to the preflight requests according to its configuration. It won’t deny any requests. The browsers are responsible for enforcing the CORS policy.
Clean-Up
Follow the steps from the Quickstart guide to uninstall Envoy Gateway and the example manifest.
Delete the SecurityPolicy:
kubectl delete securitypolicy/cors-example
Next Steps
Checkout the Developer Guide to get involved in the project.
3 - Customize EnvoyProxy
Envoy Gateway provides an EnvoyProxy CRD that can be linked to the ParametersRef in GatewayClass, allowing cluster admins to customize the managed EnvoyProxy Deployment and Service. To learn more about GatewayClass and ParametersRef, please refer to Gateway API documentation.
Installation
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Add GatewayClass ParametersRef
First, you need to add ParametersRef in GatewayClass, and refer to EnvoyProxy Config:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: custom-proxy-config
namespace: envoy-gateway-system
EOF
Customize EnvoyProxy Deployment Replicas
You can customize the EnvoyProxy Deployment Replicas via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
replicas: 2
EOF
After you apply the config, you should see the replicas of envoyproxy changes to 2. And also you can dynamically change the value.
kubectl get deployment -l gateway.envoyproxy.io/owning-gateway-name=eg
Customize EnvoyProxy Image
You can customize the EnvoyProxy Image via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
image: envoyproxy/envoy:v1.25-v0.6.0
EOF
After applying the config, you can get the deployment image, and see it has changed.
Customize EnvoyProxy Pod Annotations
You can customize the EnvoyProxy Pod Annotations via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
pod:
annotations:
custom1: deploy-annotation1
custom2: deploy-annotation2
EOF
After applying the config, you can get the envoyproxy pods, and see new annotations has been added.
Customize EnvoyProxy Deployment Resources
You can customize the EnvoyProxy Deployment Resources via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
resources:
requests:
cpu: 150m
memory: 640Mi
limits:
cpu: 500m
memory: 1Gi
EOF
Customize EnvoyProxy Deployment Env
You can customize the EnvoyProxy Deployment Env via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
env:
- name: env_a
value: env_a_value
- name: env_b
value: env_b_value
EOF
Envoy Gateway has provided two initial
envENVOY_GATEWAY_NAMESPACEandENVOY_POD_NAMEfor envoyproxy container.
After applying the config, you can get the envoyproxy deployment, and see resources has been changed.
Customize EnvoyProxy Deployment Volumes or VolumeMounts
You can customize the EnvoyProxy Deployment Volumes or VolumeMounts via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
volumeMounts:
- mountPath: /certs
name: certs
readOnly: true
pod:
volumes:
- name: certs
secret:
secretName: envoy-cert
EOF
After applying the config, you can get the envoyproxy deployment, and see resources has been changed.
Customize EnvoyProxy Service Annotations
You can customize the EnvoyProxy Service Annotations via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
provider:
type: Kubernetes
kubernetes:
envoyService:
annotations:
custom1: svc-annotation1
custom2: svc-annotation2
EOF
After applying the config, you can get the envoyproxy service, and see annotations has been added.
Customize EnvoyProxy Bootstrap Config
You can customize the EnvoyProxy bootstrap config via EnvoyProxy Config. There are two ways to customize it:
- Replace: the whole bootstrap config will be replaced by the config you provided.
- Merge: the config you provided will be merged into the default bootstrap config.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
bootstrap:
type: Replace
value: |
admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 20000
dynamic_resources:
ads_config:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
ads: {}
resource_api_version: V3
cds_config:
ads: {}
resource_api_version: V3
static_resources:
clusters:
- connect_timeout: 10s
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
typed_extension_protocol_options:
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
"explicit_http_config":
"http2_protocol_options": {}
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: "/sds/xds-certificate.json"
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: "/sds/xds-trusted-ca.json"
resource_api_version: V3
layered_runtime:
layers:
- name: runtime-0
rtds_layer:
rtds_config:
ads: {}
resource_api_version: V3
name: runtime-0
EOF
You can use egctl translate to get the default xDS Bootstrap configuration used by Envoy Gateway.
After applying the config, the bootstrap config will be overridden by the new config you provided.
Any errors in the configuration will be surfaced as status within the GatewayClass resource.
You can also validate this configuration using egctl translate.
4 - Deployment Mode
One GatewayClass per Envoy Gateway
- Envoy Gateway can accept a single GatewayClass resource. If you’ve instantiated multiple GatewayClasses, we recommend running multiple Envoy Gateway controllers in different namespaces, linking a GatewayClass to each of them.
- Support for accepting multiple GatewayClass is being tracked here.
Supported Modes
Kubernetes
- The default deployment model is - Envoy Gateway watches for resources such a
Service&HTTPRoutein all namespaces and creates managed data plane resources such as EnvoyProxyDeploymentin the namespace where Envoy Gateway is running. - Envoy Gateway also supports Namespaced deployment mode, you can watch resources in the specific namespaces by assigning
EnvoyGateway.provider.kubernetes.watch.namespacesand creates managed data plane resources in the namespace where Envoy Gateway is running. - Support for alternate deployment modes is being tracked here.
Multi-tenancy
Kubernetes
A
tenantis a group within an organization (e.g. a team or department) who shares organizational resources. We recommend eachtenantdeploy their own Envoy Gateway controller in their respectivenamespace. Below is an example of deploying Envoy Gateway by themarketingandproductteams in separate namespaces.Lets deploy Envoy Gateway in the
marketingnamespace and also watch resources only in this namespace. We are also setting the controller name to a unique string heregateway.envoyproxy.io/marketing-gatewayclass-controller.
helm install --set config.envoyGateway.gateway.controllerName=gateway.envoyproxy.io/marketing-gatewayclass-controller --set config.envoyGateway.provider.kubernetes.watch.namespaces={marketing} eg-marketing oci://docker.io/envoyproxy/gateway-helm --version v0.6.0 -n marketing --create-namespace
Lets create a GatewayClass linked to the marketing team’s Envoy Gateway controller, and as well other resources linked to it, so the backend application operated by this team can be exposed to external clients.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg-marketing
spec:
controllerName: gateway.envoyproxy.io/marketing-gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: marketing
spec:
gatewayClassName: eg-marketing
listeners:
- name: http
protocol: HTTP
port: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend
namespace: marketing
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: marketing
labels:
app: backend
service: backend
spec:
ports:
- name: http
port: 3000
targetPort: 3000
selector:
app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
namespace: marketing
spec:
replicas: 1
selector:
matchLabels:
app: backend
version: v1
template:
metadata:
labels:
app: backend
version: v1
spec:
serviceAccountName: backend
containers:
- image: gcr.io/k8s-staging-ingressconformance/echoserver:v20221109-7ee2f3e
imagePullPolicy: IfNotPresent
name: backend
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: marketing
spec:
parentRefs:
- name: eg
hostnames:
- "www.marketing.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
Lets port forward to the generated envoy proxy service in the marketing namespace and send a request to it.
export ENVOY_SERVICE=$(kubectl get svc -n marketing --selector=gateway.envoyproxy.io/owning-gateway-namespace=marketing,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
kubectl -n marketing port-forward service/${ENVOY_SERVICE} 8888:8080 &
curl --verbose --header "Host: www.marketing.example.com" http://localhost:8888/get
* Trying 127.0.0.1:8888...
* Connected to localhost (127.0.0.1) port 8888 (#0)
> GET /get HTTP/1.1
> Host: www.marketing.example.com
> User-Agent: curl/7.86.0
> Accept: */*
>
Handling connection for 8888
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Thu, 20 Apr 2023 19:19:42 GMT
< content-length: 521
< x-envoy-upstream-service-time: 0
< server: envoy
<
{
"path": "/get",
"host": "www.marketing.example.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.86.0"
],
"X-Envoy-Expected-Rq-Timeout-Ms": [
"15000"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"10.1.0.157"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"c637977c-458a-48ae-92b3-f8c429849322"
]
},
"namespace": "marketing",
"ingress": "",
"service": "",
"pod": "backend-74888f465f-bcs8f"
* Connection #0 to host localhost left intact
- Lets deploy Envoy Gateway in the
productnamespace and also watch resources only in this namespace.
helm install --set config.envoyGateway.gateway.controllerName=gateway.envoyproxy.io/product-gatewayclass-controller --set config.envoyGateway.provider.kubernetes.watch.namespaces={product} eg-product oci://docker.io/envoyproxy/gateway-helm --version v0.6.0 -n product --create-namespace
Lets create a GatewayClass linked to the product team’s Envoy Gateway controller, and as well other resources linked to it, so the backend application operated by this team can be exposed to external clients.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg-product
spec:
controllerName: gateway.envoyproxy.io/product-gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: product
spec:
gatewayClassName: eg-product
listeners:
- name: http
protocol: HTTP
port: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend
namespace: product
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: product
labels:
app: backend
service: backend
spec:
ports:
- name: http
port: 3000
targetPort: 3000
selector:
app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
namespace: product
spec:
replicas: 1
selector:
matchLabels:
app: backend
version: v1
template:
metadata:
labels:
app: backend
version: v1
spec:
serviceAccountName: backend
containers:
- image: gcr.io/k8s-staging-ingressconformance/echoserver:v20221109-7ee2f3e
imagePullPolicy: IfNotPresent
name: backend
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: product
spec:
parentRefs:
- name: eg
hostnames:
- "www.product.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
Lets port forward to the generated envoy proxy service in the product namespace and send a request to it.
export ENVOY_SERVICE=$(kubectl get svc -n product --selector=gateway.envoyproxy.io/owning-gateway-namespace=product,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
kubectl -n product port-forward service/${ENVOY_SERVICE} 8889:8080 &
curl --verbose --header "Host: www.product.example.com" http://localhost:8889/get
* Trying 127.0.0.1:8889...
* Connected to localhost (127.0.0.1) port 8889 (#0)
> GET /get HTTP/1.1
> Host: www.product.example.com
> User-Agent: curl/7.86.0
> Accept: */*
>
Handling connection for 8889
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Thu, 20 Apr 2023 19:20:17 GMT
< content-length: 517
< x-envoy-upstream-service-time: 0
< server: envoy
<
{
"path": "/get",
"host": "www.product.example.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.86.0"
],
"X-Envoy-Expected-Rq-Timeout-Ms": [
"15000"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"10.1.0.156"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"39196453-2250-4331-b756-54003b2853c2"
]
},
"namespace": "product",
"ingress": "",
"service": "",
"pod": "backend-74888f465f-64fjs"
* Connection #0 to host localhost left intact
With the below command you can ensure that you are not able to access the marketing team’s backend exposed using the www.marketing.example.com hostname
and the product team’s data plane.
curl --verbose --header "Host: www.marketing.example.com" http://localhost:8889/get
* Trying 127.0.0.1:8889...
* Connected to localhost (127.0.0.1) port 8889 (#0)
> GET /get HTTP/1.1
> Host: www.marketing.example.com
> User-Agent: curl/7.86.0
> Accept: */*
>
Handling connection for 8889
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< date: Thu, 20 Apr 2023 19:22:13 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host localhost left intact
5 - Envoy Patch Policy
This guide explains the usage of the EnvoyPatchPolicy API. Note: This API is meant for users extremely familiar with Envoy xDS semantics. Also before considering this API for production use cases, please be aware that this API is unstable and the outcome may change across versions. Use at your own risk.
Introduction
The EnvoyPatchPolicy API allows user to modify the output xDS configuration generated by Envoy Gateway intended for EnvoyProxy, using JSON Patch semantics.
Motivation
This API was introduced to allow advanced users to be able to leverage Envoy Proxy functionality not exposed by Envoy Gateway APIs today.
Quickstart
Prerequisites
- Follow the steps from the Quickstart guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Enable EnvoyPatchPolicy
By default EnvoyPatchPolicy is disabled. Lets enable it in the EnvoyGateway startup configuration
The default installation of Envoy Gateway installs a default EnvoyGateway configuration and attaches it using a
ConfigMap. In the next step, we will update this resource to enable EnvoyPatchPolicy.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: envoy-gateway-config
namespace: envoy-gateway-system
data:
envoy-gateway.yaml: |
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
provider:
type: Kubernetes
gateway:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
extensionApis:
enableEnvoyPatchPolicy: true
EOF
- After updating the
ConfigMap, you will need to restart theenvoy-gatewaydeployment so the configuration kicks in
kubectl rollout restart deployment envoy-gateway -n envoy-gateway-system
Testing
Customize Response
Lets use EnvoyProxy’s Local Reply Modification feature to return a custom response back to the client when the status code is
404Lets apply the configuration
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
name: custom-response-patch-policy
namespace: default
spec:
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: eg
namespace: default
type: JSONPatch
jsonPatches:
- type: "type.googleapis.com/envoy.config.listener.v3.Listener"
# The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
name: default/eg/http
operation:
op: add
path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
value:
mappers:
- filter:
status_code_filter:
comparison:
op: EQ
value:
default_value: 404
runtime_key: key_b
status_code: 406
body:
inline_string: "could not find what you are looking for"
EOF
- Lets edit the HTTPRoute resource from the Quickstart to only match on paths with value
/get
kubectl patch httproute backend --type=json --patch '[{
"op": "add",
"path": "/spec/rules/0/matches/0/path/value",
"value": "/get",
}]'
- Lets test it out by specifying a path apart from
/get
$ curl --header "Host: www.example.com" http://localhost:8888/find
Handling connection for 8888
could not find what you are looking for
Debugging
Runtime
- The
Statussubresource should have information about the status of the resource. Make sureAccepted=TrueandProgrammed=Trueconditions are set to ensure that the policy has been applied to Envoy Proxy.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"gateway.envoyproxy.io/v1alpha1","kind":"EnvoyPatchPolicy","metadata":{"annotations":{},"name":"custom-response-patch-policy","namespace":"default"},"spec":{"jsonPatches":[{"name":"default/eg/http","operation":{"op":"add","path":"/default_filter_chain/filters/0/typed_config/local_reply_config","value":{"mappers":[{"body":{"inline_string":"could not find what you are looking for"},"filter":{"status_code_filter":{"comparison":{"op":"EQ","value":{"default_value":404}}}}}]}},"type":"type.googleapis.com/envoy.config.listener.v3.Listener"}],"priority":0,"targetRef":{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"eg","namespace":"default"},"type":"JSONPatch"}}
creationTimestamp: "2023-07-31T21:47:53Z"
generation: 1
name: custom-response-patch-policy
namespace: default
resourceVersion: "10265"
uid: a35bda6e-a0cc-46d7-a63a-cee765174bc3
spec:
jsonPatches:
- name: default/eg/http
operation:
op: add
path: /default_filter_chain/filters/0/typed_config/local_reply_config
value:
mappers:
- body:
inline_string: could not find what you are looking for
filter:
status_code_filter:
comparison:
op: EQ
value:
default_value: 404
type: type.googleapis.com/envoy.config.listener.v3.Listener
priority: 0
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: eg
namespace: default
type: JSONPatch
status:
conditions:
- lastTransitionTime: "2023-07-31T21:48:19Z"
message: EnvoyPatchPolicy has been accepted.
observedGeneration: 1
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2023-07-31T21:48:19Z"
message: successfully applied patches.
reason: Programmed
status: "True"
type: Programmed
Offline
- You can use egctl x translate to validate the translated xds output.
Caveats
This API will always be an unstable API and the same outcome cannot be guaranteed across versions for these reasons
- The Envoy Proxy API might deprecate and remove API fields
- Envoy Gateway might alter the xDS translation creating a different xDS output
such as changing the
namefield of resources.
6 - Gateway Address
The Gateway API provides an optional Addresses field through which Envoy Gateway can set addresses for Envoy Proxy Service. The currently supported addresses are:
Installation
Install Envoy Gateway:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.6.0 -n envoy-gateway-system --create-namespace
Wait for Envoy Gateway to become available:
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
External IPs
Using the addresses in Gateway.Spec.Addresses as the External IPs of Envoy Proxy Service, this will require the address to be of type IPAddress.
Install the GatewayClass, Gateway from quickstart:
kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/v0.6.0/quickstart.yaml -n default
Set the address of the Gateway, the address settings here are for reference only:
kubectl patch gateway eg --type=json --patch '[{
"op": "add",
"path": "/spec/addresses",
"value": [{
"type": "IPAddress",
"value": "1.2.3.4"
}]
}]'
Verify the Gateway status:
kubectl get gateway
NAME CLASS ADDRESS PROGRAMMED AGE
eg eg 1.2.3.4 True 14m
Verify the Envoy Proxy Service status:
kubectl get service -n envoy-gateway-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy-default-eg-64656661 LoadBalancer 10.96.236.219 1.2.3.4 80:31017/TCP 15m
envoy-gateway ClusterIP 10.96.192.76 <none> 18000/TCP 15m
envoy-gateway-metrics-service ClusterIP 10.96.124.73 <none> 8443/TCP 15m
Note: If the Gateway.Spec.Addresses is explicitly set, it will be the only addresses that populates the Gateway status.
7 - Gateway API Metrics
Resource metrics for Gateway API objects are available using the Gateway API State Metrics project. The project also provides example dashboard for visualising the metrics using Grafana, and example alerts using Prometheus & Alertmanager.
Prerequisites
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Run the following commands to install the metrics stack, with the Gateway API State Metrics configuration, on your kubernetes cluster:
kubectl apply --server-side -f https://raw.githubusercontent.com/Kuadrant/gateway-api-state-metrics/main/config/examples/kube-prometheus/bundle_crd.yaml
kubectl apply -f https://raw.githubusercontent.com/Kuadrant/gateway-api-state-metrics/main/config/examples/kube-prometheus/bundle.yaml
Metrics and Alerts
To access the Prometheus UI, wait for the statefulset to be ready, then use the port-forward command:
# This first command may fail if the statefulset has not been created yet.
# In that case, try again until you get a message like 'Waiting for 2 pods to be ready...'
# or 'statefulset rolling update complete 2 pods...'
kubectl -n monitoring rollout status --watch --timeout=5m statefulset/prometheus-k8s
kubectl -n monitoring port-forward service/prometheus-k8s 9090:9090 > /dev/null &
Navigate to http://localhost:9090.
Metrics can be queried from the ‘Graph’ tab e.g. gatewayapi_gateway_created
See the Gateway API State Metrics README for the full list of Gateway API metrics available.
Alerts can be seen in the ‘Alerts’ tab. Gateway API specific alerts will be grouped under the ‘gateway-api.rules’ heading.
Note: Alerts are defined in a PrometheusRules custom resource in the ‘monitoring’ namespace. You can modify the alert rules by updating this resource.
Dashboards
To view the dashboards in Grafana, wait for the deployment to be ready, then use the port-forward command:
kubectl -n monitoring wait --timeout=5m deployment/grafana --for=condition=Available
kubectl -n monitoring port-forward service/grafana 3000:3000 > /dev/null &
Navigate to http://localhost:3000 and sign in with admin/admin.
The Gateway API State dashboards will be available in the ‘Default’ folder and tagged with ‘gateway-api’.
See the Gateway API State Metrics README for further information on available dashboards.
Note: Dashboards are loaded from configmaps. You can modify the dashboards in the Grafana UI, however you will need to export them from the UI and update the json in the configmaps to persist changes.
8 - Gateway API Support
As mentioned in the system design document, Envoy Gateway’s managed data plane is configured dynamically through Kubernetes resources, primarily Gateway API objects. Envoy Gateway supports configuration using the following Gateway API resources.
GatewayClass
A GatewayClass represents a “class” of gateways, i.e. which Gateways should be managed by Envoy Gateway.
Envoy Gateway supports managing a single GatewayClass resource that matches its configured controllerName and
follows Gateway API guidelines for resolving conflicts when multiple GatewayClasses exist with a matching
controllerName.
Note: If specifying GatewayClass parameters reference, it must refer to an EnvoyProxy resource.
Gateway
When a Gateway resource is created that references the managed GatewayClass, Envoy Gateway will create and manage a new Envoy Proxy deployment. Gateway API resources that reference this Gateway will configure this managed Envoy Proxy deployment.
HTTPRoute
An HTTPRoute configures routing of HTTP traffic through one or more Gateways. The following HTTPRoute filters are supported by Envoy Gateway:
requestHeaderModifier: RequestHeaderModifiers can be used to modify or add request headers before the request is proxied to its destination.responseHeaderModifier: ResponseHeaderModifiers can be used to modify or add response headers before the response is sent back to the client.requestMirror: RequestMirrors configure destinations where the requests should also be mirrored to. Responses to mirrored requests will be ignored.requestRedirect: RequestRedirects configure policied for how requests that match the HTTPRoute should be modified and then redirected.urlRewrite: UrlRewrites allow for modification of the request’s hostname and path before it is proxied to its destination.extensionRef: ExtensionRefs are used by Envoy Gateway to implement extended filters. Currently, Envoy Gateway supports rate limiting and request authentication filters. For more information about these filters, refer to the rate limiting and request authentication documentation.
Notes:
- The only BackendRef kind supported by Envoy Gateway is a Service. Routing traffic to other destinations such as arbitrary URLs is not possible.
- The
filtersfield within HTTPBackendRef is not supported.
TCPRoute
A TCPRoute configures routing of raw TCP traffic through one or more Gateways. Traffic can be forwarded to the desired BackendRefs based on a TCP port number.
Note: A TCPRoute only supports proxying in non-transparent mode, i.e. the backend will see the source IP and port of the Envoy Proxy instance instead of the client.
UDPRoute
A UDPRoute configures routing of raw UDP traffic through one or more Gateways. Traffic can be forwarded to the desired BackendRefs based on a UDP port number.
Note: Similar to TCPRoutes, UDPRoutes only support proxying in non-transparent mode i.e. the backend will see the source IP and port of the Envoy Proxy instance instead of the client.
GRPCRoute
A GRPCRoute configures routing of gRPC requests through one or more Gateways. They offer request matching by hostname, gRPC service, gRPC method, or HTTP/2 Header. Envoy Gateway supports the following filters on GRPCRoutes to provide additional traffic processing:
requestHeaderModifier: RequestHeaderModifiers can be used to modify or add request headers before the request is proxied to its destination.responseHeaderModifier: ResponseHeaderModifiers can be used to modify or add response headers before the response is sent back to the client.requestMirror: RequestMirrors configure destinations where the requests should also be mirrored to. Responses to mirrored requests will be ignored.
Notes:
- The only BackendRef kind supported by Envoy Gateway is a Service. Routing traffic to other destinations such as arbitrary URLs is not currently possible.
- The
filtersfield within HTTPBackendRef is not supported.
TLSRoute
A TLSRoute configures routing of TCP traffic through one or more Gateways. However, unlike TCPRoutes, TLSRoutes can match against TLS-specific metadata.
ReferenceGrant
A ReferenceGrant is used to allow a resource to reference another resource in a different namespace. Normally an
HTTPRoute created in namespace foo is not allowed to reference a Service in namespace bar. A ReferenceGrant permits
these types of cross-namespace references. Envoy Gateway supports the following ReferenceGrant use-cases:
- Allowing an HTTPRoute, GRPCRoute, TLSRoute, UDPRoute, or TCPRoute to reference a Service in a different namespace.
- Allowing an HTTPRoute’s
requestMirrorfilter to include a BackendRef that references a Service in a different namespace. - Allowing a Gateway’s SecretObjectReference to reference a secret in a different namespace.
9 - GRPC Routing
The GRPCRoute resource allows users to configure gRPC routing by matching HTTP/2 traffic and forwarding it to backend gRPC servers. To learn more about gRPC routing, refer to the Gateway API documentation.
Prerequisites
Install Envoy Gateway:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.6.0 -n envoy-gateway-system --create-namespace
Wait for Envoy Gateway to become available:
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
Installation
Install the gRPC routing example resources:
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/grpc-routing.yaml
The manifest installs a GatewayClass, Gateway, a Deployment, a Service, and a GRPCRoute resource. The GatewayClass is a cluster-scoped resource that represents a class of Gateways that can be instantiated.
Note: Envoy Gateway is configured by default to manage a GatewayClass with
controllerName: gateway.envoyproxy.io/gatewayclass-controller.
Verification
Check the status of the GatewayClass:
kubectl get gc --selector=example=grpc-routing
The status should reflect “Accepted=True”, indicating Envoy Gateway is managing the GatewayClass.
A Gateway represents configuration of infrastructure. When a Gateway is created, Envoy proxy infrastructure is
provisioned or configured by Envoy Gateway. The gatewayClassName defines the name of a GatewayClass used by this
Gateway. Check the status of the Gateway:
kubectl get gateways --selector=example=grpc-routing
The status should reflect “Ready=True”, indicating the Envoy proxy infrastructure has been provisioned. The status also provides the address of the Gateway. This address is used later in the guide to test connectivity to proxied backend services.
Check the status of the GRPCRoute:
kubectl get grpcroutes --selector=example=grpc-routing -o yaml
The status for the GRPCRoute should surface “Accepted=True” and a parentRef that references the example Gateway.
The example-route matches any traffic for “grpc-example.com” and forwards it to the “yages” Service.
Testing the Configuration
Before testing GRPC routing to the yages backend, get the Gateway’s address.
export GATEWAY_HOST=$(kubectl get gateway/example-gateway -o jsonpath='{.status.addresses[0].value}')
Test GRPC routing to the yages backend using the grpcurl command.
grpcurl -plaintext -authority=grpc-example.com ${GATEWAY_HOST}:80 yages.Echo/Ping
You should see the below response
{
"text": "pong"
}
Envoy Gateway also supports gRPC-Web requests for this configuration. The below curl command can be used to send a grpc-Web request with over HTTP/2. You should receive the same response seen in the previous command.
The data in the body AAAAAAA= is a base64 encoded representation of an empty message (data length 0) that the Ping RPC accepts.
curl --http2-prior-knowledge -s ${GATEWAY_HOST}:80/yages.Echo/Ping -H 'Host: grpc-example.com' -H 'Content-Type: application/grpc-web-text' -H 'Accept: application/grpc-web-text' -XPOST -d'AAAAAAA=' | base64 -d
GRPCRoute Match
The matches field can be used to restrict the route to a specific set of requests based on GRPC’s service and/or method names.
It supports two match types: Exact and RegularExpression.
Exact
Exact match is the default match type.
The following example shows how to match a request based on the service and method names for grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo,
as well as a match for all services with a method name Ping which matches yages.Echo/Ping in our deployment.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GRPCRoute
metadata:
name: yages
labels:
example: grpc-routing
spec:
parentRefs:
- name: example-gateway
hostnames:
- "grpc-example.com"
rules:
- matches:
- method:
method: ServerReflectionInfo
service: grpc.reflection.v1alpha.ServerReflection
- method:
method: Ping
backendRefs:
- group: ""
kind: Service
name: yages
port: 9000
weight: 1
EOF
Verify the GRPCRoute status:
kubectl get grpcroutes --selector=example=grpc-routing -o yaml
Test GRPC routing to the yages backend using the grpcurl command.
grpcurl -plaintext -authority=grpc-example.com ${GATEWAY_HOST}:80 yages.Echo/Ping
RegularExpression
The following example shows how to match a request based on the service and method names
with match type RegularExpression. It matches all the services and methods with pattern
/.*.Echo/Pin.+, which matches yages.Echo/Ping in our deployment.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: GRPCRoute
metadata:
name: yages
labels:
example: grpc-routing
spec:
parentRefs:
- name: example-gateway
hostnames:
- "grpc-example.com"
rules:
- matches:
- method:
method: ServerReflectionInfo
service: grpc.reflection.v1alpha.ServerReflection
- method:
method: "Pin.+"
service: ".*.Echo"
type: RegularExpression
backendRefs:
- group: ""
kind: Service
name: yages
port: 9000
weight: 1
EOF
Verify the GRPCRoute status:
kubectl get grpcroutes --selector=example=grpc-routing -o yaml
Test GRPC routing to the yages backend using the grpcurl command.
grpcurl -plaintext -authority=grpc-example.com ${GATEWAY_HOST}:80 yages.Echo/Ping
10 - HTTP Redirects
The HTTPRoute resource can issue redirects to clients or rewrite paths sent upstream using filters. Note that
HTTPRoute rules cannot use both filter types at once. Currently, Envoy Gateway only supports core
HTTPRoute filters which consist of RequestRedirect and RequestHeaderModifier at the time of this writing. To
learn more about HTTP routing, refer to the Gateway API documentation.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTPS.
Redirects
Redirects return HTTP 3XX responses to a client, instructing it to retrieve a different resource. A
RequestRedirect filter instructs Gateways to emit a redirect response to requests that match the rule.
For example, to issue a permanent redirect (301) from HTTP to HTTPS, configure requestRedirect.statusCode=301 and
requestRedirect.scheme="https":
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-to-https-filter-redirect
spec:
parentRefs:
- name: eg
hostnames:
- redirect.example
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
statusCode: 301
hostname: www.example.com
port: 443
backendRefs:
- name: backend
port: 3000
EOF
Note: 301 (default) and 302 are the only supported statusCodes.
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-to-https-filter-redirect -o yaml
Get the Gateway’s address:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Querying redirect.example/get should result in a 301 response from the example Gateway and redirecting to the
configured redirect hostname.
$ curl -L -vvv --header "Host: redirect.example" "http://${GATEWAY_HOST}/get"
...
< HTTP/1.1 301 Moved Permanently
< location: https://www.example.com/get
...
If you followed the steps in the Secure Gateways guide, you should be able to curl the redirect location.
HTTP –> HTTPS
Listeners expose the TLS setting on a per domain or subdomain basis. TLS settings of a listener are applied to all domains that satisfy the hostname criteria.
Create a root certificate and private key to sign certificates:
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/CN=example.com' -keyout CA.key -out CA.crt
openssl req -out example.com.csr -newkey rsa:2048 -nodes -keyout tls.key -subj "/CN=example.com"
Generate a self-signed wildcard certificate for example.com with *.example.com extension
cat <<EOF | openssl x509 -req -days 365 -CA CA.crt -CAkey CA.key -set_serial 0 \
-subj "/CN=example.com" \
-in example.com.csr -out tls.crt -extensions v3_req -extfile -
[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names
[alt_names]
DNS.1 = example.com
DNS.2 = *.example.com
EOF
Create the kubernetes tls secret
kubectl create secret tls example-com --key=tls.key --cert=tls.crt
Define a https listener on the existing gateway
cat <<EOF | kubectl apply -n default -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
spec:
gatewayClassName: eg
listeners:
- name: http
port: 80
protocol: HTTP
# hostname: "*.example.com"
- name: https
port: 443
protocol: HTTPS
# hostname: "*.example.com"
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: example-com
EOF
Check for any TLS certificate issues on the gateway.
kubectl -n default describe gateway eg
Create two HTTPRoutes and attach them to the HTTP and HTTPS listeners using the sectionName field.
cat <<EOF | kubectl apply -n default -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: tls-redirect
spec:
parentRefs:
- name: eg
sectionName: http
hostnames:
# - "*.example.com" # catch all hostnames
- "www.example.com"
rules:
- filters:
- type: RequestRedirect
requestRedirect:
scheme: https
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
spec:
parentRefs:
- name: eg
sectionName: https
hostnames:
- "www.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
Curl the example app through http listener:
curl --verbose --header "Host: www.example.com" http://$GATEWAY_HOST/get
Curl the example app through https listener:
curl -v -H 'Host:www.example.com' --resolve "www.example.com:443:$GATEWAY_HOST" \
--cacert CA.crt https://www.example.com:443/get
Path Redirects
Path redirects use an HTTP Path Modifier to replace either entire paths or path prefixes. For example, the HTTPRoute
below will issue a 302 redirect to all path.redirect.example requests whose path begins with /get to /status/200.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-filter-path-redirect
spec:
parentRefs:
- name: eg
hostnames:
- path.redirect.example
rules:
- matches:
- path:
type: PathPrefix
value: /get
filters:
- type: RequestRedirect
requestRedirect:
path:
type: ReplaceFullPath
replaceFullPath: /status/200
statusCode: 302
backendRefs:
- name: backend
port: 3000
EOF
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-filter-path-redirect -o yaml
Querying path.redirect.example should result in a 302 response from the example Gateway and a redirect location
containing the configured redirect path.
Query the path.redirect.example host:
curl -vvv --header "Host: path.redirect.example" "http://${GATEWAY_HOST}/get"
You should receive a 302 with a redirect location of http://path.redirect.example/status/200.
11 - HTTP Request Headers
The HTTPRoute resource can modify the headers of a request before forwarding it to the upstream service. HTTPRoute
rules cannot use both filter types at once. Currently, Envoy Gateway only supports core HTTPRoute filters which
consist of RequestRedirect and RequestHeaderModifier at the time of this writing. To learn more about HTTP routing,
refer to the Gateway API documentation.
A RequestHeaderModifier filter instructs Gateways to modify the headers in requests that match the rule
before forwarding the request upstream. Note that the RequestHeaderModifier filter will only modify headers before the
request is sent from Envoy to the upstream service and will not affect response headers returned to the downstream
client.
Prerequisites
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Adding Request Headers
The RequestHeaderModifier filter can add new headers to a request before it is sent to the upstream. If the request
does not have the header configured by the filter, then that header will be added to the request. If the request already
has the header configured by the filter, then the value of the header in the filter will be appended to the value of the
header in the request.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- headers.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
filters:
- type: RequestHeaderModifier
requestHeaderModifier:
add:
- name: "add-header"
value: "foo"
EOF
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-headers -o yaml
Get the Gateway’s address:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the upstream example app received the header add-header with the value:
something,foo
$ curl -vvv --header "Host: headers.example" "http://${GATEWAY_HOST}/get" --header "add-header: something"
...
> GET /get HTTP/1.1
> Host: headers.example
> User-Agent: curl/7.81.0
> Accept: */*
> add-header: something
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 474
< x-envoy-upstream-service-time: 0
< server: envoy
<
...
"headers": {
"Accept": [
"*/*"
],
"Add-Header": [
"something",
"foo"
],
...
Setting Request Headers
Setting headers is similar to adding headers. If the request does not have the header configured by the filter, then it will be added, but unlike adding request headers which will append the value of the header if the request already contains it, setting a header will cause the value to be replaced by the value configured in the filter.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- headers.example
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
filters:
- type: RequestHeaderModifier
requestHeaderModifier:
set:
- name: "set-header"
value: "foo"
EOF
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the upstream example app received the header add-header with the original value
something replaced by foo.
$ curl -vvv --header "Host: headers.example" "http://${GATEWAY_HOST}/get" --header "set-header: something"
...
> GET /get HTTP/1.1
> Host: headers.example
> User-Agent: curl/7.81.0
> Accept: */*
> add-header: something
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 474
< x-envoy-upstream-service-time: 0
< server: envoy
<
"headers": {
"Accept": [
"*/*"
],
"Set-Header": [
"foo"
],
...
Removing Request Headers
Headers can be removed from a request by simply supplying a list of header names.
Setting headers is similar to adding headers. If the request does not have the header configured by the filter, then it will be added, but unlike adding request headers which will append the value of the header if the request already contains it, setting a header will cause the value to be replaced by the value configured in the filter.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- headers.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
name: backend
port: 3000
weight: 1
filters:
- type: RequestHeaderModifier
requestHeaderModifier:
remove:
- "remove-header"
EOF
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the upstream example app received the header add-header, but the header
remove-header that was sent by curl was removed before the upstream received the request.
$ curl -vvv --header "Host: headers.example" "http://${GATEWAY_HOST}/get" --header "add-header: something" --header "remove-header: foo"
...
> GET /get HTTP/1.1
> Host: headers.example
> User-Agent: curl/7.81.0
> Accept: */*
> add-header: something
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 474
< x-envoy-upstream-service-time: 0
< server: envoy
<
"headers": {
"Accept": [
"*/*"
],
"Add-Header": [
"something"
],
...
Combining Filters
Headers can be added/set/removed in a single filter on the same HTTPRoute and they will all perform as expected
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- headers.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
filters:
- type: RequestHeaderModifier
requestHeaderModifier:
add:
- name: "add-header-1"
value: "foo"
set:
- name: "set-header-1"
value: "bar"
remove:
- "removed-header"
EOF
12 - HTTP Response Headers
The HTTPRoute resource can modify the headers of a response before responding it to the downstream service. To learn more about HTTP routing, refer to the Gateway API documentation.
A ResponseHeaderModifier filter instructs Gateways to modify the headers in responses that match the
rule before responding to the downstream. Note that the ResponseHeaderModifier filter will only modify headers before
the response is returned from Envoy to the downstream client and will not affect request headers forwarding to the
upstream service.
Prerequisites
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Adding Response Headers
The ResponseHeaderModifier filter can add new headers to a response before it is sent to the upstream. If the response
does not have the header configured by the filter, then that header will be added to the response. If the response
already has the header configured by the filter, then the value of the header in the filter will be appended to the
value of the header in the response.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- headers.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
filters:
- type: ResponseHeaderModifier
responseHeaderModifier:
add:
- name: "add-header"
value: "foo"
EOF
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-headers -o yaml
Get the Gateway’s address:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the downstream client received the header add-header with the value: foo
$ curl -vvv --header "Host: headers.example" "http://${GATEWAY_HOST}/get" -H 'X-Echo-Set-Header: X-Foo: value1'
...
> GET /get HTTP/1.1
> Host: headers.example
> User-Agent: curl/7.81.0
> Accept: */*
> X-Echo-Set-Header: X-Foo: value1
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 474
< x-envoy-upstream-service-time: 0
< server: envoy
< x-foo: value1
< add-header: foo
<
...
"headers": {
"Accept": [
"*/*"
],
"X-Echo-Set-Header": [
"X-Foo: value1"
]
...
Setting Response Headers
Setting headers is similar to adding headers. If the response does not have the header configured by the filter, then it will be added, but unlike adding response headers which will append the value of the header if the response already contains it, setting a header will cause the value to be replaced by the value configured in the filter.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- headers.example
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
filters:
- type: ResponseHeaderModifier
responseHeaderModifier:
set:
- name: "set-header"
value: "foo"
EOF
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the downstream client received the header set-header with the original value value1
replaced by foo.
$ curl -vvv --header "Host: headers.example" "http://${GATEWAY_HOST}/get" -H 'X-Echo-Set-Header: set-header: value1'
...
> GET /get HTTP/1.1
> Host: headers.example
> User-Agent: curl/7.81.0
> Accept: */*
> X-Echo-Set-Header: set-header: value1
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 474
< x-envoy-upstream-service-time: 0
< server: envoy
< set-header: foo
<
"headers": {
"Accept": [
"*/*"
],
"X-Echo-Set-Header": [
"set-header": value1"
]
...
Removing Response Headers
Headers can be removed from a response by simply supplying a list of header names.
Setting headers is similar to adding headers. If the response does not have the header configured by the filter, then it will be added, but unlike adding response headers which will append the value of the header if the response already contains it, setting a header will cause the value to be replaced by the value configured in the filter.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- headers.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
name: backend
port: 3000
weight: 1
filters:
- type: ResponseHeaderModifier
responseHeaderModifier:
remove:
- "remove-header"
EOF
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the header remove-header that was sent by curl was removed before the upstream
received the response.
$ curl -vvv --header "Host: headers.example" "http://${GATEWAY_HOST}/get" -H 'X-Echo-Set-Header: remove-header: value1'
...
> GET /get HTTP/1.1
> Host: headers.example
> User-Agent: curl/7.81.0
> Accept: */*
> X-Echo-Set-Header: remove-header: value1
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 474
< x-envoy-upstream-service-time: 0
< server: envoy
<
"headers": {
"Accept": [
"*/*"
],
"X-Echo-Set-Header": [
"remove-header": value1"
]
...
Combining Filters
Headers can be added/set/removed in a single filter on the same HTTPRoute and they will all perform as expected
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- headers.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
filters:
- type: ResponseHeaderModifier
responseHeaderModifier:
add:
- name: "add-header-1"
value: "foo"
set:
- name: "set-header-1"
value: "bar"
remove:
- "removed-header"
EOF
13 - HTTP Routing
The HTTPRoute resource allows users to configure HTTP routing by matching HTTP traffic and forwarding it to Kubernetes backends. Currently, the only supported backend supported by Envoy Gateway is a Service resource. This guide shows how to route traffic based on host, header, and path fields and forward the traffic to different Kubernetes Services. To learn more about HTTP routing, refer to the Gateway API documentation.
Prerequisites
Install Envoy Gateway:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.6.0 -n envoy-gateway-system --create-namespace
Wait for Envoy Gateway to become available:
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
Installation
Install the HTTP routing example resources:
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/http-routing.yaml
The manifest installs a GatewayClass, Gateway, four Deployments, four Services, and three HTTPRoute resources. The GatewayClass is a cluster-scoped resource that represents a class of Gateways that can be instantiated.
Note: Envoy Gateway is configured by default to manage a GatewayClass with
controllerName: gateway.envoyproxy.io/gatewayclass-controller.
Verification
Check the status of the GatewayClass:
kubectl get gc --selector=example=http-routing
The status should reflect “Accepted=True”, indicating Envoy Gateway is managing the GatewayClass.
A Gateway represents configuration of infrastructure. When a Gateway is created, Envoy proxy infrastructure is
provisioned or configured by Envoy Gateway. The gatewayClassName defines the name of a GatewayClass used by this
Gateway. Check the status of the Gateway:
kubectl get gateways --selector=example=http-routing
The status should reflect “Ready=True”, indicating the Envoy proxy infrastructure has been provisioned. The status also provides the address of the Gateway. This address is used later in the guide to test connectivity to proxied backend services.
The three HTTPRoute resources create routing rules on the Gateway. In order to receive traffic from a Gateway,
an HTTPRoute must be configured with parentRefs which reference the parent Gateway(s) that it should be attached to.
An HTTPRoute can match against a single set of hostnames. These hostnames are matched before any other matching
within the HTTPRoute takes place. Since example.com, foo.example.com, and bar.example.com are separate hosts with
different routing requirements, each is deployed as its own HTTPRoute - example-route, ``foo-route, and bar-route.
Check the status of the HTTPRoutes:
kubectl get httproutes --selector=example=http-routing -o yaml
The status for each HTTPRoute should surface “Accepted=True” and a parentRef that references the example Gateway.
The example-route matches any traffic for “example.com” and forwards it to the “example-svc” Service.
Testing the Configuration
Before testing HTTP routing to the example-svc backend, get the Gateway’s address.
export GATEWAY_HOST=$(kubectl get gateway/example-gateway -o jsonpath='{.status.addresses[0].value}')
Test HTTP routing to the example-svc backend.
curl -vvv --header "Host: example.com" "http://${GATEWAY_HOST}/"
A 200 status code should be returned and the body should include "pod": "example-backend-*" indicating the traffic
was routed to the example backend service. If you change the hostname to a hostname not represented in any of the
HTTPRoutes, e.g. “www.example.com”, the HTTP traffic will not be routed and a 404 should be returned.
The foo-route matches any traffic for foo.example.com and applies its routing rules to forward the traffic to the
“foo-svc” Service. Since there is only one path prefix match for /login, only foo.example.com/login/* traffic will
be forwarded. Test HTTP routing to the foo-svc backend.
curl -vvv --header "Host: foo.example.com" "http://${GATEWAY_HOST}/login"
A 200 status code should be returned and the body should include "pod": "foo-backend-*" indicating the traffic
was routed to the foo backend service. Traffic to any other paths that do not begin with /login will not be matched by
this HTTPRoute. Test this by removing /login from the request.
curl -vvv --header "Host: foo.example.com" "http://${GATEWAY_HOST}/"
The HTTP traffic will not be routed and a 404 should be returned.
Similarly, the bar-route HTTPRoute matches traffic for bar.example.com. All traffic for this hostname will be
evaluated against the routing rules. The most specific match will take precedence which means that any traffic with the
env:canary header will be forwarded to bar-svc-canary and if the header is missing or not canary then it’ll be
forwarded to bar-svc. Test HTTP routing to the bar-svc backend.
curl -vvv --header "Host: bar.example.com" "http://${GATEWAY_HOST}/"
A 200 status code should be returned and the body should include "pod": "bar-backend-*" indicating the traffic
was routed to the foo backend service.
Test HTTP routing to the bar-canary-svc backend by adding the env: canary header to the request.
curl -vvv --header "Host: bar.example.com" --header "env: canary" "http://${GATEWAY_HOST}/"
A 200 status code should be returned and the body should include "pod": "bar-canary-backend-*" indicating the
traffic was routed to the foo backend service.
14 - HTTP URL Rewrite
HTTPURLRewriteFilter defines a filter that modifies a request during forwarding. At most one of these filters may be used on a Route rule. This MUST NOT be used on the same Route rule as a HTTPRequestRedirect filter.
Prerequisites
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Rewrite URL Prefix Path
You can configure to rewrite the prefix in the url like below. In this example, any curls to
http://${GATEWAY_HOST}/get/xxx will be rewritten to http://${GATEWAY_HOST}/replace/xxx.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-filter-url-rewrite
spec:
parentRefs:
- name: eg
hostnames:
- path.rewrite.example
rules:
- matches:
- path:
value: "/get"
filters:
- type: URLRewrite
urlRewrite:
path:
type: ReplacePrefixMatch
replacePrefixMatch: /replace
backendRefs:
- name: backend
port: 3000
EOF
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-filter-url-rewrite -o yaml
Get the Gateway’s address:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Querying http://${GATEWAY_HOST}/get/origin/path should rewrite to
http://${GATEWAY_HOST}/replace/origin/path.
$ curl -L -vvv --header "Host: path.rewrite.example" "http://${GATEWAY_HOST}/get/origin/path"
...
> GET /get/origin/path HTTP/1.1
> Host: path.rewrite.example
> User-Agent: curl/7.85.0
> Accept: */*
>
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Wed, 21 Dec 2022 11:03:28 GMT
< content-length: 503
< x-envoy-upstream-service-time: 0
< server: envoy
<
{
"path": "/replace/origin/path",
"host": "path.rewrite.example",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.85.0"
],
"X-Envoy-Expected-Rq-Timeout-Ms": [
"15000"
],
"X-Envoy-Original-Path": [
"/get/origin/path"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"fd84b842-9937-4fb5-83c7-61470d854b90"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "backend-6fdd4b9bd8-8vlc5"
...
You can see that the X-Envoy-Original-Path is /get/origin/path, but the actual path is /replace/origin/path.
Rewrite URL Full Path
You can configure to rewrite the fullpath in the url like below. In this example, any request sent to
http://${GATEWAY_HOST}/get/origin/path/xxxx will be rewritten to
http://${GATEWAY_HOST}/force/replace/fullpath.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-filter-url-rewrite
spec:
parentRefs:
- name: eg
hostnames:
- path.rewrite.example
rules:
- matches:
- path:
type: PathPrefix
value: "/get/origin/path"
filters:
- type: URLRewrite
urlRewrite:
path:
type: ReplaceFullPath
replaceFullPath: /force/replace/fullpath
backendRefs:
- name: backend
port: 3000
EOF
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-filter-url-rewrite -o yaml
Querying http://${GATEWAY_HOST}/get/origin/path/extra should rewrite the request to
http://${GATEWAY_HOST}/force/replace/fullpath.
$ curl -L -vvv --header "Host: path.rewrite.example" "http://${GATEWAY_HOST}/get/origin/path/extra"
...
> GET /get/origin/path/extra HTTP/1.1
> Host: path.rewrite.example
> User-Agent: curl/7.85.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Wed, 21 Dec 2022 11:09:31 GMT
< content-length: 512
< x-envoy-upstream-service-time: 0
< server: envoy
<
{
"path": "/force/replace/fullpath",
"host": "path.rewrite.example",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.85.0"
],
"X-Envoy-Expected-Rq-Timeout-Ms": [
"15000"
],
"X-Envoy-Original-Path": [
"/get/origin/path/extra"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"8ab774d6-9ffa-4faa-abbb-f45b0db00895"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "backend-6fdd4b9bd8-8vlc5"
...
You can see that the X-Envoy-Original-Path is /get/origin/path/extra, but the actual path is
/force/replace/fullpath.
Rewrite Host Name
You can configure to rewrite the hostname like below. In this example, any requests sent to
http://${GATEWAY_HOST}/get with --header "Host: path.rewrite.example" will rewrite host into envoygateway.io.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-filter-url-rewrite
spec:
parentRefs:
- name: eg
hostnames:
- path.rewrite.example
rules:
- matches:
- path:
type: PathPrefix
value: "/get"
filters:
- type: URLRewrite
urlRewrite:
hostname: "envoygateway.io"
backendRefs:
- name: backend
port: 3000
EOF
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-filter-url-rewrite -o yaml
Querying http://${GATEWAY_HOST}/get with --header "Host: path.rewrite.example" will rewrite host into
envoygateway.io.
$ curl -L -vvv --header "Host: path.rewrite.example" "http://${GATEWAY_HOST}/get"
...
> GET /get HTTP/1.1
> Host: path.rewrite.example
> User-Agent: curl/7.85.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Wed, 21 Dec 2022 11:15:15 GMT
< content-length: 481
< x-envoy-upstream-service-time: 0
< server: envoy
<
{
"path": "/get",
"host": "envoygateway.io",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.85.0"
],
"X-Envoy-Expected-Rq-Timeout-Ms": [
"15000"
],
"X-Forwarded-Host": [
"path.rewrite.example"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"39aa447c-97b9-45a3-a675-9fb266ab1af0"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "backend-6fdd4b9bd8-8vlc5"
...
You can see that the X-Forwarded-Host is path.rewrite.example, but the actual host is envoygateway.io.
15 - HTTPRoute Request Mirroring
The HTTPRoute resource allows one or more backendRefs to be provided. Requests will be routed to these upstreams. It is possible to divide the traffic between these backends using Traffic Splitting, but it is also possible to mirror requests to another Service instead. Request mirroring is accomplished using Gateway API’s HTTPRequestMirrorFilter on the HTTPRoute.
When requests are made to a HTTPRoute that uses a HTTPRequestMirrorFilter, the response will never come from the backendRef defined in the filter. Responses from the mirror backendRef are always ignored.
Installation
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Mirroring the Traffic
Next, create a new Deployment and Service to mirror requests to. The following example will use
a second instance of the application deployed in the quickstart.
kubectl apply -f - <<EOF
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend-2
---
apiVersion: v1
kind: Service
metadata:
name: backend-2
labels:
app: backend-2
service: backend-2
spec:
ports:
- name: http
port: 3000
targetPort: 3000
selector:
app: backend-2
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-2
spec:
replicas: 1
selector:
matchLabels:
app: backend-2
version: v1
template:
metadata:
labels:
app: backend-2
version: v1
spec:
serviceAccountName: backend-2
containers:
- image: gcr.io/k8s-staging-ingressconformance/echoserver:v20221109-7ee2f3e
imagePullPolicy: IfNotPresent
name: backend-2
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
EOF
Then create an HTTPRoute that uses a HTTPRequestMirrorFilter to send requests to the original
service from the quickstart, and mirror request to the service that was just deployed.
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-mirror
spec:
parentRefs:
- name: eg
hostnames:
- backends.example
rules:
- matches:
- path:
type: PathPrefix
value: /
filters:
- type: RequestMirror
requestMirror:
backendRef:
kind: Service
name: backend-2
port: 3000
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
EOF
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-mirror -o yaml
Get the Gateway’s address:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Querying backends.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate which pod handled the request. There is only one pod in the deployment for the example app
from the quickstart, so it will be the same on all subsequent requests.
$ curl -v --header "Host: backends.example" "http://${GATEWAY_HOST}/get"
...
> GET /get HTTP/1.1
> Host: backends.example
> User-Agent: curl/7.81.0
> Accept: */*
> add-header: something
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 474
< x-envoy-upstream-service-time: 0
< server: envoy
<
...
"namespace": "default",
"ingress": "",
"service": "",
"pod": "backend-79665566f5-s589f"
...
Check the logs of the pods and you will see that the original deployment and the new deployment each got a request:
$ kubectl logs deploy/backend && kubectl logs deploy/backend-2
...
Starting server, listening on port 3000 (http)
Echoing back request made to /get to client (10.42.0.10:41566)
Starting server, listening on port 3000 (http)
Echoing back request made to /get to client (10.42.0.10:45096)
Multiple BackendRefs
When an HTTPRoute has multiple backendRefs and an HTTPRequestMirrorFilter, traffic splitting will still behave the same as it normally would for the main backendRefs while the backendRef of the HTTPRequestMirrorFilter will continue receiving mirrored copies of the incoming requests.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-mirror
spec:
parentRefs:
- name: eg
hostnames:
- backends.example
rules:
- matches:
- path:
type: PathPrefix
value: /
filters:
- type: RequestMirror
requestMirror:
backendRef:
kind: Service
name: backend-2
port: 3000
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
- group: ""
kind: Service
name: backend-3
port: 3000
EOF
Multiple HTTPRequestMirrorFilters
Multiple HTTPRequestMirrorFilters are not supported on the same HTTPRoute rule. When attempting to do so, the admission webhook will reject the configuration.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-mirror
spec:
parentRefs:
- name: eg
hostnames:
- backends.example
rules:
- matches:
- path:
type: PathPrefix
value: /
filters:
- type: RequestMirror
requestMirror:
backendRef:
kind: Service
name: backend-2
port: 3000
- type: RequestMirror
requestMirror:
backendRef:
kind: Service
name: backend-3
port: 3000
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
EOF
Error from server: error when creating "STDIN": admission webhook "validate.gateway.networking.k8s.io" denied the request: spec.rules[0].filters: Invalid value: "RequestMirror": cannot be used multiple times in the same rule
16 - HTTPRoute Traffic Splitting
The HTTPRoute resource allows one or more backendRefs to be provided. Requests will be routed to these upstreams
if they match the rules of the HTTPRoute. If an invalid backendRef is configured, then HTTP responses will be returned
with status code 500 for all requests that would have been sent to that backend.
Installation
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Single backendRef
When a single backendRef is configured in a HTTPRoute, it will receive 100% of the traffic.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- backends.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
EOF
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-headers -o yaml
Get the Gateway’s address:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Querying backends.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate which pod handled the request. There is only one pod in the deployment for the example app
from the quickstart, so it will be the same on all subsequent requests.
$ curl -vvv --header "Host: backends.example" "http://${GATEWAY_HOST}/get"
...
> GET /get HTTP/1.1
> Host: backends.example
> User-Agent: curl/7.81.0
> Accept: */*
> add-header: something
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 474
< x-envoy-upstream-service-time: 0
< server: envoy
<
...
"namespace": "default",
"ingress": "",
"service": "",
"pod": "backend-79665566f5-s589f"
...
Multiple backendRefs
If multiple backendRefs are configured, then traffic will be split between the backendRefs equally unless a weight is configured.
First, create a second instance of the example app from the quickstart:
cat <<EOF | kubectl apply -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend-2
---
apiVersion: v1
kind: Service
metadata:
name: backend-2
labels:
app: backend-2
service: backend-2
spec:
ports:
- name: http
port: 3000
targetPort: 3000
selector:
app: backend-2
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-2
spec:
replicas: 1
selector:
matchLabels:
app: backend-2
version: v1
template:
metadata:
labels:
app: backend-2
version: v1
spec:
serviceAccountName: backend-2
containers:
- image: gcr.io/k8s-staging-ingressconformance/echoserver:v20221109-7ee2f3e
imagePullPolicy: IfNotPresent
name: backend-2
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
EOF
Then create an HTTPRoute that uses both the app from the quickstart and the second instance that was just created
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- backends.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
- group: ""
kind: Service
name: backend-2
port: 3000
EOF
Querying backends.example/get should result in 200 responses from the example Gateway and the output from the
example app that indicates which pod handled the request should switch between the first pod and the second one from the
new deployment on subsequent requests.
$ curl -vvv --header "Host: backends.example" "http://${GATEWAY_HOST}/get"
...
> GET /get HTTP/1.1
> Host: backends.example
> User-Agent: curl/7.81.0
> Accept: */*
> add-header: something
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< content-length: 474
< x-envoy-upstream-service-time: 0
< server: envoy
<
...
"namespace": "default",
"ingress": "",
"service": "",
"pod": "backend-75bcd4c969-lsxpz"
...
Weighted backendRefs
If multiple backendRefs are configured and an un-even traffic split between the backends is desired, then the weight
field can be used to control the weight of requests to each backend. If weight is not configured for a backendRef it is
assumed to be 1.
The weight field in a backendRef controls the distribution of the traffic split. The proportion of
requests to a single backendRef is calculated by dividing its weight by the sum of all backendRef weights in the
HTTPRoute. The weight is not a percentage and the sum of all weights does not need to add up to 100.
The HTTPRoute below will configure the gateway to send 80% of the traffic to the backend service, and 20% to the backend-2 service.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- backends.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 8
- group: ""
kind: Service
name: backend-2
port: 3000
weight: 2
EOF
Invalid backendRefs
backendRefs can be considered invalid for the following reasons:
- The
groupfield is configured to something other than"". Currently, only the core API group (specified by omitting the group field or setting it to an empty string) is supported - The
kindfield is configured to anything other thanService. Envoy Gateway currently only supports Kubernetes Service backendRefs - The backendRef configures a service with a
namespacenot permitted by any existing ReferenceGrants - The
portfield is not configured or is configured to a port that does not exist on the Service - The named Service configured by the backendRef cannot be found
Modifying the above example to make the backend-2 backendRef invalid by using a port that does not exist on the Service
will result in 80% of the traffic being sent to the backend service, and 20% of the traffic receiving an HTTP response
with status code 500.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-headers
spec:
parentRefs:
- name: eg
hostnames:
- backends.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 8
- group: ""
kind: Service
name: backend-2
port: 9000
weight: 2
EOF
Querying backends.example/get should result in 200 responses 80% of the time, and 500 responses 20% of the time.
$ curl -vvv --header "Host: backends.example" "http://${GATEWAY_HOST}/get"
> GET /get HTTP/1.1
> Host: backends.example
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< server: envoy
< content-length: 0
<
17 - JWT Authentication
This guide provides instructions for configuring JSON Web Token (JWT) authentication. JWT authentication checks
if an incoming request has a valid JWT before routing the request to a backend service. Currently, Envoy Gateway only
supports validating a JWT from an HTTP header, e.g. Authorization: Bearer <token>.
Envoy Gateway introduces a new CRD called SecurityPolicy that allows the user to configure JWT authentication. This instantiated resource can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Prerequisites
Follow the steps from the Quickstart guide to install Envoy Gateway and the example manifest. For GRPC - follow the steps from the GRPC Routing example. Before proceeding, you should be able to query the example backend using HTTP or GRPC.
Configuration
Allow requests with a valid JWT by creating an SecurityPolicy and attaching it to the example HTTPRoute or GRPCRoute.
HTTPRoute
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/jwt/jwt.yaml
Two HTTPRoute has been created, one for /foo and another for /bar. A SecurityPolicy has been created and targeted
HTTPRoute foo to authenticate requests for /foo. The HTTPRoute bar is not targeted by the SecurityPolicy and will allow
unauthenticated requests to /bar.
Verify the HTTPRoute configuration and status:
kubectl get httproute/foo -o yaml
kubectl get httproute/bar -o yaml
The SecurityPolicy is configured for JWT authentication and uses a single JSON Web Key Set (JWKS) provider for authenticating the JWT.
Verify the SecurityPolicy configuration:
kubectl get securitypolicy/jwt-example -o yaml
GRPCRoute
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/jwt/grpc-jwt.yaml
A SecurityPolicy has been created and targeted GRPCRoute yages to authenticate all requests for yages service..
Verify the GRPCRoute configuration and status:
kubectl get grpcroute/yages -o yaml
The SecurityPolicy is configured for JWT authentication and uses a single JSON Web Key Set (JWKS) provider for authenticating the JWT.
Verify the SecurityPolicy configuration:
kubectl get securitypolicy/jwt-example -o yaml
Testing
Ensure the GATEWAY_HOST environment variable from the Quickstart guide is set. If not, follow the
Quickstart instructions to set the variable.
echo $GATEWAY_HOST
HTTPRoute
Verify that requests to /foo are denied without a JWT:
curl -sS -o /dev/null -H "Host: www.example.com" -w "%{http_code}\n" http://$GATEWAY_HOST/foo
A 401 HTTP response code should be returned.
Get the JWT used for testing request authentication:
TOKEN=$(curl https://raw.githubusercontent.com/envoyproxy/gateway/main/examples/kubernetes/jwt/test.jwt -s) && echo "$TOKEN" | cut -d '.' -f2 - | base64 --decode
Note: The above command decodes and returns the token’s payload. You can replace f2 with f1 to view the token’s
header.
Verify that a request to /foo with a valid JWT is allowed:
curl -sS -o /dev/null -H "Host: www.example.com" -H "Authorization: Bearer $TOKEN" -w "%{http_code}\n" http://$GATEWAY_HOST/foo
A 200 HTTP response code should be returned.
Verify that requests to /bar are allowed without a JWT:
curl -sS -o /dev/null -H "Host: www.example.com" -w "%{http_code}\n" http://$GATEWAY_HOST/bar
GRPCRoute
Verify that requests to yagesservice are denied without a JWT:
grpcurl -plaintext -authority=grpc-example.com ${GATEWAY_HOST}:80 yages.Echo/Ping
You should see the below response
Error invoking method "yages.Echo/Ping": rpc error: code = Unauthenticated desc = failed to query for service descriptor "yages.Echo": Jwt is missing
Get the JWT used for testing request authentication:
TOKEN=$(curl https://raw.githubusercontent.com/envoyproxy/gateway/main/examples/kubernetes/jwt/test.jwt -s) && echo "$TOKEN" | cut -d '.' -f2 - | base64 --decode
Note: The above command decodes and returns the token’s payload. You can replace f2 with f1 to view the token’s
header.
Verify that a request to yages service with a valid JWT is allowed:
grpcurl -plaintext -H "authorization: Bearer $TOKEN" -authority=grpc-example.com ${GATEWAY_HOST}:80 yages.Echo/Ping
You should see the below response
{
"text": "pong"
}
Clean-Up
Follow the steps from the Quickstart guide to uninstall Envoy Gateway and the example manifest.
Delete the SecurityPolicy:
kubectl delete securitypolicy/jwt-example
Next Steps
Checkout the Developer Guide to get involved in the project.
18 - Multicluster Service Routing
The Multicluster Service API ServiceImport object can be used as part of the GatewayAPI backendRef for configuring routes. For more information about multicluster service API follow sig documentation.
We will use Submariner project for setting up the multicluster environment for exporting the service to be routed from peer clusters.
Setting KIND clusters and installing Submariner.
- We will be using KIND clusters to demonstrate this example.
git clone https://github.com/submariner-io/submariner-operator
cd submariner-operator
make clusters
Note: remain in submariner-operator directory for the rest of the steps in this section
- Install subctl:
curl -Ls https://get.submariner.io | VERSION=v0.14.6 bash
- Set up multicluster service API and submariner for cross cluster traffic using ServiceImport
subctl deploy-broker --kubeconfig output/kubeconfigs/kind-config-cluster1 --globalnet
subctl join --kubeconfig output/kubeconfigs/kind-config-cluster1 broker-info.subm --clusterid cluster1 --natt=false
subctl join --kubeconfig output/kubeconfigs/kind-config-cluster2 broker-info.subm --clusterid cluster2 --natt=false
Once the above steps are done and all the pods are up in both the clusters. We are ready for installing envoy gateway.
Install EnvoyGateway
Install the Gateway API CRDs and Envoy Gateway in cluster1:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.6.0 -n envoy-gateway-system --create-namespace --kubeconfig output/kubeconfigs/kind-config-cluster1
Wait for Envoy Gateway to become available:
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available --kubeconfig output/kubeconfigs/kind-config-cluster1
Install Application
Install the backend application in cluster2 and export it through subctl command.
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/application.yaml --kubeconfig output/kubeconfigs/kind-config-cluster2
subctl export service backend --namespace default --kubeconfig output/kubeconfigs/kind-config-cluster2
Create Gateway API Objects
Create the Gateway API objects GatewayClass, Gateway and HTTPRoute in cluster1 to set up the routing.
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/multicluster-service.yaml --kubeconfig output/kubeconfigs/kind-config-cluster1
Testing the Configuration
Get the name of the Envoy service created the by the example Gateway:
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
Port forward to the Envoy service:
kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8888:80 &
Curl the example app through Envoy proxy:
curl --verbose --header "Host: www.example.com" http://localhost:8888/get
19 - Proxy Observability
Envoy Gateway provides observability for the ControlPlane and the underlying EnvoyProxy instances. This guide show you how to config proxy observability, includes metrics, logs, and traces.
Prerequisites
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
FluentBit is used to collect logs from the EnvoyProxy instances and forward them to Loki. Install FluentBit:
helm repo add fluent https://fluent.github.io/helm-charts
helm repo update
helm upgrade --install fluent-bit fluent/fluent-bit -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/fluent-bit/helm-values.yaml -n monitoring --create-namespace --version 0.30.4
Loki is used to store logs. Install Loki:
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/loki/loki.yaml -n monitoring
Tempo is used to store traces. Install Tempo:
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install tempo grafana/tempo -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/tempo/helm-values.yaml -n monitoring --create-namespace --version 1.3.1
OpenTelemetry Collector offers a vendor-agnostic implementation of how to receive, process and export telemetry data. Install OTel-Collector:
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update
helm upgrade --install otel-collector open-telemetry/opentelemetry-collector -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/otel-collector/helm-values.yaml -n monitoring --create-namespace --version 0.60.0
Expose endpoints:
LOKI_IP=$(kubectl get svc loki -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
TEMPO_IP=$(kubectl get svc tempo -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
Metrics
By default, Envoy Gateway expose metrics with prometheus endpoint.
Verify metrics:
export ENVOY_POD_NAME=$(kubectl get pod -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward pod/$ENVOY_POD_NAME -n envoy-gateway-system 19001:19001
# check metrics
curl localhost:19001/stats/prometheus | grep "default/backend/rule/0/match/0-www"
You can disable metrics by setting the telemetry.metrics.prometheus.disable to true in the EnvoyProxy CRD.
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/metric/disable-prometheus.yaml
Envoy Gateway can send metrics to OpenTelemetry Sink. Send metrics to OTel-Collector:
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/metric/otel-sink.yaml
Verify OTel-Collector metrics:
export OTEL_POD_NAME=$(kubectl get pod -n monitoring --selector=app.kubernetes.io/name=opentelemetry-collector -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward pod/$OTEL_POD_NAME -n monitoring 19001:19001
# check metrics
curl localhost:19001/metrics | grep "default/backend/rule/0/match/0-www"
Logs
By default, Envoy Gateway send logs to stdout in default text format. Verify logs from loki:
curl -s "http://$LOKI_IP:3100/loki/api/v1/query_range" --data-urlencode "query={job=\"fluentbit\"}" | jq '.data.result[0].values'
If you want to disable it, set the telemetry.accesslog.disable to true in the EnvoyProxy CRD.
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/accesslog/disable-accesslog.yaml
Envoy Gateway can send logs to OpenTelemetry Sink.
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/accesslog/otel-accesslog.yaml
Verify logs from loki:
curl -s "http://$LOKI_IP:3100/loki/api/v1/query_range" --data-urlencode "query={exporter=\"OTLP\"}" | jq '.data.result[0].values'
Traces
By default, Envoy Gateway doesn’t send traces to OpenTelemetry Sink.
You can enable traces by setting the telemetry.tracing in the EnvoyProxy CRD.
Note: Envoy Gateway use 100% sample rate, which means all requests will be traced. This may cause performance issues.
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/tracing/default.yaml
Verify traces from tempo:
curl -s "http://$TEMPO_IP:3100/api/search" --data-urlencode "q={ component=envoy }" | jq .traces
curl -s "http://$TEMPO_IP:3100/api/traces/<trace_id>" | jq
20 - Rate Limit
Rate limit is a feature that allows the user to limit the number of incoming requests to a predefined value based on attributes within the traffic flow.
Here are some reasons why you may want to implement Rate limits
- To prevent malicious activity such as DDoS attacks.
- To prevent applications and its resources (such as a database) from getting overloaded.
- To create API limits based on user entitlements.
Envoy Gateway supports Global rate limiting, where the rate limit is common across all the instances of Envoy proxies where its applied i.e. if the data plane has 2 replicas of Envoy running, and the rate limit is 10 requests/second, this limit is common and will be hit if 5 requests pass through the first replica and 5 requests pass through the second replica within the same second.
Envoy Gateway introduces a new CRD called BackendTrafficPolicy that allows the user to describe their rate limit intent. This instantiated resource can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Prerequisites
Install Envoy Gateway
- Follow the steps from the Quickstart Guide to install Envoy Gateway and the HTTPRoute example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Install Redis
- The global rate limit feature is based on Envoy Ratelimit which requires a Redis instance as its caching layer.
Lets install a Redis deployment in the
redis-systemnamespce.
cat <<EOF | kubectl apply -f -
kind: Namespace
apiVersion: v1
metadata:
name: redis-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
namespace: redis-system
labels:
app: redis
spec:
replicas: 1
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- image: redis:6.0.6
imagePullPolicy: IfNotPresent
name: redis
resources:
limits:
cpu: 1500m
memory: 512Mi
requests:
cpu: 200m
memory: 256Mi
---
apiVersion: v1
kind: Service
metadata:
name: redis
namespace: redis-system
labels:
app: redis
annotations:
spec:
ports:
- name: redis
port: 6379
protocol: TCP
targetPort: 6379
selector:
app: redis
---
EOF
Enable Global Rate limit in Envoy Gateway
- The default installation of Envoy Gateway installs a default EnvoyGateway configuration and attaches it
using a
ConfigMap. In the next step, we will update this resource to enable rate limit in Envoy Gateway as well as configure the URL for the Redis instance used for Global rate limiting.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: envoy-gateway-config
namespace: envoy-gateway-system
data:
envoy-gateway.yaml: |
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
provider:
type: Kubernetes
gateway:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
rateLimit:
backend:
type: Redis
redis:
url: redis.redis-system.svc.cluster.local:6379
EOF
- After updating the
ConfigMap, you will need to restart theenvoy-gatewaydeployment so the configuration kicks in
kubectl rollout restart deployment envoy-gateway -n envoy-gateway-system
Rate Limit Specific User
Here is an example of a rate limit implemented by the application developer to limit a specific user by matching on a custom x-user-id header
with a value set to one.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: policy-httproute
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: http-ratelimit
namespace: default
rateLimit:
type: Global
global:
rules:
- clientSelectors:
- headers:
- name: x-user-id
value: one
limit:
requests: 3
unit: Hour
EOF
HTTPRoute
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-ratelimit
spec:
parentRefs:
- name: eg
hostnames:
- ratelimit.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
EOF
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-ratelimit -o yaml
Get the Gateway’s address:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Lets query ratelimit.example/get 4 times. We should receive a 200 response from the example Gateway for the first 3 requests
and then receive a 429 status code for the 4th request since the limit is set at 3 requests/Hour for the request which contains the header x-user-id
and value one.
for i in {1..4}; do curl -I --header "Host: ratelimit.example" --header "x-user-id: one" http://${GATEWAY_HOST}/get ; sleep 1; done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:31 GMT
content-length: 460
x-envoy-upstream-service-time: 4
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:32 GMT
content-length: 460
x-envoy-upstream-service-time: 2
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:33 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Wed, 08 Feb 2023 02:33:34 GMT
server: envoy
transfer-encoding: chunked
You should be able to send requests with the x-user-id header and a different value and receive successful responses from the server.
for i in {1..4}; do curl -I --header "Host: ratelimit.example" --header "x-user-id: two" http://${GATEWAY_HOST}/get ; sleep 1; done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:36 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:37 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:38 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:39 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
Rate Limit Distinct Users
Here is an example of a rate limit implemented by the application developer to limit distinct users who can be differentiated based on the
value in the x-user-id header. Here, user one (recognised from the traffic flow using the header x-user-id and value one) will be rate limited at 3 requests/hour
and so will user two (recognised from the traffic flow using the header x-user-id and value two).
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: policy-httproute
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: http-ratelimit
namespace: default
rateLimit:
type: Global
global:
rules:
- clientSelectors:
- headers:
- type: Distinct
name: x-user-id
limit:
requests: 3
unit: Hour
EOF
HTTPRoute
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-ratelimit
spec:
parentRefs:
- name: eg
hostnames:
- ratelimit.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
EOF
Lets run the same command again with the header x-user-id and value one set in the request. We should the first 3 requests succeeding and
the 4th request being rate limited.
for i in {1..4}; do curl -I --header "Host: ratelimit.example" --header "x-user-id: one" http://${GATEWAY_HOST}/get ; sleep 1; done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:31 GMT
content-length: 460
x-envoy-upstream-service-time: 4
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:32 GMT
content-length: 460
x-envoy-upstream-service-time: 2
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:33 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Wed, 08 Feb 2023 02:33:34 GMT
server: envoy
transfer-encoding: chunked
You should see the same behavior when the value for header x-user-id is set to two and 4 requests are sent.
for i in {1..4}; do curl -I --header "Host: ratelimit.example" --header "x-user-id: two" http://${GATEWAY_HOST}/get ; sleep 1; done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:31 GMT
content-length: 460
x-envoy-upstream-service-time: 4
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:32 GMT
content-length: 460
x-envoy-upstream-service-time: 2
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:33 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Wed, 08 Feb 2023 02:33:34 GMT
server: envoy
transfer-encoding: chunked
Rate Limit All Requests
This example shows you how to rate limit all requests matching the HTTPRoute rule at 3 requests/Hour by leaving the clientSelectors field unset.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: policy-httproute
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: http-ratelimit
namespace: default
rateLimit:
type: Global
global:
rules:
- limit:
requests: 3
unit: Hour
EOF
HTTPRoute
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-ratelimit
spec:
parentRefs:
- name: eg
hostnames:
- ratelimit.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
EOF
for i in {1..4}; do curl -I --header "Host: ratelimit.example" http://${GATEWAY_HOST}/get ; sleep 1; done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:31 GMT
content-length: 460
x-envoy-upstream-service-time: 4
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:32 GMT
content-length: 460
x-envoy-upstream-service-time: 2
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:33 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Wed, 08 Feb 2023 02:33:34 GMT
server: envoy
transfer-encoding: chunked
Rate Limit Client IP Addresses
Here is an example of a rate limit implemented by the application developer to limit distinct users who can be differentiated based on their
IP address (also reflected in the X-Forwarded-For header).
Note: EG supports two kinds of rate limit for the IP address: exact and distinct.
- exact means that all IP addresses within the specified Source IP CIDR share the same rate limit bucket.
- distinct means that each IP address within the specified Source IP CIDR has its own rate limit bucket.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: policy-httproute
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: http-ratelimit
namespace: default
rateLimit:
type: Global
global:
rules:
- clientSelectors:
- sourceCIDR:
value: 0.0.0.0/0
type: distinct
limit:
requests: 3
unit: Hour
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-ratelimit
spec:
parentRefs:
- name: eg
hostnames:
- ratelimit.example
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
EOF
for i in {1..4}; do curl -I --header "Host: ratelimit.example" http://${GATEWAY_HOST}/get ; sleep 1; done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 28 Mar 2023 08:28:45 GMT
content-length: 512
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 28 Mar 2023 08:28:46 GMT
content-length: 512
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 28 Mar 2023 08:28:48 GMT
content-length: 512
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Tue, 28 Mar 2023 08:28:48 GMT
server: envoy
transfer-encoding: chunked
Rate Limit Jwt Claims
Here is an example of a rate limit implemented by the application developer to limit distinct users who can be differentiated based on the value of the Jwt claims carried.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: SecurityPolicy
metadata:
name: jwt-example
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: example
jwt:
providers:
- name: example
remoteJWKS:
uri: https://raw.githubusercontent.com/envoyproxy/gateway/main/examples/kubernetes/jwt/jwks.json
claimToHeaders:
- claim: name
header: x-claim-name
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: BackendTrafficPolicy
metadata:
name: policy-httproute
spec:
targetRef:
group: gateway.networking.k8s.io
kind: HTTPRoute
name: example
rateLimit:
type: Global
global:
rules:
- clientSelectors:
- headers:
- name: x-claim-name
value: John Doe
limit:
requests: 3
unit: Hour
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: example
spec:
parentRefs:
- name: eg
hostnames:
- ratelimit.example
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /foo
EOF
Get the JWT used for testing request authentication:
TOKEN=$(curl https://raw.githubusercontent.com/envoyproxy/gateway/main/examples/kubernetes/jwt/test.jwt -s) && echo "$TOKEN" | cut -d '.' -f2 - | base64 --decode
TOKEN1=$(curl https://raw.githubusercontent.com/envoyproxy/gateway/main/examples/kubernetes/jwt/with-different-claim.jwt -s) && echo "$TOKEN1" | cut -d '.' -f2 - | base64 --decode
Rate limit by carrying TOKEN
for i in {1..4}; do curl -I --header "Host: ratelimit.example" --header "Authorization: Bearer $TOKEN" http://${GATEWAY_HOST}/foo ; sleep 1; done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:00:25 GMT
content-length: 561
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:00:26 GMT
content-length: 561
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:00:27 GMT
content-length: 561
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Mon, 12 Jun 2023 12:00:28 GMT
server: envoy
transfer-encoding: chunked
No Rate Limit by carrying TOKEN1
for i in {1..4}; do curl -I --header "Host: ratelimit.example" --header "Authorization: Bearer $TOKEN1" http://${GATEWAY_HOST}/foo ; sleep 1; done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:02:34 GMT
content-length: 556
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:02:35 GMT
content-length: 556
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:02:36 GMT
content-length: 556
x-envoy-upstream-service-time: 1
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:02:37 GMT
content-length: 556
x-envoy-upstream-service-time: 0
server: envoy
(Optional) Editing Kubernetes Resources settings for the Rate Limit Service
The default installation of Envoy Gateway installs a default EnvoyGateway configuration and provides the initial rate limit kubernetes resources settings. such as
replicasis 1, requests resources cpu is100m, memory is512Mi. the others like containerimage,securityContext,envand podannotationsandsecurityContextcan be modified by modifying theConfigMap.tls.certificateRefset the client certificate for redis server TLS connections.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: ConfigMap
metadata:
name: envoy-gateway-config
namespace: envoy-gateway-system
data:
envoy-gateway.yaml: |
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
provider:
type: Kubernetes
kubernetes:
rateLimitDeployment:
replicas: 1
container:
image: envoyproxy/ratelimit:master
env:
- name: CACHE_KEY_PREFIX
value: "eg:rl:"
resources:
requests:
cpu: 100m
memory: 512Mi
securityContext:
runAsUser: 2000
allowPrivilegeEscalation: false
pod:
annotations:
key1: val1
key2: val2
securityContext:
runAsUser: 1000
runAsGroup: 3000
fsGroup: 2000
fsGroupChangePolicy: "OnRootMismatch"
gateway:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
rateLimit:
backend:
type: Redis
redis:
url: redis.redis-system.svc.cluster.local:6379
tls:
certificateRef:
name: ratelimit-cert
EOF
- After updating the
ConfigMap, you will need to restart theenvoy-gatewaydeployment so the configuration kicks in
kubectl rollout restart deployment envoy-gateway -n envoy-gateway-system
21 - Secure Gateways
This guide will help you get started using secure Gateways. The guide uses a self-signed CA, so it should be used for testing and demonstration purposes only.
Prerequisites
- OpenSSL to generate TLS assets.
Installation
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
TLS Certificates
Generate the certificates and keys used by the Gateway to terminate client TLS connections.
Create a root certificate and private key to sign certificates:
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt
Create a certificate and a private key for www.example.com:
openssl req -out www.example.com.csr -newkey rsa:2048 -nodes -keyout www.example.com.key -subj "/CN=www.example.com/O=example organization"
openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in www.example.com.csr -out www.example.com.crt
Store the cert/key in a Secret:
kubectl create secret tls example-cert --key=www.example.com.key --cert=www.example.com.crt
Update the Gateway from the Quickstart guide to include an HTTPS listener that listens on port 443 and references the
example-cert Secret:
kubectl patch gateway eg --type=json --patch '[{
"op": "add",
"path": "/spec/listeners/-",
"value": {
"name": "https",
"protocol": "HTTPS",
"port": 443,
"tls": {
"mode": "Terminate",
"certificateRefs": [{
"kind": "Secret",
"group": "",
"name": "example-cert",
}],
},
},
}]'
Verify the Gateway status:
kubectl get gateway/eg -o yaml
Testing
Clusters without External LoadBalancer Support
Get the name of the Envoy service created the by the example Gateway:
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
Port forward to the Envoy service:
kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8443:443 &
Query the example app through Envoy proxy:
curl -v -HHost:www.example.com --resolve "www.example.com:8443:127.0.0.1" \
--cacert example.com.crt https://www.example.com:8443/get
Clusters with External LoadBalancer Support
Get the External IP of the Gateway:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Query the example app through the Gateway:
curl -v -HHost:www.example.com --resolve "www.example.com:443:${GATEWAY_HOST}" \
--cacert example.com.crt https://www.example.com/get
Multiple HTTPS Listeners
Create a TLS cert/key for the additional HTTPS listener:
openssl req -out foo.example.com.csr -newkey rsa:2048 -nodes -keyout foo.example.com.key -subj "/CN=foo.example.com/O=example organization"
openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in foo.example.com.csr -out foo.example.com.crt
Store the cert/key in a Secret:
kubectl create secret tls foo-cert --key=foo.example.com.key --cert=foo.example.com.crt
Create another HTTPS listener on the example Gateway:
kubectl patch gateway eg --type=json --patch '[{
"op": "add",
"path": "/spec/listeners/-",
"value": {
"name": "https-foo",
"protocol": "HTTPS",
"port": 443,
"hostname": "foo.example.com",
"tls": {
"mode": "Terminate",
"certificateRefs": [{
"kind": "Secret",
"group": "",
"name": "foo-cert",
}],
},
},
}]'
Update the HTTPRoute to route traffic for hostname foo.example.com to the example backend service:
kubectl patch httproute backend --type=json --patch '[{
"op": "add",
"path": "/spec/hostnames/-",
"value": "foo.example.com",
}]'
Verify the Gateway status:
kubectl get gateway/eg -o yaml
Follow the steps in the Testing section to test connectivity to the backend app through both Gateway
listeners. Replace www.example.com with foo.example.com to test the new HTTPS listener.
Cross Namespace Certificate References
A Gateway can be configured to reference a certificate in a different namespace. This is allowed by a ReferenceGrant created in the target namespace. Without the ReferenceGrant, a cross-namespace reference is invalid.
Before proceeding, ensure you can query the HTTPS backend service from the Testing section.
To demonstrate cross namespace certificate references, create a ReferenceGrant that allows Gateways from the “default” namespace to reference Secrets in the “envoy-gateway-system” namespace:
$ cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: ReferenceGrant
metadata:
name: example
namespace: envoy-gateway-system
spec:
from:
- group: gateway.networking.k8s.io
kind: Gateway
namespace: default
to:
- group: ""
kind: Secret
EOF
Delete the previously created Secret:
kubectl delete secret/example-cert
The Gateway HTTPS listener should now surface the Ready: False status condition and the example HTTPS backend should
no longer be reachable through the Gateway.
kubectl get gateway/eg -o yaml
Recreate the example Secret in the envoy-gateway-system namespace:
kubectl create secret tls example-cert -n envoy-gateway-system --key=www.example.com.key --cert=www.example.com.crt
Update the Gateway HTTPS listener with namespace: envoy-gateway-system, for example:
$ cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
- name: https
protocol: HTTPS
port: 443
tls:
mode: Terminate
certificateRefs:
- kind: Secret
group: ""
name: example-cert
namespace: envoy-gateway-system
EOF
The Gateway HTTPS listener status should now surface the Ready: True condition and you should once again be able to
query the HTTPS backend through the Gateway.
Lastly, test connectivity using the above Testing section.
Clean-Up
Follow the steps from the Quickstart Guide to uninstall Envoy Gateway and the example manifest.
Delete the Secrets:
kubectl delete secret/example-cert
kubectl delete secret/foo-cert
RSA + ECDSA Dual stack certificates
This section gives a walkthrough to generate RSA and ECDSA derived certificates and keys for the Server, which can then be configured in the Gateway listener, to terminate TLS traffic.
Prerequisites
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Follow the steps in the TLS Certificates section in the guide to generate self-signed RSA derived Server certificate and private key, and configure those in the Gateway listener configuration to terminate HTTPS traffic.
Pre-checks
While testing in Cluster without External LoadBalancer Support, we can query the example app through Envoy proxy while enforcing an RSA cipher, as shown below:
curl -v -HHost:www.example.com --resolve "www.example.com:8443:127.0.0.1" \
--cacert example.com.crt https://www.example.com:8443/get -Isv --ciphers ECDHE-RSA-CHACHA20-POLY1305 --tlsv1.2 --tls-max 1.2
Since the Secret configured at this point is an RSA based Secret, if we enforce the usage of an ECDSA cipher, the call should fail as follows
$ curl -v -HHost:www.example.com --resolve "www.example.com:8443:127.0.0.1" \
--cacert example.com.crt https://www.example.com:8443/get -Isv --ciphers ECDHE-ECDSA-CHACHA20-POLY1305 --tlsv1.2 --tls-max 1.2
* Added www.example.com:8443:127.0.0.1 to DNS cache
* Hostname www.example.com was found in DNS cache
* Trying 127.0.0.1:8443...
* Connected to www.example.com (127.0.0.1) port 8443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* Cipher selection: ECDHE-ECDSA-CHACHA20-POLY1305
* CAfile: example.com.crt
* CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure
* Closing connection 0
Moving forward in the doc, we will be configuring the existing Gateway listener to accept both kinds of ciphers.
TLS Certificates
Reuse the CA certificate and key pair generated in the Secure Gateways guide and use this CA to sign both RSA and ECDSA Server certificates.
Note the CA certificate and key names are example.com.crt and example.com.key respectively.
Create an ECDSA certificate and a private key for www.example.com:
openssl ecparam -noout -genkey -name prime256v1 -out www.example.com.ecdsa.key
openssl req -new -SHA384 -key www.example.com.ecdsa.key -nodes -out www.example.com.ecdsa.csr -subj "/CN=www.example.com/O=example organization"
openssl x509 -req -SHA384 -days 365 -in www.example.com.ecdsa.csr -CA example.com.crt -CAkey example.com.key -CAcreateserial -out www.example.com.ecdsa.crt
Store the cert/key in a Secret:
kubectl create secret tls example-cert-ecdsa --key=www.example.com.ecdsa.key --cert=www.example.com.ecdsa.crt
Patch the Gateway with this additional ECDSA Secret:
kubectl patch gateway eg --type=json --patch '[{
"op": "add",
"path": "/spec/listeners/1/tls/certificateRefs/-",
"value": {
"name": "example-cert-ecdsa",
},
}]'
Verify the Gateway status:
kubectl get gateway/eg -o yaml
Testing
Again, while testing in Cluster without External LoadBalancer Support, we can query the example app through Envoy proxy while enforcing an RSA cipher, which should work as it did before:
curl -v -HHost:www.example.com --resolve "www.example.com:8443:127.0.0.1" \
--cacert example.com.crt https://www.example.com:8443/get -Isv --ciphers ECDHE-RSA-CHACHA20-POLY1305 --tlsv1.2 --tls-max 1.2
...
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-CHACHA20-POLY1305
...
Additionally, querying the example app while enforcing an ECDSA cipher should also work now:
curl -v -HHost:www.example.com --resolve "www.example.com:8443:127.0.0.1" \
--cacert example.com.crt https://www.example.com:8443/get -Isv --ciphers ECDHE-ECDSA-CHACHA20-POLY1305 --tlsv1.2 --tls-max 1.2
...
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
...
SNI based Certificate selection
This sections gives a walkthrough to generate multiple certificates corresponding to different FQDNs. The same Gateway listener can then be configured to terminate TLS traffic for multiple FQDNs based on the SNI matching.
Prerequisites
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Follow the steps in the TLS Certificates section in the guide to generate self-signed RSA derived Server certificate and private key, and configure those in the Gateway listener configuration to terminate HTTPS traffic.
Additional Configurations
Using the TLS Certificates section in the guide we first generate additional Secret for another Host www.sample.com.
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=sample Inc./CN=sample.com' -keyout sample.com.key -out sample.com.crt
openssl req -out www.sample.com.csr -newkey rsa:2048 -nodes -keyout www.sample.com.key -subj "/CN=www.sample.com/O=sample organization"
openssl x509 -req -days 365 -CA sample.com.crt -CAkey sample.com.key -set_serial 0 -in www.sample.com.csr -out www.sample.com.crt
kubectl create secret tls sample-cert --key=www.sample.com.key --cert=www.sample.com.crt
Note that all occurrences of example.com were just replaced with sample.com
Next we update the Gateway configuration to accommodate the new Certificate which will be used to Terminate TLS traffic:
kubectl patch gateway eg --type=json --patch '[{
"op": "add",
"path": "/spec/listeners/1/tls/certificateRefs/-",
"value": {
"name": "sample-cert",
},
}]'
Finally, we update the HTTPRoute to route traffic for hostname www.sample.com to the example backend service:
kubectl patch httproute backend --type=json --patch '[{
"op": "add",
"path": "/spec/hostnames/-",
"value": "www.sample.com",
}]'
Testing
Clusters without External LoadBalancer Support
Get the name of the Envoy service created the by the example Gateway:
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
Port forward to the Envoy service:
kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8443:443 &
Query the example app through Envoy proxy:
curl -v -HHost:www.example.com --resolve "www.example.com:8443:127.0.0.1" \
--cacert example.com.crt https://www.example.com:8443/get -I
Similarly, query the sample app through the same Envoy proxy:
curl -v -HHost:www.sample.com --resolve "www.sample.com:8443:127.0.0.1" \
--cacert sample.com.crt https://www.sample.com:8443/get -I
Since the multiple certificates are configured on the same Gateway listener, Envoy was able to provide the client with appropriate certificate based on the SNI in the client request.
Clusters with External LoadBalancer Support
Refer to the steps mentioned earlier in the guide under Testing in clusters with External LoadBalancer Support
Next Steps
Checkout the Developer Guide to get involved in the project.
22 - TCP Routing
TCPRoute provides a way to route TCP requests. When combined with a Gateway listener, it can be used to forward connections on the port specified by the listener to a set of backends specified by the TCPRoute. To learn more about HTTP routing, refer to the Gateway API documentation.
Installation
Install Envoy Gateway:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.6.0 -n envoy-gateway-system --create-namespace
Wait for Envoy Gateway to become available:
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
Configuration
In this example, we have one Gateway resource and two TCPRoute resources that distribute the traffic with the following rules:
All TCP streams on port 8088 of the Gateway are forwarded to port 3001 of foo Kubernetes Service.
All TCP streams on port 8089 of the Gateway are forwarded to port 3002 of bar Kubernetes Service.
In this example two TCP listeners will be applied to the Gateway in order to route them to two separate backend
TCPRoutes, note that the protocol set for the listeners on the Gateway is TCP:
Install the GatewayClass and a tcp-gateway Gateway first.
cat <<EOF | kubectl apply -f -
kind: GatewayClass
apiVersion: gateway.networking.k8s.io/v1
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: tcp-gateway
spec:
gatewayClassName: eg
listeners:
- name: foo
protocol: TCP
port: 8088
allowedRoutes:
kinds:
- kind: TCPRoute
- name: bar
protocol: TCP
port: 8089
allowedRoutes:
kinds:
- kind: TCPRoute
EOF
Install two services foo and bar, which are binded to backend-1 and backend-2.
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Service
metadata:
name: foo
labels:
app: backend-1
spec:
ports:
- name: http
port: 3001
targetPort: 3000
selector:
app: backend-1
---
apiVersion: v1
kind: Service
metadata:
name: bar
labels:
app: backend-2
spec:
ports:
- name: http
port: 3002
targetPort: 3000
selector:
app: backend-2
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-1
spec:
replicas: 1
selector:
matchLabels:
app: backend-1
version: v1
template:
metadata:
labels:
app: backend-1
version: v1
spec:
containers:
- image: gcr.io/k8s-staging-ingressconformance/echoserver:v20221109-7ee2f3e
imagePullPolicy: IfNotPresent
name: backend-1
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: SERVICE_NAME
value: foo
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend-2
spec:
replicas: 1
selector:
matchLabels:
app: backend-2
version: v1
template:
metadata:
labels:
app: backend-2
version: v1
spec:
containers:
- image: gcr.io/k8s-staging-ingressconformance/echoserver:v20221109-7ee2f3e
imagePullPolicy: IfNotPresent
name: backend-2
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: SERVICE_NAME
value: bar
EOF
Install two TCPRoutes tcp-app-1 and tcp-app-2 with different sectionName:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
name: tcp-app-1
spec:
parentRefs:
- name: tcp-gateway
sectionName: foo
rules:
- backendRefs:
- name: foo
port: 3001
---
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: TCPRoute
metadata:
name: tcp-app-2
spec:
parentRefs:
- name: tcp-gateway
sectionName: bar
rules:
- backendRefs:
- name: bar
port: 3002
EOF
In the above example we separate the traffic for the two separate backend TCP Services by using the sectionName field in the parentRefs:
spec:
parentRefs:
- name: tcp-gateway
sectionName: foo
This corresponds directly with the name in the listeners in the Gateway:
listeners:
- name: foo
protocol: TCP
port: 8088
- name: bar
protocol: TCP
port: 8089
In this way each TCPRoute “attaches” itself to a different port on the Gateway so that the foo service
is taking traffic for port 8088 from outside the cluster and bar service takes the port 8089 traffic.
Before testing, please get the tcp-gateway Gateway’s address first:
export GATEWAY_HOST=$(kubectl get gateway/tcp-gateway -o jsonpath='{.status.addresses[0].value}')
You can try to use nc to test the TCP connections of envoy gateway with different ports, and you can see them succeeded:
nc -zv ${GATEWAY_HOST} 8088
nc -zv ${GATEWAY_HOST} 8089
You can also try to send requests to envoy gateway and get responses as shown below:
curl -i "http://${GATEWAY_HOST}:8088"
HTTP/1.1 200 OK
Content-Type: application/json
X-Content-Type-Options: nosniff
Date: Tue, 03 Jan 2023 10:18:36 GMT
Content-Length: 267
{
"path": "/",
"host": "xxx.xxx.xxx.xxx:8088",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.85.0"
]
},
"namespace": "default",
"ingress": "",
"service": "foo",
"pod": "backend-1-c6c5fb958-dl8vl"
}
You can see that the traffic routing to foo service when sending request to 8088 port.
curl -i "http://${GATEWAY_HOST}:8089"
HTTP/1.1 200 OK
Content-Type: application/json
X-Content-Type-Options: nosniff
Date: Tue, 03 Jan 2023 10:19:28 GMT
Content-Length: 267
{
"path": "/",
"host": "xxx.xxx.xxx.xxx:8089",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.85.0"
]
},
"namespace": "default",
"ingress": "",
"service": "bar",
"pod": "backend-2-98fcff498-hcmgb"
}
You can see that the traffic routing to bar service when sending request to 8089 port.
23 - TLS Passthrough
This guide will walk through the steps required to configure TLS Passthrough via Envoy Gateway. Unlike configuring Secure Gateways, where the Gateway terminates the client TLS connection, TLS Passthrough allows the application itself to terminate the TLS connection, while the Gateway routes the requests to the application based on SNI headers.
Prerequisites
- OpenSSL to generate TLS assets.
Installation
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
TLS Certificates
Generate the certificates and keys used by the Service to terminate client TLS connections. For the application, we’ll deploy a sample echoserver app, with the certificates loaded in the application Pod.
Note: These certificates will not be used by the Gateway, but will remain in the application scope.
Create a root certificate and private key to sign certificates:
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt
Create a certificate and a private key for passthrough.example.com:
openssl req -out passthrough.example.com.csr -newkey rsa:2048 -nodes -keyout passthrough.example.com.key -subj "/CN=passthrough.example.com/O=some organization"
openssl x509 -req -sha256 -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in passthrough.example.com.csr -out passthrough.example.com.crt
Store the cert/keys in A Secret:
kubectl create secret tls server-certs --key=passthrough.example.com.key --cert=passthrough.example.com.crt
Deployment
Deploy TLS Passthrough application Deployment, Service and TLSRoute:
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/tls-passthrough.yaml
Patch the Gateway from the Quickstart guide to include a TLS listener that listens on port 6443 and is configured for
TLS mode Passthrough:
kubectl patch gateway eg --type=json --patch '[{
"op": "add",
"path": "/spec/listeners/-",
"value": {
"name": "tls",
"protocol": "TLS",
"hostname": "passthrough.example.com",
"tls": {"mode": "Passthrough"},
"port": 6443,
},
}]'
Testing
Clusters without External LoadBalancer Support
Get the name of the Envoy service created the by the example Gateway:
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
Port forward to the Envoy service:
kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 6043:6443 &
Curl the example app through Envoy proxy:
curl -v --resolve "passthrough.example.com:6043:127.0.0.1" https://passthrough.example.com:6043 \
--cacert passthrough.example.com.crt
Clusters with External LoadBalancer Support
You can also test the same functionality by sending traffic to the External IP of the Gateway:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Curl the example app through the Gateway, e.g. Envoy proxy:
curl -v -HHost:passthrough.example.com --resolve "passthrough.example.com:6443:${GATEWAY_HOST}" \
--cacert example.com.crt https://passthrough.example.com:6443/get
Clean-Up
Follow the steps from the Quickstart Guide to uninstall Envoy Gateway and the example manifest.
Delete the Secret:
kubectl delete secret/server-certs
Next Steps
Checkout the Developer Guide to get involved in the project.
24 - TLS Termination for TCP
This guide will walk through the steps required to configure TLS Terminate mode for TCP traffic via Envoy Gateway. The guide uses a self-signed CA, so it should be used for testing and demonstration purposes only.
Prerequisites
- OpenSSL to generate TLS assets.
Installation
Follow the steps from the Quickstart Guide to install Envoy Gateway.
TLS Certificates
Generate the certificates and keys used by the Gateway to terminate client TLS connections.
Create a root certificate and private key to sign certificates:
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -subj '/O=example Inc./CN=example.com' -keyout example.com.key -out example.com.crt
Create a certificate and a private key for www.example.com:
openssl req -out www.example.com.csr -newkey rsa:2048 -nodes -keyout www.example.com.key -subj "/CN=www.example.com/O=example organization"
openssl x509 -req -days 365 -CA example.com.crt -CAkey example.com.key -set_serial 0 -in www.example.com.csr -out www.example.com.crt
Store the cert/key in a Secret:
kubectl create secret tls example-cert --key=www.example.com.key --cert=www.example.com.crt
Install the TLS Termination for TCP example resources:
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/tls-termination.yaml
Verify the Gateway status:
kubectl get gateway/eg -o yaml
Testing
Clusters without External LoadBalancer Support
Get the name of the Envoy service created the by the example Gateway:
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
Port forward to the Envoy service:
kubectl -n envoy-gateway-system port-forward service/${ENVOY_SERVICE} 8443:443 &
Query the example app through Envoy proxy:
curl -v -HHost:www.example.com --resolve "www.example.com:8443:127.0.0.1" \
--cacert example.com.crt https://www.example.com:8443/get
Clusters with External LoadBalancer Support
Get the External IP of the Gateway:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Query the example app through the Gateway:
curl -v -HHost:www.example.com --resolve "www.example.com:443:${GATEWAY_HOST}" \
--cacert example.com.crt https://www.example.com/get
25 - UDP Routing
The UDPRoute resource allows users to configure UDP routing by matching UDP traffic and forwarding it to Kubernetes backends. This guide will use CoreDNS example to walk you through the steps required to configure UDPRoute on Envoy Gateway.
Note: UDPRoute allows Envoy Gateway to operate as a non-transparent proxy between a UDP client and server. The lack of transparency means that the upstream server will see the source IP and port of the Gateway instead of the client. For additional information, refer to Envoy’s UDP proxy documentation.
Prerequisites
Install Envoy Gateway:
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v0.6.0 -n envoy-gateway-system --create-namespace
Wait for Envoy Gateway to become available:
kubectl wait --timeout=5m -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
Installation
Install CoreDNS in the Kubernetes cluster as the example backend. The installed CoreDNS is listening on UDP port 53 for DNS lookups.
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/kubernetes/udp-routing-example-backend.yaml
Wait for the CoreDNS deployment to become available:
kubectl wait --timeout=5m deployment/coredns --for=condition=Available
Update the Gateway from the Quickstart guide to include a UDP listener that listens on UDP port 5300:
kubectl patch gateway eg --type=json --patch '[{
"op": "add",
"path": "/spec/listeners/-",
"value": {
"name": "coredns",
"protocol": "UDP",
"port": 5300,
"allowedRoutes": {
"kinds": [{
"kind": "UDPRoute"
}]
}
},
}]'
Verify the Gateway status:
kubectl get gateway/eg -o yaml
Configuration
Create a UDPRoute resource to route UDP traffic received on Gateway port 5300 to the CoredDNS backend.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1alpha2
kind: UDPRoute
metadata:
name: coredns
spec:
parentRefs:
- name: eg
sectionName: coredns
rules:
- backendRefs:
- name: coredns
port: 53
EOF
Verify the UDPRoute status:
kubectl get udproute/coredns -o yaml
Testing
Get the External IP of the Gateway:
export GATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Use dig command to query the dns entry foo.bar.com through the Gateway.
dig @${GATEWAY_HOST} -p 5300 foo.bar.com
You should see the result of the dns query as the below output, which means that the dns query has been successfully routed to the backend CoreDNS.
Note: 49.51.177.138 is the resolved address of GATEWAY_HOST.
; <<>> DiG 9.18.1-1ubuntu1.1-Ubuntu <<>> @49.51.177.138 -p 5300 foo.bar.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58125
;; flags: qr aa rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 24fb86eba96ebf62 (echoed)
;; QUESTION SECTION:
;foo.bar.com. IN A
;; ADDITIONAL SECTION:
foo.bar.com. 0 IN A 10.244.0.19
_udp.foo.bar.com. 0 IN SRV 0 0 42376 .
;; Query time: 1 msec
;; SERVER: 49.51.177.138#5300(49.51.177.138) (UDP)
;; WHEN: Fri Jan 13 10:20:34 UTC 2023
;; MSG SIZE rcvd: 114
Clean-Up
Follow the steps from the Quickstart Guide to uninstall Envoy Gateway.
Delete the CoreDNS example manifest and the UDPRoute:
kubectl delete deploy/coredns
kubectl delete service/coredns
kubectl delete cm/coredns
kubectl delete udproute/coredns
Next Steps
Checkout the Developer Guide to get involved in the project.
26 - Use egctl
egctl is a command line tool to provide additional functionality for Envoy Gateway users.
egctl experimental translate
This subcommand allows users to translate from an input configuration type to an output configuration type.
In the below example, we will translate the Kubernetes resources (including the Gateway API resources) into xDS resources.
cat <<EOF | egctl x translate --from gateway-api --to xds -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: v1
kind: Namespace
metadata:
name: default
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: default
labels:
app: backend
service: backend
spec:
clusterIP: "1.1.1.1"
type: ClusterIP
ports:
- name: http
port: 3000
targetPort: 3000
protocol: TCP
selector:
app: backend
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: default
spec:
parentRefs:
- name: eg
hostnames:
- "www.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
configKey: default-eg
configs:
- '@type': type.googleapis.com/envoy.admin.v3.BootstrapConfigDump
bootstrap:
admin:
accessLog:
- name: envoy.access_loggers.file
typedConfig:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socketAddress:
address: 127.0.0.1
portValue: 19000
dynamicResources:
cdsConfig:
apiConfigSource:
apiType: DELTA_GRPC
grpcServices:
- envoyGrpc:
clusterName: xds_cluster
setNodeOnFirstMessageOnly: true
transportApiVersion: V3
resourceApiVersion: V3
ldsConfig:
apiConfigSource:
apiType: DELTA_GRPC
grpcServices:
- envoyGrpc:
clusterName: xds_cluster
setNodeOnFirstMessageOnly: true
transportApiVersion: V3
resourceApiVersion: V3
layeredRuntime:
layers:
- name: runtime-0
rtdsLayer:
name: runtime-0
rtdsConfig:
apiConfigSource:
apiType: DELTA_GRPC
grpcServices:
- envoyGrpc:
clusterName: xds_cluster
transportApiVersion: V3
resourceApiVersion: V3
staticResources:
clusters:
- connectTimeout: 10s
loadAssignment:
clusterName: xds_cluster
endpoints:
- lbEndpoints:
- endpoint:
address:
socketAddress:
address: envoy-gateway
portValue: 18000
name: xds_cluster
transportSocket:
name: envoy.transport_sockets.tls
typedConfig:
'@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
commonTlsContext:
tlsCertificateSdsSecretConfigs:
- name: xds_certificate
sdsConfig:
pathConfigSource:
path: /sds/xds-certificate.json
resourceApiVersion: V3
tlsParams:
tlsMaximumProtocolVersion: TLSv1_3
validationContextSdsSecretConfig:
name: xds_trusted_ca
sdsConfig:
pathConfigSource:
path: /sds/xds-trusted-ca.json
resourceApiVersion: V3
type: STRICT_DNS
typedExtensionProtocolOptions:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
'@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicitHttpConfig:
http2ProtocolOptions: {}
- '@type': type.googleapis.com/envoy.admin.v3.ClustersConfigDump
dynamicActiveClusters:
- cluster:
'@type': type.googleapis.com/envoy.config.cluster.v3.Cluster
commonLbConfig:
localityWeightedLbConfig: {}
connectTimeout: 10s
dnsLookupFamily: V4_ONLY
loadAssignment:
clusterName: default-backend-rule-0-match-0-www.example.com
endpoints:
- lbEndpoints:
- endpoint:
address:
socketAddress:
address: 1.1.1.1
portValue: 3000
loadBalancingWeight: 1
loadBalancingWeight: 1
locality: {}
name: default-backend-rule-0-match-0-www.example.com
outlierDetection: {}
type: STATIC
- '@type': type.googleapis.com/envoy.admin.v3.ListenersConfigDump
dynamicListeners:
- activeState:
listener:
'@type': type.googleapis.com/envoy.config.listener.v3.Listener
accessLog:
- filter:
responseFlagFilter:
flags:
- NR
name: envoy.access_loggers.file
typedConfig:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
address:
socketAddress:
address: 0.0.0.0
portValue: 10080
defaultFilterChain:
filters:
- name: envoy.filters.network.http_connection_manager
typedConfig:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
accessLog:
- name: envoy.access_loggers.file
typedConfig:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
httpFilters:
- name: envoy.filters.http.router
typedConfig:
'@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
rds:
configSource:
apiConfigSource:
apiType: DELTA_GRPC
grpcServices:
- envoyGrpc:
clusterName: xds_cluster
setNodeOnFirstMessageOnly: true
transportApiVersion: V3
resourceApiVersion: V3
routeConfigName: default-eg-http
statPrefix: http
upgradeConfigs:
- upgradeType: websocket
useRemoteAddress: true
name: default-eg-http
- '@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
dynamicRouteConfigs:
- routeConfig:
'@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
name: default-eg-http
virtualHosts:
- domains:
- www.example.com
name: default-eg-http-www.example.com
routes:
- match:
prefix: /
route:
cluster: default-backend-rule-0-match-0-www.example.com
resourceType: all
You can also use the --type/-t flag to retrieve specific output types. In the below example, we will translate the Kubernetes resources (including the Gateway API resources) into xDS route resources.
cat <<EOF | egctl x translate --from gateway-api --to xds -t route -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: v1
kind: Namespace
metadata:
name: default
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: default
labels:
app: backend
service: backend
spec:
clusterIP: "1.1.1.1"
type: ClusterIP
ports:
- name: http
port: 3000
targetPort: 3000
protocol: TCP
selector:
app: backend
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: default
spec:
parentRefs:
- name: eg
hostnames:
- "www.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
'@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
configKey: default-eg
dynamicRouteConfigs:
- routeConfig:
'@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
name: default-eg-http
virtualHosts:
- domains:
- www.example.com
name: default-eg-http
routes:
- match:
prefix: /
route:
cluster: default-backend-rule-0-match-0-www.example.com
resourceType: route
Add Missing Resources
You can pass the --add-missing-resources flag to use dummy non Gateway API resources instead of specifying them explicitly.
For example, this will provide the similar result as the above:
cat <<EOF | egctl x translate --add-missing-resources --from gateway-api --to gateway-api -t route -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: default
spec:
parentRefs:
- name: eg
hostnames:
- "www.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
You can see the output contains a EnvoyProxy resource that can be used as a starting point to modify the xDS bootstrap resource for the managed Envoy Proxy fleet.
envoyProxy:
metadata:
creationTimestamp: null
name: default-envoy-proxy
namespace: envoy-gateway-system
spec:
bootstrap: |
admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 19000
dynamic_resources:
ads_config:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
ads: {}
resource_api_version: V3
cds_config:
ads: {}
resource_api_version: V3
static_resources:
clusters:
- connect_timeout: 10s
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
typed_extension_protocol_options:
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
"explicit_http_config":
"http2_protocol_options": {}
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: "/sds/xds-certificate.json"
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: "/sds/xds-trusted-ca.json"
resource_api_version: V3
layered_runtime:
layers:
- name: runtime-0
rtds_layer:
rtds_config:
ads: {}
resource_api_version: V3
name: runtime-0
logging: {}
status: {}
gatewayClass:
metadata:
creationTimestamp: null
name: eg
namespace: envoy-gateway-system
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: default-envoy-proxy
namespace: envoy-gateway-system
status:
conditions:
- lastTransitionTime: "2023-04-19T20:30:46Z"
message: Valid GatewayClass
reason: Accepted
status: "True"
type: Accepted
gateways:
- metadata:
creationTimestamp: null
name: eg
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
port: 80
protocol: HTTP
status:
listeners:
- attachedRoutes: 1
conditions:
- lastTransitionTime: "2023-04-19T20:30:46Z"
message: Sending translated listener configuration to the data plane
reason: Programmed
status: "True"
type: Programmed
- lastTransitionTime: "2023-04-19T20:30:46Z"
message: Listener has been successfully translated
reason: Accepted
status: "True"
type: Accepted
name: http
supportedKinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
- group: gateway.networking.k8s.io
kind: GRPCRoute
httpRoutes:
- metadata:
creationTimestamp: null
name: backend
namespace: default
spec:
hostnames:
- www.example.com
parentRefs:
- name: eg
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
status:
parents:
- conditions:
- lastTransitionTime: "2023-04-19T20:30:46Z"
message: Route is accepted
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2023-04-19T20:30:46Z"
message: Resolved all the Object references for the Route
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
name: eg
Sometimes you might find that egctl doesn’t provide an expected result. For example, the following example provides an empty route resource:
cat <<EOF | egctl x translate --from gateway-api --type route --to xds -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
spec:
gatewayClassName: eg
listeners:
- name: tls
protocol: TLS
port: 8443
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
spec:
parentRefs:
- name: eg
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
xds:
envoy-gateway-system-eg:
'@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
Validating Gateway API Configuration
You can add an additional target gateway-api to show the processed Gateway API resources. For example, translating the above resources with the new argument shows that the HTTPRoute is rejected because there is no ready listener for it:
cat <<EOF | egctl x translate --from gateway-api --type route --to gateway-api,xds -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
spec:
gatewayClassName: eg
listeners:
- name: tls
protocol: TLS
port: 8443
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
spec:
parentRefs:
- name: eg
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
gatewayClass:
metadata:
creationTimestamp: null
name: eg
namespace: envoy-gateway-system
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
status:
conditions:
- lastTransitionTime: "2023-04-19T20:54:52Z"
message: Valid GatewayClass
reason: Accepted
status: "True"
type: Accepted
gateways:
- metadata:
creationTimestamp: null
name: eg
namespace: envoy-gateway-system
spec:
gatewayClassName: eg
listeners:
- name: tls
port: 8443
protocol: TLS
status:
listeners:
- attachedRoutes: 0
conditions:
- lastTransitionTime: "2023-04-19T20:54:52Z"
message: Listener must have TLS set when protocol is TLS.
reason: Invalid
status: "False"
type: Programmed
name: tls
supportedKinds:
- group: gateway.networking.k8s.io
kind: TLSRoute
httpRoutes:
- metadata:
creationTimestamp: null
name: backend
namespace: envoy-gateway-system
spec:
parentRefs:
- name: eg
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
status:
parents:
- conditions:
- lastTransitionTime: "2023-04-19T20:54:52Z"
message: There are no ready listeners for this parent ref
reason: NoReadyListeners
status: "False"
type: Accepted
- lastTransitionTime: "2023-04-19T20:54:52Z"
message: Service envoy-gateway-system/backend not found
reason: BackendNotFound
status: "False"
type: ResolvedRefs
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
name: eg
xds:
envoy-gateway-system-eg:
'@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
27 - Using cert-manager For TLS Termination
This guide shows how to set up cert-manager to automatically create certificates and secrets for use by Envoy Gateway. It will first show how to enable the self-sign issuer, which is useful to test that cert-manager and Envoy Gateway can talk to each other. Then it shows how to use Let’s Encrypt’s staging environment. Changing to the Let’s Encrypt production environment is straight-forward after that.
Prerequisites
- A Kubernetes cluster and a configured
kubectl. - The
helmcommand. - The
curlcommand or similar for testing HTTPS requests. - For the ACME HTTP-01 challenge to work
- your Gateway must be reachable on the public Internet.
- the domain name you use (we use
www.example.com) must point to the Gateway’s external IP(s).
Installation
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Deploying cert-manager
This is a summary of cert-manager Installation with Helm.
Installing cert-manager is straight-forward, but currently (v1.12) requires setting a feature gate to enable the Gateway API support.
$ helm repo add jetstack https://charts.jetstack.io
$ helm upgrade --install --create-namespace --namespace cert-manager --set installCRDs=true --set featureGates=ExperimentalGatewayAPISupport=true cert-manager jetstack/cert-manager
You should now have cert-manager running with nothing to do:
$ kubectl wait --for=condition=Available deployment -n cert-manager --all
deployment.apps/cert-manager condition met
deployment.apps/cert-manager-cainjector condition met
deployment.apps/cert-manager-webhook condition met
$ kubectl get -n cert-manager deployment
NAME READY UP-TO-DATE AVAILABLE AGE
cert-manager 1/1 1 1 42m
cert-manager-cainjector 1/1 1 1 42m
cert-manager-webhook 1/1 1 1 42m
A Self-Signing Issuer
cert-manager can have any number of issuer configurations. The simplest issuer type is SelfSigned. It simply takes the certificate request and signs it with the private key it generates for the TLS Secret.
Self-signed certificates don't provide any help in establishing trust between certificates.
However, they are great for initial testing, due to their simplicity.
To install self-signing, run
$ kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: selfsigned
spec:
selfSigned: {}
EOF
Creating a TLS Gateway Listener
We now have to patch the example Gateway to reference cert-manager:
$ kubectl patch gateway/eg --patch '
metadata:
annotations:
cert-manager.io/cluster-issuer: selfsigned
cert-manager.io/common-name: "Hello World!"
spec:
listeners:
- name: https
protocol: HTTPS
hostname: www.example.com
port: 443
tls:
mode: Terminate
certificateRefs:
- kind: Secret
name: eg-https
' --type=merge
You could instead create a new Gateway serving HTTPS, if you’d prefer. cert-manager doesn’t care, but we’ll keep it all together in this guide.
Nowadays, X.509 certificates don’t use the subject Common Name for hostname matching, so you can set it to whatever you want, or leave it empty. The important parts here are
- the annotation referencing the “selfsigned” ClusterIssuer we created above,
- the
hostname, which is required (but see #6051 for computing it based on attached HTTPRoutes), and - the named Secret, which is what cert-manager will create for us.
The annotations are documented in Supported Annotations.
Patching the Gateway makes cert-manager create a self-signed certificate within a few seconds.
Eventually, the Gateway becomes Programmed again:
$ kubectl wait --for=condition=Programmed gateway/eg
gateway.gateway.networking.k8s.io/eg condition met
Testing The Gateway
See Testing in Secure Gateways for the general idea.
Since we have a self-signed certificate, curl will by default reject it, requiring the -k flag:
$ curl -kv -HHost:www.example.com https://127.0.0.1/get
...
* Server certificate:
* subject: CN=Hello World!
...
< HTTP/2 200
...
How cert-manager and Envoy Gateway Interact
This explains cert-manager Concepts in an Envoy Gateway context.
In the interaction between the two, cert-manager does all the heavy lifting.
It subscribes to changes to Gateway resources (using the gateway-shim component.)
For any Gateway it finds, it looks for any TLS listeners, and the associated tls.certificateRefs.
Note that while Gateway API supports multiple refs here, Envoy Gateway only uses one.
cert-manager also looks at the hostname of the listener to figure out which hosts the certificate is expected to cover.
More than one listener can use the same certificate Secret, which means cert-manager needs to find all listeners using the same Secret before deciding what to do.
If the certificatRef points to a valid certificate, given the hostnames found in listeners, cert-manager has nothing to do.
If there is no valid certificate, or it is about to expire, cert-manager’s gateway-shim creates a Certificate resource, or updates the existing one.
cert-manager then follows the Certificate Lifecycle.
To know how to issue the certificate, an ClusterIssuer is configured, and referenced through annotations on the Gateway resource, which you did above.
Once a matching ClusterIssuer is found, that plugin does what needs to be done to acquire a signed certificate.
In the case of the ACME protocol (used by Let’s Encrypt,) cert-manager can also use an HTTP Gateway to solve the HTTP-01 challenge type. This is the other side of cert-manager’s Gateway API support: the ACME issuer creates a temporary HTTPRoute, lets the ACME server(s) query it, and deletes it again.
cert-manager then updates the Secret that the Gateway’s listener points to in tls.certificateRefs.
Envoy Gateway picks up that the Secret has changed, and reloads the corresponding Envoy Proxy Deployments with the new private key and certificate.
As you can imagine, cert-manager requires quite broad permissions to update Secrets in any namespace, so the security-minded reader may want to look at the RBAC resources the Helm chart creates.
Using the ACME Issuer With Let’s Encrypt and HTTP-01
We will start using the Let’s Encrypt staging environment, to spare their production environment. Our Gateway already contains an HTTP listener, so we will use that for the HTTP-01 challenges.
$ CERT_MANAGER_CONTACT_EMAIL=$(git config user.email) # Or whatever...
$ kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
server: https://acme-staging-v02.api.letsencrypt.org/directory
email: "$CERT_MANAGER_CONTACT_EMAIL"
privateKeySecretRef:
name: letsencrypt-staging-account-key
solvers:
- http01:
gatewayHTTPRoute:
parentRefs:
- kind: Gateway
name: eg
namespace: default
EOF
The important parts are
- using
spec.acmewith a server URI and contact email address, and - referencing our plain HTTP gateway so the challenge HTTPRoute is attached to the right place.
Check the account registration process using the Ready condition:
$ kubectl wait --for=condition=Ready clusterissuer/letsencrypt-staging
$ kubectl describe clusterissuer/letsencrypt-staging
...
Status:
Acme:
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/123456789
Conditions:
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
...
Now we’re ready to update the Gateway annotation to use this issuer instead:
$ kubectl annotate --overwrite gateway/eg cert-manager.io/cluster-issuer=letsencrypt-staging
The Gateway should be picked up by cert-manager, which will create a new certificate for you, and replace the Secret.
You should see a new CertificateRequest to track:
$ kubectl get certificaterequest
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
eg-https-xxxxx True True selfsigned system:serviceaccount:cert-manager:cert-manager 42m
eg-https-xxxxx True True letsencrypt-staging system:serviceaccount:cert-manager:cert-manager 42m
Testing The Gateway
We still require the -k flag, since the Let’s Encrypt staging environment CA is not widely trusted.
$ curl -kv -HHost:www.example.com https://127.0.0.1/get
...
* Server certificate:
* subject: CN=Hello World!
* issuer: C=US; O=(STAGING) Let's Encrypt; CN=(STAGING) Ersatz Edamame E1
...
< HTTP/2 200
...
Using The Let’s Encrypt Production Environment
Changing to the production environment is just a matter of replacing the server URI:
$ CERT_MANAGER_CONTACT_EMAIL=$(git config user.email) # Or whatever...
$ kubectl apply -f - <<EOF
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory # Removed "-staging".
email: "$CERT_MANAGER_CONTACT_EMAIL"
privateKeySecretRef:
name: letsencrypt-account-key # Removed "-staging".
solvers:
- http01:
gatewayHTTPRoute:
parentRefs:
- kind: Gateway
name: eg
namespace: default
EOF
And now you can update the Gateway listener to point to letsencrypt instead:
$ kubectl annotate --overwrite gateway/eg cert-manager.io/cluster-issuer=letsencrypt
As before, track it by looking at CertificateRequests.
Testing The Gateway
Once the certificate has been replaced, we should finally be able to get rid of the -k flag:
$ curl -v -HHost:www.example.com --resolve www.example.com:127.0.0.1 https://www.example.com/get
...
* Server certificate:
* subject: CN=Hello World!
* issuer: C=US; O=Let's Encrypt; CN=R3
...
< HTTP/2 200
...
Collecting Garbage
You probably want to set the cert-manager.io/revision-history-limit annotation on your Gateway to make cert-manager prune the CertificateRequest history.
cert-manager deletes unused Certificate resources, and they are updated in-place when possible, so there should be no need for cleaning up Certificate resources.
The deletion is based on whether a Gateway still holds a tls.certificateRefs that requires the Certificate.
If you remove a TLS listener from a Gateway, you may still have a Secret lingering. cert-manager can clean it up using a flag.
Issuer Namespaces
We have used ClusterIssuer resources in this tutorial. They are not bound to any namespace, and will read annotations from Gateways in any namespace. You could also use Issuer, which is bound to a namespace. This is useful e.g. if you want to use different ACME accounts for different namespaces.
If you change the issuer kind, you also need to change the annotation key from cert-manager.io/clusterissuer to cert-manager.io/issuer.
Using ExternalDNS
The ExternalDNS controller maintains DNS records based on Kubernetes resources. Together with cert-manager, this can be used to fully automate hostname management. It can use various source resources, among them Gateway Routes. Just specify a Gateway Route resource, let ExternalDNS create the domain records, and then cert-manager the TLS certificate.
The tutorial on Gateway API uses kubectl. They also have a Helm chart, which is easier to customize. The only thing relevant to Envoy Gateway is to set the sources:
# values.yaml
sources:
- gateway-httproute
- gateway-grpcroute
- gateway-tcproute
- gateway-tlsroute
- gateway-udproute
Monitoring Progress / Troubleshooting
You can monitor progress in several ways:
The Issuer has a Ready condition (though this is rather boring for the selfSigned type):
$ kubectl get issuer --all-namespaces
NAMESPACE NAME READY AGE
default selfsigned True 42m
The Gateway will say when it has an invalid certificate:
$ kubectl describe gateway/eg
...
Conditions:
Message: Secret default/eg-https does not exist.
Reason: InvalidCertificateRef
Status: False
Type: ResolvedRefs
...
Message: Listener is invalid, see other Conditions for details.
Reason: Invalid
Status: False
Type: Programmed
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BadConfig 42m cert-manager-gateway-shim Skipped a listener block: spec.listeners[1].hostname: Required value: the hostname cannot be empty
The main question is if cert-manager has picked up on the Gateway.
I.e., has it created a Certificate for it?
The above describe contains an event from cert-manager-gateway-shim telling you of one such issue.
Be aware that if you have a non-TLS listener in the Gateway, like we did, there will be events saying it is not eligible, which is of course expected.
Another option is looking at Deployment logs.
The cert-manager logs are not very verbose by default, but setting the Helm value global.logLevel to 6 will enable all debug logs (the default is 2.)
This will make them verbose enough to say why a Gateway wasn’t considered (e.g. missing hostname, or tls.mode is not Terminate.)
$ kubectl logs -n cert-manager deployment/cert-manager
...
Simply listing Certificate resources may be useful, even if it just gives a yes/no answer:
$ kubectl get certificate --all-namespaces
NAMESPACE NAME READY SECRET AGE
default eg-https True eg-https 42m
If there is a Certificate, then the gateway-shim has recognized the Gateway.
But is there a CertificateRequest for it?
(BTW, don’t confuse this with a CertificateSigningRequest, which is a Kubernetes core resource type representing the same thing.)
$ kubectl get certificaterequest --all-namespaces
NAMESPACE NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
default eg-https-xxxxx True True selfsigned system:serviceaccount:cert-manager:cert-manager 42m
The ACME issuer also has Order and Challenge resources to watch:
$ kubectl get order --all-namespaces -o wide
NAME STATE ISSUER REASON AGE
order.acme.cert-manager.io/envoy-https-xxxxx-123456789 pending letsencrypt-staging 42m
$ kubectl get challenge --all-namespaces
NAME STATE DOMAIN AGE
challenge.acme.cert-manager.io/envoy-https-xxxxx-123456789-1234567890 pending www.example.com 42m
Using kubetctl get -o wide or kubectl describe on the Challenge will explain its state more.
$ kubectl get order --all-namespaces -o wide
NAME STATE ISSUER REASON AGE
order.acme.cert-manager.io/envoy-https-xxxxx-123456789 valid letsencrypt-staging 42m
Finally, since cert-manager creates the Secret referenced by the Gateway listener as its last step, we can also look for that:
$ kubectl get secret secret/eg-https
NAME TYPE DATA AGE
eg-https kubernetes.io/tls 3 42m
Clean Up
- Uninstall cert-manager:
helm uninstall --namespace cert-manager cert-manager - Delete the
cert-managernamespace:kubectl delete namespace/cert-manager - Delete the
httpslistener fromgateway/eg. - Delete
secret/eg-https.
See Also
28 - Visualising metrics using Grafana
Envoy Gateway provides support for exposing Envoy Proxy metrics to a Prometheus instance. This guide shows you how to visualise the metrics exposed to prometheus using grafana.
Prerequisites
Follow the steps from the Quickstart Guide to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Follow the steps from the Proxy Observability to enable prometheus metrics.
Prometheus is used to scrape metrics from the Envoy Proxy instances. Install Prometheus:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update
helm upgrade --install prometheus prometheus-community/prometheus -n monitoring --create-namespace
Grafana is used to visualise the metrics exposed by the envoy proxy instances. Install Grafana:
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm upgrade --install grafana grafana/grafana -f https://raw.githubusercontent.com/envoyproxy/gateway/v0.6.0/examples/grafana/helm-values.yaml -n monitoring --create-namespace
Expose endpoints:
GRAFANA_IP=$(kubectl get svc grafana -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
Connecting Grafana with Prometheus datasource
To visualise metrics from Prometheus, we have to connect Grafana with Prometheus. If you installed Grafana from the command from prerequisites sections, the prometheus datasource should be already configured.
You can also add the data source manually by following the instructions from Grafana Docs.
Accessing Grafana
You can access the Grafana instance by visiting http://{GRAFANA_IP}, derived in prerequisites.
To log in to Grafana, use the credentials admin:admin.
Envoy Gateway has examples of dashboard for you to get started:
Envoy Proxy Global
Envoy Clusters
Envoy Pod Resources
You can load the above dashboards in your Grafana to get started. Please refer to Grafana docs for importing dashboards.