This project is under active development. Many features are not complete. We would love for you to Get Involved!
Envoy Gateway is an open source project for managing Envoy Proxy as a standalone or Kubernetes-based application
gateway. Gateway API resources are used to dynamically provision and configure the managed Envoy Proxies.
Ready to get started?
1 - Tasks
Learn Envoy Gateway hands-on through tasks
1.1 - Quickstart
Get started with Envoy Gateway in a few simple steps.
This “quick start” will help you get started with Envoy Gateway in a few simple steps.
Note: In case your Kubernetes cluster, does not have a LoadBalancer implementation, we recommend installing one
so the Gateway resource has an Address associated with it. We recommend using MetalLB.
Note: quickstart.yaml defines that Envoy Gateway will listen for
traffic on port 80 on its globally-routable IP address, to make it easy to use
browsers to test Envoy Gateway. When Envoy Gateway sees that its Listener is
using a privileged port (<1024), it will map this internally to an
unprivileged port, so that Envoy Gateway doesn’t need additional privileges.
It’s important to be aware of this mapping, since you may need to take it into
consideration when debugging.
Testing the Configuration
Get the name of the Envoy service created the by the example Gateway:
exportENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
You can also test the same functionality by sending traffic to the External IP. To get the external IP of the
Envoy service, run:
exportGATEWAY_HOST=$(kubectl get svc/${ENVOY_SERVICE} -n envoy-gateway-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
In certain environments, the load balancer may be exposed using a hostname, instead of an IP address. If so, replace
ip in the above command with hostname.
Review the Tasks section for the scenario matching your use case. The Envoy Gateway tasks are organized by category: traffic management, security, extensibility, observability, and operations.
Clean-Up
Use the steps in this section to uninstall everything from the quickstart.
Delete the GatewayClass, Gateway, HTTPRoute and Example App:
Checkout the Developer Guide to get involved in the project.
1.2 - Traffic
This section includes Traffic Management tasks.
1.2.1 - Circuit Breakers
Envoy circuit breakers can be used to fail quickly and apply back-pressure in response to upstream service degradation.
Envoy Gateway supports the following circuit breaker thresholds:
Concurrent Connections: limit the connections that Envoy can establish to the upstream service. When this threshold is met, new connections will not be established, and some requests will be queued until an existing connection becomes available.
Concurrent Requests: limit on concurrent requests in-flight from Envoy to the upstream service. When this threshold is met, requests will be queued.
Pending Requests: limit the pending request queue size. When this threshold is met, overflowing requests will be terminated with a 503 status code.
Envoy’s circuit breakers are distributed: counters are not synchronized across different Envoy processes. The default Envoy and Envoy Gateway circuit breaker threshold values (1024) may be too strict for high-throughput systems.
Envoy Gateway introduces a new CRD called BackendTrafficPolicy that allows the user to describe their desired circuit breaker thresholds.
This instantiated resource can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Note: There are distinct circuit breaker counters for each BackendReference in an xRoute rule. Even if a BackendTrafficPolicy targets a Gateway, each BackendReference in that gateway still has separate circuit breaker counter.
Prerequisites
Install Envoy Gateway
Follow the installation step from the Quickstart to install Envoy Gateway and sample resources.
Install the hey load testing tool
The hey CLI will be used to generate load and measure response times. Follow the installation instruction from the Hey project docs.
Test and customize circuit breaker settings
This example will simulate a degraded backend that responds within 10 seconds by adding the ?delay=10s query parameter to API calls. The hey tool will be used to generate 100 concurrent requests.
The default circuit breaker threshold (1024) is not met. As a result, requests do not overflow: all requests are proxied upstream and both Envoy and clients wait for 10s.
In order to fail fast, apply a BackendTrafficPolicy that limits concurrent requests to 10 and pending requests to 0.
With the new circuit breaker settings, and due to the slowness of the backend, only the first 10 concurrent requests were proxied, while the other 90 overflowed.
The ClientTrafficPolicy API allows system administrators to configure
the behavior for how the Envoy Proxy server behaves with downstream clients.
Motivation
This API was added as a new policy attachment resource that can be applied to Gateway resources and it is meant to hold settings for configuring behavior of the connection between the downstream client and Envoy Proxy listener. It is the counterpart to the BackendTrafficPolicy API resource.
Quickstart
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
* Trying 172.18.255.202:80...
* Connected to 172.18.255.202 (172.18.255.202) port 80(#0)> GET /get HTTP/1.1
> Host: www.example.com
> User-Agent: curl/8.1.2
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0curl: (56) Recv failure: Connection reset by peer
Curl the example app through Envoy proxy once again, now sending HAProxy PROXY protocol header at the beginning of the connection with –haproxy-protocol flag:
You should expect 200 response status, see that X-Forwarded-Proto was preserved and X-Envoy-External-Address was set to the leftmost address in the X-Forwarded-For header:
This feature allows you to limit the take taken by the Envoy Proxy fleet to receive the entire request from the client, which is useful in preventing certain clients from consuming too much memory in Envoy
This example configures the HTTP request timeout for the client, please check out the details here.
The connection limit features allows users to limit the number of concurrently active TCP connections on a Gateway or a Listener.
When the connection limit is reached, new connections are closed immediately by Envoy proxy. It’s possible to configure a delay for connection rejection.
Users may want to limit the number of connections for several reasons:
Protect resources like CPU and Memory.
Ensure that different listeners can receive a fair share of global resources.
Protect from malicious activity like DoS attacks.
Envoy Gateway introduces a new CRD called Client Traffic Policy that allows the user to describe their desired connection limit settings.
This instantiated resource can be linked to a Gateway.
The Envoy connection limit implementation is distributed: counters are not synchronized between different envoy proxies.
When a Client Traffic Policy is attached to a gateway, the connection limit will apply differently based on the
Listener protocol in use:
HTTP: all HTTP listeners in a Gateway will share a common connection counter, and a limit defined by the policy.
HTTPS/TLS: each HTTPS/TLS listener will have a dedicated connection counter, and a limit defined by the policy.
Prerequisites
Install Envoy Gateway
Follow the steps from the Quickstart to install Envoy Gateway and the HTTPRoute example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Install the hey load testing tool
The hey CLI will be used to generate load and measure response times. Follow the installation instruction from the Hey project docs.
Test and customize connection limit settings
This example we use hey to open 10 connections and execute 1 RPS per connection for 10 seconds.
With the new connection limit, only 5 of 10 connections are established, and so only 50 requests succeed.
1.2.4 - Fault Injection
Envoy fault injection can be used to inject delays and abort requests to mimic failure scenarios such as service failures and overloads.
Envoy Gateway supports the following fault scenarios:
delay fault: inject a custom fixed delay into the request with a certain probability to simulate delay failures.
abort fault: inject a custom response code into the response with a certain probability to simulate abort failures.
Envoy Gateway introduces a new CRD called BackendTrafficPolicy that allows the user to describe their desired fault scenarios.
This instantiated resource can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
For GRPC - follow the steps from the GRPC Routing example.
Before proceeding, you should be able to query the example backend using HTTP or GRPC.
Install the hey load testing tool
The hey CLI will be used to generate load and measure response times. Follow the installation instruction from the Hey project docs.
Configuration
Allow requests with a valid faultInjection by creating an BackendTrafficPolicy and attaching it to the example HTTPRoute or GRPCRoute.
Two HTTPRoute resources were created, one for /foo and another for /bar. fault-injection-abort BackendTrafficPolicy has been created and targeted HTTPRoute foo to abort requests for /foo. fault-injection-delay BackendTrafficPolicy has been created and targeted HTTPRoute foo to delay 2s requests for /bar.
Verify the HTTPRoute configuration and status:
kubectl get httproute/foo -o yaml
kubectl get httproute/bar -o yaml
Verify the BackendTrafficPolicy configuration:
kubectl get backendtrafficpolicy/fault-injection-50-percent-abort -o yaml
kubectl get backendtrafficpolicy/fault-injection-delay -o yaml
The Gateway API provides an optional Addresses field through which Envoy Gateway can set addresses for Envoy Proxy Service.
Depending on the Service Type, the addresses of gateway can be used as:
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
External IPs
Using the addresses in Gateway.Spec.Addresses as the External IPs of Envoy Proxy Service,
this will require the address to be of type IPAddress and the ServiceType to be of LoadBalancer or NodePort.
The Envoy Gateway deploys Envoy Proxy Service as LoadBalancer by default,
so you can set the address of the Gateway directly (the address settings here are for reference only):
NAME CLASS ADDRESS PROGRAMMED AGE
eg eg 1.2.3.4 True 14m
Verify the Envoy Proxy Service status:
kubectl get service -n envoy-gateway-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy-default-eg-64656661 LoadBalancer 10.96.236.219 1.2.3.4 80:31017/TCP 15m
envoy-gateway ClusterIP 10.96.192.76 <none> 18000/TCP 15m
envoy-gateway-metrics-service ClusterIP 10.96.124.73 <none> 8443/TCP 15m
Note: If the Gateway.Spec.Addresses is explicitly set, it will be the only addresses that populates the Gateway status.
Cluster IP
Using the addresses in Gateway.Spec.Addresses as the Cluster IP of Envoy Proxy Service,
this will require the address to be of type IPAddress and the ServiceType to be of ClusterIP.
1.2.6 - Gateway API Support
As mentioned in the system design document, Envoy Gateway’s managed data plane is configured dynamically through
Kubernetes resources, primarily Gateway API objects. Envoy Gateway supports configuration using the following Gateway API resources.
GatewayClass
A GatewayClass represents a “class” of gateways, i.e. which Gateways should be managed by Envoy Gateway.
Envoy Gateway supports managing a single GatewayClass resource that matches its configured controllerName and
follows Gateway API guidelines for resolving conflicts when multiple GatewayClasses exist with a matching
controllerName.
When a Gateway resource is created that references the managed GatewayClass, Envoy Gateway will create and manage a
new Envoy Proxy deployment. Gateway API resources that reference this Gateway will configure this managed Envoy Proxy
deployment.
HTTPRoute
A HTTPRoute configures routing of HTTP traffic through one or more Gateways. The following HTTPRoute filters are
supported by Envoy Gateway:
requestHeaderModifier: RequestHeaderModifiers
can be used to modify or add request headers before the request is proxied to its destination.
responseHeaderModifier: ResponseHeaderModifiers
can be used to modify or add response headers before the response is sent back to the client.
requestMirror: RequestMirrors
configure destinations where the requests should also be mirrored to. Responses to mirrored requests will be ignored.
requestRedirect: RequestRedirects
configure policied for how requests that match the HTTPRoute should be modified and then redirected.
urlRewrite: UrlRewrites
allow for modification of the request’s hostname and path before it is proxied to its destination.
extensionRef: ExtensionRefs are used by Envoy Gateway to implement extended filters. Currently, Envoy Gateway
supports rate limiting and request authentication filters. For more information about these filters, refer to the
rate limiting and request authentication documentation.
Notes:
The only BackendRef kind supported by Envoy Gateway is a Service. Routing traffic to other destinations such
as arbitrary URLs is not possible.
The filters field within HTTPBackendRef is not supported.
TCPRoute
A TCPRoute configures routing of raw TCP traffic through one or more Gateways. Traffic can be forwarded to the
desired BackendRefs based on a TCP port number.
Note: A TCPRoute only supports proxying in non-transparent mode, i.e. the backend will see the source IP and port of
the Envoy Proxy instance instead of the client.
UDPRoute
A UDPRoute configures routing of raw UDP traffic through one or more Gateways. Traffic can be forwarded to the
desired BackendRefs based on a UDP port number.
Note: Similar to TCPRoutes, UDPRoutes only support proxying in non-transparent mode i.e. the backend will see the
source IP and port of the Envoy Proxy instance instead of the client.
GRPCRoute
A GRPCRoute configures routing of gRPC requests through one or more Gateways. They offer request matching by
hostname, gRPC service, gRPC method, or HTTP/2 Header. Envoy Gateway supports the following filters on GRPCRoutes to
provide additional traffic processing:
requestHeaderModifier: RequestHeaderModifiers
can be used to modify or add request headers before the request is proxied to its destination.
responseHeaderModifier: ResponseHeaderModifiers
can be used to modify or add response headers before the response is sent back to the client.
requestMirror: RequestMirrors
configure destinations where the requests should also be mirrored to. Responses to mirrored requests will be ignored.
Notes:
The only BackendRef kind supported by Envoy Gateway is a Service. Routing traffic to other
destinations such as arbitrary URLs is not currently possible.
The filters field within HTTPBackendRef is not supported.
TLSRoute
A TLSRoute configures routing of TCP traffic through one or more Gateways. However, unlike TCPRoutes, TLSRoutes
can match against TLS-specific metadata.
ReferenceGrant
A ReferenceGrant is used to allow a resource to reference another resource in a different namespace. Normally an
HTTPRoute created in namespace foo is not allowed to reference a Service in namespace bar. A ReferenceGrant permits
these types of cross-namespace references. Envoy Gateway supports the following ReferenceGrant use-cases:
Allowing an HTTPRoute, GRPCRoute, TLSRoute, UDPRoute, or TCPRoute to reference a Service in a different namespace.
Allowing an HTTPRoute’s requestMirror filter to include a BackendRef that references a Service in a different
namespace.
Allowing a Gateway’s SecretObjectReference to reference a secret in a different namespace.
1.2.7 - Global Rate Limit
Rate limit is a feature that allows the user to limit the number of incoming requests to a predefined value based on attributes within the traffic flow.
Here are some reasons why you may want to implement Rate limits
To prevent malicious activity such as DDoS attacks.
To prevent applications and its resources (such as a database) from getting overloaded.
Global rate limiting applies a shared rate limit to the traffic flowing through all the instances of Envoy proxies where it is configured.
i.e. if the data plane has 2 replicas of Envoy running, and the rate limit is 10 requests/second, this limit is shared and will be hit
if 5 requests pass through the first replica and 5 requests pass through the second replica within the same second.
Envoy Gateway introduces a new CRD called BackendTrafficPolicy that allows the user to describe their rate limit intent. This instantiated resource
can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Note: Limit is applied per route. Even if a BackendTrafficPolicy targets a gateway, each route in that gateway
still has a separate rate limit bucket. For example, if a gateway has 2 routes, and the limit is 100r/s, then each route
has its own 100r/s rate limit bucket.
Prerequisites
Install Envoy Gateway
Follow the steps from the Quickstart to install Envoy Gateway and the HTTPRoute example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Install Redis
The global rate limit feature is based on Envoy Ratelimit which requires a Redis instance as its caching layer.
Lets install a Redis deployment in the redis-system namespce.
The default installation of Envoy Gateway installs a default EnvoyGateway configuration and attaches it
using a ConfigMap. In the next step, we will update this resource to enable rate limit in Envoy Gateway
as well as configure the URL for the Redis instance used for Global rate limiting.
Here is an example of a rate limit implemented by the application developer to limit a specific user by matching on a custom x-user-id header
with a value set to one.
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-ratelimit -o yaml
Get the Gateway’s address:
exportGATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Let’s query ratelimit.example/get 4 times. We should receive a 200 response from the example Gateway for the first 3 requests
and then receive a 429 status code for the 4th request since the limit is set at 3 requests/Hour for the request which contains the header x-user-id
and value one.
for i in {1..4};do curl -I --header "Host: ratelimit.example" --header "x-user-id: one" http://${GATEWAY_HOST}/get ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:31 GMT
content-length: 460
x-envoy-upstream-service-time: 4
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:32 GMT
content-length: 460
x-envoy-upstream-service-time: 2
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:33 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Wed, 08 Feb 2023 02:33:34 GMT
server: envoy
transfer-encoding: chunked
You should be able to send requests with the x-user-id header and a different value and receive successful responses from the server.
for i in {1..4};do curl -I --header "Host: ratelimit.example" --header "x-user-id: two" http://${GATEWAY_HOST}/get ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:36 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:37 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:38 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:39 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
Rate Limit Distinct Users
Here is an example of a rate limit implemented by the application developer to limit distinct users who can be differentiated based on the
value in the x-user-id header. Here, user one (recognised from the traffic flow using the header x-user-id and value one) will be rate limited at 3 requests/hour
and so will user two (recognised from the traffic flow using the header x-user-id and value two).
Lets run the same command again with the header x-user-id and value one set in the request. We should the first 3 requests succeeding and
the 4th request being rate limited.
for i in {1..4};do curl -I --header "Host: ratelimit.example" --header "x-user-id: one" http://${GATEWAY_HOST}/get ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:31 GMT
content-length: 460
x-envoy-upstream-service-time: 4
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:32 GMT
content-length: 460
x-envoy-upstream-service-time: 2
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:33 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Wed, 08 Feb 2023 02:33:34 GMT
server: envoy
transfer-encoding: chunked
You should see the same behavior when the value for header x-user-id is set to two and 4 requests are sent.
for i in {1..4};do curl -I --header "Host: ratelimit.example" --header "x-user-id: two" http://${GATEWAY_HOST}/get ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:31 GMT
content-length: 460
x-envoy-upstream-service-time: 4
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:32 GMT
content-length: 460
x-envoy-upstream-service-time: 2
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:33 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Wed, 08 Feb 2023 02:33:34 GMT
server: envoy
transfer-encoding: chunked
Rate Limit All Requests
This example shows you how to rate limit all requests matching the HTTPRoute rule at 3 requests/Hour by leaving the clientSelectors field unset.
for i in {1..4};do curl -I --header "Host: ratelimit.example" http://${GATEWAY_HOST}/get ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:31 GMT
content-length: 460
x-envoy-upstream-service-time: 4
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:32 GMT
content-length: 460
x-envoy-upstream-service-time: 2
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:33 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Wed, 08 Feb 2023 02:33:34 GMT
server: envoy
transfer-encoding: chunked
Rate Limit Client IP Addresses
Here is an example of a rate limit implemented by the application developer to limit distinct users who can be differentiated based on their
IP address (also reflected in the X-Forwarded-For header).
Note: EG supports two kinds of rate limit for the IP address: Exact and Distinct.
Exact means that all IP addresses within the specified Source IP CIDR share the same rate limit bucket.
Distinct means that each IP address within the specified Source IP CIDR has its own rate limit bucket.
for i in {1..4};do curl -I --header "Host: ratelimit.example" http://${GATEWAY_HOST}/get ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 28 Mar 2023 08:28:45 GMT
content-length: 512
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 28 Mar 2023 08:28:46 GMT
content-length: 512
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Tue, 28 Mar 2023 08:28:48 GMT
content-length: 512
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Tue, 28 Mar 2023 08:28:48 GMT
server: envoy
transfer-encoding: chunked
Rate Limit Jwt Claims
Here is an example of a rate limit implemented by the application developer to limit distinct users who can be differentiated based on the value of the Jwt claims carried.
for i in {1..4};do curl -I --header "Host: ratelimit.example" --header "Authorization: Bearer $TOKEN" http://${GATEWAY_HOST}/foo ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:00:25 GMT
content-length: 561
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:00:26 GMT
content-length: 561
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:00:27 GMT
content-length: 561
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Mon, 12 Jun 2023 12:00:28 GMT
server: envoy
transfer-encoding: chunked
No Rate Limit by carrying TOKEN1
for i in {1..4};do curl -I --header "Host: ratelimit.example" --header "Authorization: Bearer $TOKEN1" http://${GATEWAY_HOST}/foo ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:02:34 GMT
content-length: 556
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:02:35 GMT
content-length: 556
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:02:36 GMT
content-length: 556
x-envoy-upstream-service-time: 1
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Mon, 12 Jun 2023 12:02:37 GMT
content-length: 556
x-envoy-upstream-service-time: 0
server: envoy
(Optional) Editing Kubernetes Resources settings for the Rate Limit Service
The default installation of Envoy Gateway installs a default EnvoyGateway configuration and provides the initial rate
limit kubernetes resources settings. such as replicas is 1, requests resources cpu is 100m, memory is 512Mi. the others
like container image, securityContext, env and pod annotations and securityContext can be modified by modifying the ConfigMap.
tls.certificateRef set the client certificate for redis server TLS connections.
The GRPCRoute resource allows users to configure gRPC routing by matching HTTP/2 traffic and forwarding it to backend gRPC servers.
To learn more about gRPC routing, refer to the Gateway API documentation.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
The manifest installs a GatewayClass, Gateway, a Deployment, a Service, and a GRPCRoute resource.
The GatewayClass is a cluster-scoped resource that represents a class of Gateways that can be instantiated.
Note: Envoy Gateway is configured by default to manage a GatewayClass with
controllerName: gateway.envoyproxy.io/gatewayclass-controller.
Verification
Check the status of the GatewayClass:
kubectl get gc --selector=example=grpc-routing
The status should reflect “Accepted=True”, indicating Envoy Gateway is managing the GatewayClass.
A Gateway represents configuration of infrastructure. When a Gateway is created, Envoy proxy infrastructure is
provisioned or configured by Envoy Gateway. The gatewayClassName defines the name of a GatewayClass used by this
Gateway. Check the status of the Gateway:
kubectl get gateways --selector=example=grpc-routing
The status should reflect “Ready=True”, indicating the Envoy proxy infrastructure has been provisioned. The status also
provides the address of the Gateway. This address is used later to test connectivity to proxied backend services.
Check the status of the GRPCRoute:
kubectl get grpcroutes --selector=example=grpc-routing -o yaml
The status for the GRPCRoute should surface “Accepted=True” and a parentRef that references the example Gateway.
The example-route matches any traffic for “grpc-example.com” and forwards it to the “yages” Service.
Testing the Configuration
Before testing GRPC routing to the yages backend, get the Gateway’s address.
exportGATEWAY_HOST=$(kubectl get gateway/example-gateway -o jsonpath='{.status.addresses[0].value}')
Test GRPC routing to the yages backend using the grpcurl command.
Envoy Gateway also supports gRPC-Web requests for this configuration. The below curl command can be used to send a grpc-Web request with over HTTP/2. You should receive the same response seen in the previous command.
The data in the body AAAAAAA= is a base64 encoded representation of an empty message (data length 0) that the Ping RPC accepts.
The matches field can be used to restrict the route to a specific set of requests based on GRPC’s service and/or method names.
It supports two match types: Exact and RegularExpression.
Exact
Exact match is the default match type.
The following example shows how to match a request based on the service and method names for grpc.reflection.v1alpha.ServerReflection/ServerReflectionInfo,
as well as a match for all services with a method name Ping which matches yages.Echo/Ping in our deployment.
The following example shows how to match a request based on the service and method names
with match type RegularExpression. It matches all the services and methods with pattern
/.*.Echo/Pin.+, which matches yages.Echo/Ping in our deployment.
The HTTPRoute resource can issue redirects to clients or rewrite paths sent upstream using filters. Note that
HTTPRoute rules cannot use both filter types at once. Currently, Envoy Gateway only supports coreHTTPRoute filters which consist of RequestRedirect and RequestHeaderModifier at the time of this writing. To
learn more about HTTP routing, refer to the Gateway API documentation.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTPS.
Redirects
Redirects return HTTP 3XX responses to a client, instructing it to retrieve a different resource. A
RequestRedirect filter instructs Gateways to emit a redirect response to requests that match the rule.
For example, to issue a permanent redirect (301) from HTTP to HTTPS, configure requestRedirect.statusCode=301 and
requestRedirect.scheme="https":
If you followed the steps in the Secure Gateways task, you should be able to curl the redirect
location.
HTTP –> HTTPS
Listeners expose the TLS setting on a per domain or subdomain basis. TLS settings of a listener are applied to all domains that satisfy the hostname criteria.
Create a root certificate and private key to sign certificates:
Path redirects use an HTTP Path Modifier to replace either entire paths or path prefixes. For example, the HTTPRoute
below will issue a 302 redirect to all path.redirect.example requests whose path begins with /get to /status/200.
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-filter-path-redirect -o yaml
Querying path.redirect.example should result in a 302 response from the example Gateway and a redirect location
containing the configured redirect path.
You should receive a 302 with a redirect location of http://path.redirect.example/status/200.
1.2.10 - HTTP Request Headers
The HTTPRoute resource can modify the headers of a request before forwarding it to the upstream service. HTTPRoute
rules cannot use both filter types at once. Currently, Envoy Gateway only supports coreHTTPRoute filters which
consist of RequestRedirect and RequestHeaderModifier at the time of this writing. To learn more about HTTP routing,
refer to the Gateway API documentation.
A RequestHeaderModifier filter instructs Gateways to modify the headers in requests that match the rule
before forwarding the request upstream. Note that the RequestHeaderModifier filter will only modify headers before the
request is sent from Envoy to the upstream service and will not affect response headers returned to the downstream
client.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Adding Request Headers
The RequestHeaderModifier filter can add new headers to a request before it is sent to the upstream. If the request
does not have the header configured by the filter, then that header will be added to the request. If the request already
has the header configured by the filter, then the value of the header in the filter will be appended to the value of the
header in the request.
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-headers -o yaml
Get the Gateway’s address:
exportGATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the upstream example app received the header add-header with the value:
something,foo
Setting headers is similar to adding headers. If the request does not have the header configured by the filter, then it
will be added, but unlike adding request headers which will append the value of the header if
the request already contains it, setting a header will cause the value to be replaced by the value configured in the
filter.
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the upstream example app received the header add-header with the original value
something replaced by foo.
Headers can be removed from a request by simply supplying a list of header names.
Setting headers is similar to adding headers. If the request does not have the header configured by the filter, then it
will be added, but unlike adding request headers which will append the value of the header if
the request already contains it, setting a header will cause the value to be replaced by the value configured in the
filter.
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the upstream example app received the header add-header, but the header
remove-header that was sent by curl was removed before the upstream received the request.
The HTTPRoute resource can modify the headers of a response before responding it to the downstream service. To learn
more about HTTP routing, refer to the Gateway API documentation.
A ResponseHeaderModifier filter instructs Gateways to modify the headers in responses that match the
rule before responding to the downstream. Note that the ResponseHeaderModifier filter will only modify headers before
the response is returned from Envoy to the downstream client and will not affect request headers forwarding to the
upstream service.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Adding Response Headers
The ResponseHeaderModifier filter can add new headers to a response before it is sent to the upstream. If the response
does not have the header configured by the filter, then that header will be added to the response. If the response
already has the header configured by the filter, then the value of the header in the filter will be appended to the
value of the header in the response.
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-headers -o yaml
Get the Gateway’s address:
exportGATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the downstream client received the header add-header with the value: foo
Setting headers is similar to adding headers. If the response does not have the header configured by the filter, then it
will be added, but unlike adding response headers which will append the value of the header
if the response already contains it, setting a header will cause the value to be replaced by the value configured in the
filter.
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the downstream client received the header set-header with the original value value1
replaced by foo.
Headers can be removed from a response by simply supplying a list of header names.
Setting headers is similar to adding headers. If the response does not have the header configured by the filter, then it
will be added, but unlike adding response headers which will append the value of the header
if the response already contains it, setting a header will cause the value to be replaced by the value configured in the
filter.
Querying headers.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate that the header remove-header that was sent by curl was removed before the upstream
received the response.
The HTTPRoute resource allows users to configure HTTP routing by matching HTTP traffic and forwarding it to
Kubernetes backends. Currently, the only supported backend supported by Envoy Gateway is a Service resource. This task
shows how to route traffic based on host, header, and path fields and forward the traffic to different Kubernetes
Services. To learn more about HTTP routing, refer to the Gateway API documentation.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
The manifest installs a GatewayClass, Gateway, four Deployments, four Services, and three HTTPRoute resources.
The GatewayClass is a cluster-scoped resource that represents a class of Gateways that can be instantiated.
Note: Envoy Gateway is configured by default to manage a GatewayClass with
controllerName: gateway.envoyproxy.io/gatewayclass-controller.
Verification
Check the status of the GatewayClass:
kubectl get gc --selector=example=http-routing
The status should reflect “Accepted=True”, indicating Envoy Gateway is managing the GatewayClass.
A Gateway represents configuration of infrastructure. When a Gateway is created, Envoy proxy infrastructure is
provisioned or configured by Envoy Gateway. The gatewayClassName defines the name of a GatewayClass used by this
Gateway. Check the status of the Gateway:
kubectl get gateways --selector=example=http-routing
The status should reflect “Ready=True”, indicating the Envoy proxy infrastructure has been provisioned. The status also
provides the address of the Gateway. This address is used later to test connectivity to proxied backend
services.
The three HTTPRoute resources create routing rules on the Gateway. In order to receive traffic from a Gateway,
an HTTPRoute must be configured with parentRefs which reference the parent Gateway(s) that it should be attached to.
An HTTPRoute can match against a single set of hostnames. These hostnames are matched before any other matching
within the HTTPRoute takes place. Since example.com, foo.example.com, and bar.example.com are separate hosts with
different routing requirements, each is deployed as its own HTTPRoute - example-route, ``foo-route, and bar-route.
Check the status of the HTTPRoutes:
kubectl get httproutes --selector=example=http-routing -o yaml
The status for each HTTPRoute should surface “Accepted=True” and a parentRef that references the example Gateway.
The example-route matches any traffic for “example.com” and forwards it to the “example-svc” Service.
Testing the Configuration
Before testing HTTP routing to the example-svc backend, get the Gateway’s address.
exportGATEWAY_HOST=$(kubectl get gateway/example-gateway -o jsonpath='{.status.addresses[0].value}')
A 200 status code should be returned and the body should include "pod": "example-backend-*" indicating the traffic
was routed to the example backend service. If you change the hostname to a hostname not represented in any of the
HTTPRoutes, e.g. “www.example.com”, the HTTP traffic will not be routed and a 404 should be returned.
The foo-route matches any traffic for foo.example.com and applies its routing rules to forward the traffic to the
“foo-svc” Service. Since there is only one path prefix match for /login, only foo.example.com/login/* traffic will
be forwarded. Test HTTP routing to the foo-svc backend.
A 200 status code should be returned and the body should include "pod": "foo-backend-*" indicating the traffic
was routed to the foo backend service. Traffic to any other paths that do not begin with /login will not be matched by
this HTTPRoute. Test this by removing /login from the request.
The HTTP traffic will not be routed and a 404 should be returned.
Similarly, the bar-route HTTPRoute matches traffic for bar.example.com. All traffic for this hostname will be
evaluated against the routing rules. The most specific match will take precedence which means that any traffic with the
env:canary header will be forwarded to bar-svc-canary and if the header is missing or not canary then it’ll be
forwarded to bar-svc. Test HTTP routing to the bar-svc backend.
A 200 status code should be returned and the body should include "pod": "bar-canary-backend-*" indicating the
traffic was routed to the foo backend service.
JWT Claims Based Routing
Users can route to a specific backend by matching on JWT claims.
This can be achieved, by defining a SecurityPolicy with a jwt configuration that does the following
Converts jwt claims to headers, which can be used for header based routing
Sets the recomputeRoute field to true. This is required so that the incoming request matches on a fallback/catch all route where the JWT can be authenticated, the claims from the JWT can be converted to headers, and then the route match can be recomputed to match based on the updated headers.
For this feature to work please make sure
you have a fallback route rule defined, the backend for this route rule can be invalid.
The SecurityPolicy is applied to both the fallback route as well as the route with the claim header matches, to avoid spoofing.
The HTTPRouteTimeouts resource allows users to configure request timeouts and response timeouts for an HTTPRouteRule. This task shows how to configure timeouts.
request: Request specifies the maximum duration for a gateway to respond to an HTTP request.
backendRequest: BackendRequest specifies a timeout for an individual request from the gateway to a backend.
Note: The Request duration must be >= BackendRequest duration
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Verification
backend has the ability to delay responses; we use it as the backend to control response time.
request timeout
We configure the backend to delay responses by 3 seconds, then we set the request timeout to 4 seconds. Envoy Gateway will successfully respond to the request.
* Trying 127.0.0.1:80...
* Connected to 127.0.0.1 (127.0.0.1) port 80
> GET /?delay=3s HTTP/1.1
> Host: timeout.example.com
> User-Agent: curl/8.6.0
> Accept: */*
>< HTTP/1.1 504 Gateway Timeout
< content-length: 24
< content-type: text/plain
< date: Mon, 04 Mar 2024 02:35:03 GMT
<
* Connection #0 to host 127.0.0.1 left intact
upstream request timeout
1.2.14 - HTTP URL Rewrite
HTTPURLRewriteFilter defines a filter that modifies a request during forwarding. At most one of these filters may be
used on a Route rule. This MUST NOT be used on the same Route rule as a HTTPRequestRedirect filter.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Rewrite URL Prefix Path
You can configure to rewrite the prefix in the url like below. In this example, any curls to
http://${GATEWAY_HOST}/get/xxx will be rewritten to http://${GATEWAY_HOST}/replace/xxx.
You can see that the X-Envoy-Original-Path is /get/origin/path, but the actual path is /replace/origin/path.
Rewrite URL Full Path
You can configure to rewrite the fullpath in the url like below. In this example, any request sent to
http://${GATEWAY_HOST}/get/origin/path/xxxx will be rewritten to
http://${GATEWAY_HOST}/force/replace/fullpath.
You can see that the X-Envoy-Original-Path is /get/origin/path/extra, but the actual path is
/force/replace/fullpath.
Rewrite Host Name
You can configure to rewrite the hostname like below. In this example, any requests sent to
http://${GATEWAY_HOST}/get with --header "Host: path.rewrite.example" will rewrite host into envoygateway.io.
You can see that the X-Forwarded-Host is path.rewrite.example, but the actual host is envoygateway.io.
1.2.15 - HTTP3
This task will help you get started using HTTP3 using EG.
This task uses a self-signed CA, so it should be used for testing and demonstration purposes only.
Prerequisites
OpenSSL to generate TLS assets.
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
TLS Certificates
Generate the certificates and keys used by the Gateway to terminate client TLS connections.
Create a root certificate and private key to sign certificates:
The HTTPRoute resource allows one or more backendRefs to be provided. Requests will be routed to these upstreams. It is possible to divide the traffic between these backends using Traffic Splitting, but it is also possible to mirror requests to another Service instead. Request mirroring is accomplished using Gateway API’s HTTPRequestMirrorFilter on the HTTPRoute.
When requests are made to a HTTPRoute that uses a HTTPRequestMirrorFilter, the response will never come from the backendRef defined in the filter. Responses from the mirror backendRef are always ignored.
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Mirroring the Traffic
Next, create a new Deployment and Service to mirror requests to. The following example will use
a second instance of the application deployed in the quickstart.
Then create an HTTPRoute that uses a HTTPRequestMirrorFilter to send requests to the original
service from the quickstart, and mirror request to the service that was just deployed.
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-mirror -o yaml
Get the Gateway’s address:
exportGATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Querying backends.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate which pod handled the request. There is only one pod in the deployment for the example app
from the quickstart, so it will be the same on all subsequent requests.
Check the logs of the pods and you will see that the original deployment and the new deployment each got a request:
$ kubectl logs deploy/backend && kubectl logs deploy/backend-2
...
Starting server, listening on port 3000(http)Echoing back request made to /get to client (10.42.0.10:41566)Starting server, listening on port 3000(http)Echoing back request made to /get to client (10.42.0.10:45096)
Multiple BackendRefs
When an HTTPRoute has multiple backendRefs and an HTTPRequestMirrorFilter, traffic splitting will still behave the same as it normally would for the main backendRefs while the backendRef of the HTTPRequestMirrorFilter will continue receiving mirrored copies of the incoming requests.
Multiple HTTPRequestMirrorFilters are not supported on the same HTTPRouterule. When attempting to do so, the admission webhook will reject the configuration.
Error from server: error when creating "STDIN": admission webhook "validate.gateway.networking.k8s.io" denied the request: spec.rules[0].filters: Invalid value: "RequestMirror": cannot be used multiple times in the same rule
1.2.17 - HTTPRoute Traffic Splitting
The HTTPRoute resource allows one or more backendRefs to be provided. Requests will be routed to these upstreams
if they match the rules of the HTTPRoute. If an invalid backendRef is configured, then HTTP responses will be returned
with status code 500 for all requests that would have been sent to that backend.
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Single backendRef
When a single backendRef is configured in a HTTPRoute, it will receive 100% of the traffic.
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-headers -o yaml
Get the Gateway’s address:
exportGATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Querying backends.example/get should result in a 200 response from the example Gateway and the output from the
example app should indicate which pod handled the request. There is only one pod in the deployment for the example app
from the quickstart, so it will be the same on all subsequent requests.
Querying backends.example/get should result in 200 responses from the example Gateway and the output from the
example app that indicates which pod handled the request should switch between the first pod and the second one from the
new deployment on subsequent requests.
If multiple backendRefs are configured and an un-even traffic split between the backends is desired, then the weight
field can be used to control the weight of requests to each backend. If weight is not configured for a backendRef it is
assumed to be 1.
The weight field in a backendRef controls the distribution of the traffic split. The proportion of
requests to a single backendRef is calculated by dividing its weight by the sum of all backendRef weights in the
HTTPRoute. The weight is not a percentage and the sum of all weights does not need to add up to 100.
The HTTPRoute below will configure the gateway to send 80% of the traffic to the backend service, and 20% to the
backend-2 service.
backendRefs can be considered invalid for the following reasons:
The group field is configured to something other than "". Currently, only the core API group (specified by
omitting the group field or setting it to an empty string) is supported
The kind field is configured to anything other than Service. Envoy Gateway currently only supports Kubernetes
Service backendRefs
The backendRef configures a service with a namespace not permitted by any existing ReferenceGrants
The port field is not configured or is configured to a port that does not exist on the Service
The named Service configured by the backendRef cannot be found
Modifying the above example to make the backend-2 backendRef invalid by using a port that does not exist on the Service
will result in 80% of the traffic being sent to the backend service, and 20% of the traffic receiving an HTTP response
with status code 500.
Querying backends.example/get should result in 200 responses 80% of the time, and 500 responses 20% of the time.
$ curl -vvv --header "Host: backends.example""http://${GATEWAY_HOST}/get"> GET /get HTTP/1.1
> Host: backends.example
> User-Agent: curl/7.81.0
> Accept: */*
>* Mark bundle as not supporting multiuse
< HTTP/1.1 500 Internal Server Error
< server: envoy
< content-length: 0
<
1.2.18 - Local Rate Limit
Rate limit is a feature that allows the user to limit the number of incoming requests to a predefined value based on attributes within the traffic flow.
Here are some reasons why you may want to implement Rate limits
To prevent malicious activity such as DDoS attacks.
To prevent applications and its resources (such as a database) from getting overloaded.
Local rate limiting applies rate limits to the traffic flowing through a single instance of Envoy proxy. This means
that if the data plane has 2 replicas of Envoy running, and the rate limit is 10 requests/second, each replica will allow
10 requests/second. This is in contrast to Global Rate Limiting which applies rate limits to the traffic flowing through
all instances of Envoy proxy.
Envoy Gateway introduces a new CRD called BackendTrafficPolicy that allows the user to describe their rate limit intent.
This instantiated resource can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Note: Limit is applied per route. Even if a BackendTrafficPolicy targets a gateway, each route in that gateway
still has a separate rate limit bucket. For example, if a gateway has 2 routes, and the limit is 100r/s, then each route
has its own 100r/s rate limit bucket.
Prerequisites
Install Envoy Gateway
Follow the steps from the Quickstart to install Envoy Gateway and the HTTPRoute example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Rate Limit Specific User
Here is an example of a rate limit implemented by the application developer to limit a specific user by matching on a custom x-user-id header
with a value set to one.
The HTTPRoute status should indicate that it has been accepted and is bound to the example Gateway.
kubectl get httproute/http-ratelimit -o yaml
Get the Gateway’s address:
exportGATEWAY_HOST=$(kubectl get gateway/eg -o jsonpath='{.status.addresses[0].value}')
Let’s query ratelimit.example/get 4 times. We should receive a 200 response from the example Gateway for the first 3 requests
and then receive a 429 status code for the 4th request since the limit is set at 3 requests/Hour for the request which contains the header x-user-id
and value one.
for i in {1..4};do curl -I --header "Host: ratelimit.example" --header "x-user-id: one" http://${GATEWAY_HOST}/get ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:31 GMT
content-length: 460
x-envoy-upstream-service-time: 4
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:32 GMT
content-length: 460
x-envoy-upstream-service-time: 2
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:33 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Wed, 08 Feb 2023 02:33:34 GMT
server: envoy
transfer-encoding: chunked
You should be able to send requests with the x-user-id header and a different value and receive successful responses from the server.
for i in {1..4};do curl -I --header "Host: ratelimit.example" --header "x-user-id: two" http://${GATEWAY_HOST}/get ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:36 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:37 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:38 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:34:39 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
Rate Limit All Requests
This example shows you how to rate limit all requests matching the HTTPRoute rule at 3 requests/Hour by leaving the clientSelectors field unset.
for i in {1..4};do curl -I --header "Host: ratelimit.example" http://${GATEWAY_HOST}/get ; sleep 1;done
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:31 GMT
content-length: 460
x-envoy-upstream-service-time: 4
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:32 GMT
content-length: 460
x-envoy-upstream-service-time: 2
server: envoy
HTTP/1.1 200 OK
content-type: application/json
x-content-type-options: nosniff
date: Wed, 08 Feb 2023 02:33:33 GMT
content-length: 460
x-envoy-upstream-service-time: 0
server: envoy
HTTP/1.1 429 Too Many Requests
x-envoy-ratelimited: true
date: Wed, 08 Feb 2023 02:33:34 GMT
server: envoy
transfer-encoding: chunked
Note: Local rate limiting does not support distinct matching. If you want to rate limit based on distinct values,
you should use Global Rate Limiting.
1.2.19 - Multicluster Service Routing
The Multicluster Service API ServiceImport object can be used as part of the GatewayAPI backendRef for configuring routes. For more information about multicluster service API follow sig documentation.
We will use Submariner project for setting up the multicluster environment for exporting the service to be routed from peer clusters.
Setting KIND clusters and installing Submariner.
We will be using KIND clusters to demonstrate this example.
git clone https://github.com/submariner-io/submariner-operator
cd submariner-operator
make clusters
Note: remain in submariner-operator directory for the rest of the steps in this section
A retry setting specifies the maximum number of times an Envoy proxy attempts to connect to a service if the initial call fails. Retries can enhance service availability and application performance by making sure that calls don’t fail permanently because of transient problems such as a temporarily overloaded service or network. The interval between retries prevents the called service from being overwhelmed with requests.
Envoy Gateway supports the following retry settings:
NumRetries: is the number of retries to be attempted. Defaults to 2.
RetryOn: specifies the retry trigger condition.
PerRetryPolicy: is the retry policy to be applied per retry attempt.
Envoy Gateway introduces a new CRD called BackendTrafficPolicy that allows the user to describe their desired retry settings. This instantiated resource can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Note: There are distinct circuit breaker counters for each BackendReference in an xRoute rule. Even if a BackendTrafficPolicy targets a Gateway, each BackendReference in that gateway still has separate circuit breaker counter.
Prerequisites
Follow the installation step from the Quickstart to install Envoy Gateway and sample resources.
Test and customize retry settings
Before applying a BackendTrafficPolicy with retry setting to a route, let’s test the default retry settings.
Routing to endpoints outside the Kubernetes cluster where Envoy Gateway and its corresponding Envoy Proxy fleet is
running is a common use case. This can be achieved by defining FQDN addresses in a EndpointSlice.
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
TCPRoute provides a way to route TCP requests. When combined with a Gateway listener, it can be used to forward
connections on the port specified by the listener to a set of backends specified by the TCPRoute. To learn more about
HTTP routing, refer to the Gateway API documentation.
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Configuration
In this example, we have one Gateway resource and two TCPRoute resources that distribute the traffic with the following
rules:
All TCP streams on port 8088 of the Gateway are forwarded to port 3001 of foo Kubernetes Service.
All TCP streams on port 8089 of the Gateway are forwarded to port 3002 of bar Kubernetes Service.
In this example two TCP listeners will be applied to the Gateway in order to route them to two separate backend
TCPRoutes, note that the protocol set for the listeners on the Gateway is TCP:
Install the GatewayClass and a tcp-gateway Gateway first.
In this way each TCPRoute “attaches” itself to a different port on the Gateway so that the foo service
is taking traffic for port 8088 from outside the cluster and bar service takes the port 8089 traffic.
Before testing, please get the tcp-gateway Gateway’s address first:
exportGATEWAY_HOST=$(kubectl get gateway/tcp-gateway -o jsonpath='{.status.addresses[0].value}')
You can try to use nc to test the TCP connections of envoy gateway with different ports, and you can see them succeeded:
You can see that the traffic routing to bar service when sending request to 8089 port.
1.2.23 - UDP Routing
The UDPRoute resource allows users to configure UDP routing by matching UDP traffic and forwarding it to Kubernetes
backends. This task will use CoreDNS example to walk you through the steps required to configure UDPRoute on Envoy
Gateway.
Note: UDPRoute allows Envoy Gateway to operate as a non-transparent proxy between a UDP client and server. The lack
of transparency means that the upstream server will see the source IP and port of the Gateway instead of the client.
For additional information, refer to Envoy’s UDP proxy documentation.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Installation
Install CoreDNS in the Kubernetes cluster as the example backend. The installed CoreDNS is listening on
UDP port 53 for DNS lookups.
Checkout the Developer Guide to get involved in the project.
1.3 - Security
This section includes Security tasks.
1.3.1 - Accelerating TLS Handshakes using Private Key Provider in Envoy
TLS operations can be accelerated or the private key can be protected using specialized hardware. This can be leveraged in Envoy using Envoy Private Key Provider is added to Envoy.
Today, there are two private key providers implemented in Envoy as contrib extensions:
Both of them are used to accelerate the TLS handshake through the hardware capabilities.
This task will walk you through the steps required to configure TLS Termination mode for TCP traffic while also using the Envoy Private Key Provider to accelerate the TLS handshake by leveraging QAT and the HW accelerator available on Intel SPR/EMR Xeon server platforms.
Prerequisites
For QAT
Install Linux kernel 5.17 or similar
Ensure the node has QAT devices by checking the QAT physical function devices presented. Supported Devices
Verification of the plugin deployment and detection of QAT hardware can be confirmed by examining the resource allocations on the nodes:
kubectl get node -o yaml| grep qat.intel.com
For CryptoMB:
It required the node with 3rd generation Intel Xeon Scalable processor server processors, or later.
For kubernetes Cluster, if not all nodes that support Intel® AVX-512 in Kubernetes cluster, you need to add some labels to divide these two kinds of nodes manually or using NFD.
kubectl get nodes -l feature.node.kubernetes.io/cpu-cpuid.AVX512F,feature.node.kubernetes.io/cpu-cpuid.AVX512DQ,feature.node.kubernetes.io/cpu-cpuid.AVX512BW,feature.node.kubernetes.io/cpu-cpuid.AVX512VBMI2,feature.node.kubernetes.io/cpu-cpuid.AVX512IFMA
Check CPUIDS manually on the node if without using NFD:
Follow the steps from the Quickstart to install Envoy Gateway.
Enable the EnvoyPatchPolicy feature, which will allow us to directly configure the Private Key Provider Envoy Filter, since Envoy Gateway does not directly expose this functionality.
Using the envoyproxy image with contrib extensions and add qat resources requesting, ensure the k8s scheduler find out a machine with required resource.
Using the envoyproxy image with contrib extensions and add the node affinity to scheduling the Envoy Gateway pod on the machine with required CPU instructions.
Or using preferredDuringSchedulingIgnoredDuringExecution for best effort scheduling, or not doing any node affinity, just doing the random scheduling. The CryptoMB private key provider supports software fallback if the required CPU instructions aren’t here.
Apply EnvoyPatchPolicy to enable private key provider
You will see a performance boost after private key provider enabled. For example, you will get results as below.
Without private key provider:
All done43069 calls (plus 10 warmup) 6.966 ms avg, 1435.4 qps
With CryptoMB private key provider, the QPS is over 2 times than without private key provider.
All done93983 calls (plus 128 warmup) 40.880 ms avg, 3130.5 qps
With QAT private key provider, the QPS is over 3 times than without private key provider
All done134746 calls (plus 128 warmup) 28.505 ms avg, 4489.6 qps
1.3.2 - Backend TLS: Gateway to Backend
This task demonstrates how TLS can be achieved between the Gateway and a backend.
This task uses a self-signed CA, so it should be used for testing and demonstration purposes only.
Envoy Gateway supports the Gateway-API defined BackendTLSPolicy.
Prerequisites
OpenSSL to generate TLS assets.
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
TLS Certificates
Generate the certificates and keys used by the backend to terminate TLS connections from the Gateways.
Create a root certificate and private key to sign certificates:
Create a BackendTLSPolicy instructing Envoy Gateway to establish a TLS connection with the backend and validate the backend certificate is issued by a trusted CA and contains an appropriate DNS SAN.
This task provides instructions for configuring HTTP Basic authentication.
HTTP Basic authentication checks if an incoming request has a valid username and password before routing the request to
a backend service.
Envoy Gateway introduces a new CRD called SecurityPolicy that allows the user to configure HTTP Basic
authentication.
This instantiated resource can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Configuration
Envoy Gateway uses .htpasswd format to store the username-password pairs for authentication.
The file must be stored in a kubernetes secret and referenced in the SecurityPolicy configuration.
The secret is an Opaque secret, and the username-password pairs must be stored in the key “.htpasswd”.
Create a root certificate
Create a root certificate and private key to sign certificates:
First, create a .htpasswd file with the username and password you want to use for authentication.
Note: Please always use HTTPS with Basic Authentication. This prevents credentials from being transmitted in plain text.
The input password won’t be saved, instead, a hash will be generated and saved in the output file. When a request
tries to access protected resources, the password in the “Authorization” HTTP header will be hashed and compared with the
saved hash.
Note: only SHA hash algorithm is supported for now.
htpasswd -cbs .htpasswd foo bar
You can also add more users to the file:
htpasswd -bs .htpasswd foo1 bar1
Create a basic-auth secret
Next, create a kubernetes secret with the generated .htpasswd file in the previous step.
The request should be allowed and you should see the response from the backend service.
## Clean-UpFollow the steps from the [Quickstart](../../quickstart) to uninstall Envoy Gateway and the example manifest.
Delete the SecurityPolicy and the secret
```shell
kubectl delete securitypolicy/basic-auth-example
kubectl delete secret/basic-auth
kubectl delete secret/example-cert
Next Steps
Checkout the Developer Guide to get involved in the project.
1.3.4 - CORS
This task provides instructions for configuring Cross-Origin Resource Sharing (CORS) on Envoy Gateway.
CORS defines a way for client web applications that are loaded in one domain to interact with resources in a different
domain.
Envoy Gateway introduces a new CRD called SecurityPolicy that allows the user to configure CORS.
This instantiated resource can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Configuration
When configuring CORS either an origin with a precise hostname can be configured or an hostname containing a wildcard prefix,
allowing all subdomains of the specified hostname.
In addition to that the entire origin (with or without specifying a scheme) can be a wildcard to allow all origins.
The below example defines a SecurityPolicy that allows CORS for all HTTP requests originating from www.foo.com.
You won’t see any CORS headers in the response, indicating that the request from http://www.bar.com was not allowed.
If you try to send a request from http://www.foo.com:8080, you should also see similar response because the port number
8080 is not included in the allowed origins.
CORS specification requires that the browsers to send a preflight request to the server to ask if it’s allowed
to access the limited resource in another domains. The browsers are supposed to follow the response from the server to
determine whether to send the actual request or not. The CORS filter only response to the preflight requests according to
its configuration. It won’t deny any requests. The browsers are responsible for enforcing the CORS policy.
The targeted HTTPRoute or the HTTPRoutes that the targeted Gateway routes to must allow the OPTIONS method for the CORS
filter to work. Otherwise, the OPTIONS request won’t match the routes and the CORS filter won’t be invoked.
Clean-Up
Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.
Delete the SecurityPolicy:
kubectl delete securitypolicy/cors-example
Next Steps
Checkout the Developer Guide to get involved in the project.
1.3.5 - External Authorization
This task provides instructions for configuring external authentication.
External authorization calls an external HTTP or gRPC service to check whether an incoming HTTP request is authorized
or not. If the request is deemed unauthorized, then the request will be denied with a 403 (Forbidden) response. If the
request is authorized, then the request will be allowed to proceed to the backend service.
Envoy Gateway introduces a new CRD called SecurityPolicy that allows the user to configure external authorization.
This instantiated resource can be linked to a Gateway and HTTPRoute resource.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Verify the Gateway status:
kubectl get gateway/eg -o yaml
HTTP External Authorization Service
Installation
Install a demo HTTP service that will be used as the external authorization service:
Create a new SecurityPolicy resource to configure the external authorization. This SecurityPolicy targets the HTTPRoute
“myApp” created in the previous step. It calls the HTTP external authorization service “http-ext-auth” on port 9002 for
authorization. The headersToBackend field specifies the headers that will be sent to the backend service if the request
is successfully authorized.
The request should be allowed and you should see the response from the backend service.
Because the x-current-user header from the auth response has been sent to the backend service,
you should see the x-current-user header in the response.
"X-Current-User": [
"user1"
],
GRPC External Authorization Service
Installation
Install a demo gRPC service that will be used as the external authorization service. The demo gRPC service is enabled
with TLS and a BackendTLSConfig is created to configure the communication between the Envoy proxy and the gRPC service.
Note: TLS is optional for HTTP or gRPC external authorization services. However, enabling TLS is recommended for enhanced
security in production environments.
The HTTPRoute created in the previous section is still valid and can be used with the gRPC auth service, but if you have
not created the HTTPRoute, you can create it now.
Create a new HTTPRoute resource to route traffic on the path /myapp to the backend service.
Update the SecurityPolicy that was created in the previous section to use the gRPC external authorization service.
It calls the gRPC external authorization service “grpc-ext-auth” on port 9002 for authorization.
kubectl get securitypolicy/ext-auth-example -o yaml
Because the gRPC external authorization service is enabled with TLS, a BackendTLSConfig needs to be created to configure
the communication between the Envoy proxy and the gRPC auth service.
Checkout the Developer Guide to get involved in the project.
1.3.6 - JWT Authentication
This task provides instructions for configuring JSON Web Token (JWT) authentication. JWT authentication checks
if an incoming request has a valid JWT before routing the request to a backend service. Currently, Envoy Gateway only
supports validating a JWT from an HTTP header, e.g. Authorization: Bearer <token>.
Envoy Gateway introduces a new CRD called SecurityPolicy that allows the user to configure JWT authentication.
This instantiated resource can be linked to a Gateway, HTTPRoute or GRPCRoute resource.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
For GRPC - follow the steps from the GRPC Routing example.
Before proceeding, you should be able to query the example backend using HTTP or GRPC.
Configuration
Allow requests with a valid JWT by creating an SecurityPolicy and attaching it to the example
HTTPRoute or GRPCRoute.
Two HTTPRoute has been created, one for /foo and another for /bar. A SecurityPolicy has been created and targeted
HTTPRoute foo to authenticate requests for /foo. The HTTPRoute bar is not targeted by the SecurityPolicy and will allow unauthenticated requests to /bar.
Verify the HTTPRoute configuration and status:
kubectl get httproute/foo -o yaml
kubectl get httproute/bar -o yaml
The SecurityPolicy is configured for JWT authentication and uses a single JSON Web Key Set (JWKS)
provider for authenticating the JWT.
Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.
Delete the SecurityPolicy:
kubectl delete securitypolicy/jwt-example
Next Steps
Checkout the Developer Guide to get involved in the project.
1.3.7 - Mutual TLS: External Clients to the Gateway
This task demonstrates how mutual TLS can be achieved between external clients and the Gateway.
This task uses a self-signed CA, so it should be used for testing and demonstration purposes only.
Prerequisites
OpenSSL to generate TLS assets.
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
TLS Certificates
Generate the certificates and keys used by the Gateway to terminate client TLS connections.
Create a root certificate and private key to sign certificates:
This task provides instructions for configuring OpenID Connect (OIDC) authentication.
OpenID Connect (OIDC) is an authentication standard built on top of OAuth 2.0.
It enables client applications to rely on authentication that is performed by an OpenID Connect Provider (OP)
to verify the identity of a user.
Envoy Gateway introduces a new CRD called SecurityPolicy that allows the user to configure OIDC
authentication.
This instantiated resource can be linked to a Gateway and HTTPRoute resource.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Verify the Gateway status:
kubectl get gateway/eg -o yaml
OIDC can be configured at the Gateway level to authenticate all the HTTPRoutes that are associated with the Gateway with
the same OIDC configuration, or at the HTTPRoute level to authenticate each HTTPRoute with different OIDC configurations.
This task demonstrates the configuration of OIDC at the HTTPRoute level.
Let’s create an HTTPRoute that represents an application protected by OIDC.
This task uses Google as the OIDC provider to demonstrate the configuration of OIDC. However, EG works with any OIDC
providers, including Auth0, Azure AD, Keycloak, Okta, OneLogin, Salesforce, UAA, etc.
Register an OIDC application
Follow the steps in the Google OIDC documentation to register an OIDC application. Please make sure the
redirect URL is set to the one you configured in the SecurityPolicy that you will create in the step below. In this example,
the redirect URL is http://www.example.com:8080/myapp/oauth2/callback.
After registering the application, you should have the following information:
Client ID: The client ID of the OIDC application.
Client Secret: The client secret of the OIDC application.
Create a kubernetes secret
Next, create a kubernetes secret with the Client Secret created in the previous step. The secret is an Opaque secret,
and the Client Secret must be stored in the key “client-secret”.
Note: please replace the ${CLIENT_SECRET} with the actual Client Secret that you got from the previous step.
$ kubectl create secret generic my-app-client-secret --from-literal=client-secret=${CLIENT_SECRET}secret "my-app-client-secret" created
Create a SecurityPolicy
Please notice that the redirectURL and logoutPath must match the target HTTPRoute. In this example, the target
HTTPRoute is configured to match the host www.example.com and the path /myapp, so the redirectURL must be prefixed
with http://www.example.com:8080/myapp, and logoutPath must be prefixed with/myapp, otherwise the OIDC authentication
will fail because the redirect and logout requests will not match the target HTTPRoute and therefore can’t be processed
by the OAuth2 filter on that HTTPRoute.
Note: please replace the ${CLIENT_ID} in the below yaml snippet with the actual Client ID that you got from the OIDC provider.
Put www.example.com in the /etc/hosts file in your test machine, so we can use this host name to access the gateway from a browser:
...
127.0.0.1 www.example.com
Open a browser and navigate to the http://www.example.com:8080/myapp address. You should be redirected to the Google
login page. After you successfully login, you should see the response from the backend service.
Clean-Up
Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.
Delete the SecurityPolicy, the secret and the HTTPRoute:
Checkout the Developer Guide to get involved in the project.
1.3.9 - Secure Gateways
This task will help you get started using secure Gateways.
This task uses a self-signed CA, so it should be used for testing and demonstration purposes only.
Prerequisites
OpenSSL to generate TLS assets.
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
TLS Certificates
Generate the certificates and keys used by the Gateway to terminate client TLS connections.
Create a root certificate and private key to sign certificates:
Follow the steps in the Testing section to test connectivity to the backend app through both Gateway
listeners. Replace www.example.com with foo.example.com to test the new HTTPS listener.
Cross Namespace Certificate References
A Gateway can be configured to reference a certificate in a different namespace. This is allowed by a ReferenceGrant
created in the target namespace. Without the ReferenceGrant, a cross-namespace reference is invalid.
Before proceeding, ensure you can query the HTTPS backend service from the Testing section.
To demonstrate cross namespace certificate references, create a ReferenceGrant that allows Gateways from the “default”
namespace to reference Secrets in the “envoy-gateway-system” namespace:
The Gateway HTTPS listener should now surface the Ready: False status condition and the example HTTPS backend should
no longer be reachable through the Gateway.
kubectl get gateway/eg -o yaml
Recreate the example Secret in the envoy-gateway-system namespace:
The Gateway HTTPS listener status should now surface the Ready: True condition and you should once again be able to
query the HTTPS backend through the Gateway.
This section gives a walkthrough to generate RSA and ECDSA derived certificates and keys for the Server, which can then be configured in the Gateway listener, to terminate TLS traffic.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Follow the steps in the TLS Certificates section to generate self-signed RSA derived Server certificate and private key, and configure those in the Gateway listener configuration to terminate HTTPS traffic.
Since the Secret configured at this point is an RSA based Secret, if we enforce the usage of an ECDSA cipher, the call should fail as follows
$ curl -v -HHost:www.example.com --resolve "www.example.com:8443:127.0.0.1"\
--cacert example.com.crt https://www.example.com:8443/get -Isv --ciphers ECDHE-ECDSA-CHACHA20-POLY1305 --tlsv1.2 --tls-max 1.2
* Added www.example.com:8443:127.0.0.1 to DNS cache
* Hostname www.example.com was found in DNS cache
* Trying 127.0.0.1:8443...
* Connected to www.example.com (127.0.0.1) port 8443(#0)* ALPN: offers h2
* ALPN: offers http/1.1
* Cipher selection: ECDHE-ECDSA-CHACHA20-POLY1305
* CAfile: example.com.crt
* CApath: none
* (304)(OUT), TLS handshake, Client hello (1):
* error:1404B410:SSL routines:ST_CONNECT:sslv3 alert handshake failure
* Closing connection 0
Moving forward in the doc, we will be configuring the existing Gateway listener to accept both kinds of ciphers.
TLS Certificates
Reuse the CA certificate and key pair generated in the Secure Gateways task and use this CA to sign both RSA and ECDSA Server certificates.
Note the CA certificate and key names are example.com.crt and example.com.key respectively.
Create an ECDSA certificate and a private key for www.example.com:
Again, while testing in Cluster without External LoadBalancer Support, we can query the example app through Envoy proxy while enforcing an RSA cipher, which should work as it did before:
This sections gives a walkthrough to generate multiple certificates corresponding to different FQDNs. The same Gateway listener can then be configured to terminate TLS traffic for multiple FQDNs based on the SNI matching.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Follow the steps in the TLS Certificates section to generate self-signed RSA derived Server certificate and private key, and configure those in the Gateway listener configuration to terminate HTTPS traffic.
Additional Configurations
Using the TLS Certificates section, we first generate additional Secret for another Host www.sample.com.
Since the multiple certificates are configured on the same Gateway listener, Envoy was able to provide the client with appropriate certificate based on the SNI in the client request.
Checkout the Developer Guide to get involved in the project.
1.3.10 - Threat Model
Envoy Gateway Threat Model and End User Recommendations
About
This work was performed by ControlPlane and commissioned by the Linux Foundation. ControlPlane is a global cloud native and open source cybersecurity consultancy, trusted as the partner of choice in securing: multinational banks; major public clouds; international financial institutions; critical national infrastructure programs; multinational oil and gas companies, healthcare and insurance providers; and global media firms.
Threat Modelling Team
James Callaghan, Torin van den Bulk, Eduardo Olarte
Reviewers
Arko Dasgupta, Matt Turner, Zack Butcher, Marco De Benedictis
Introduction
As we embrace the proliferation of microservice-based architectures in the cloud-native landscape, simplicity in setup and configuration becomes paramount as DevOps teams face the challenge of choosing between numerous similar technologies. One such choice which every team deploying to Kubernetes faces is what to use as an ingress controller. With a plethora of options available, and the existence of vendor-specific annotations leading to small inconsistencies between implementations, the Gateway API project was introduced by the SIG-NETWORK community, with the goal of eventually replacing the Ingress resource.
Envoy Gateway is configured by Gateway API resources, and serves as an intuitive and feature-rich wrapper over the widely acclaimed Envoy Proxy. With a convenient setup based on Kubernetes (K8s) manifests, Envoy Gateway streamlines the management of Envoy Proxy instances in an edge-proxy setting, reducing the operational overhead of managing low-level Envoy configurations. Envoy Gateway benefits cloud-native DevOps teams through its role-oriented configuration, providing granular control based on Role-Based Access Control (RBAC) principles. These features form the basis of our exploration into Envoy Gateway and the rich feature set it brings to the table.
In this threat model, we aim to provide an analysis of Envoy Gateway’s design components and their capabilities (at version 1.0) through a threat-driven approach. It should be noted that this does not constitute a security audit of the Envoy Gateway project, but instead focuses on different possible deployment topologies for Envoy Gateway with the goal of deriving recommendations and best practice guidance for end users.
The Envoy Gateway project recommends a multi-tenancy model whereby each tenant deploys their own Envoy Gateway controller in a namespace which they own. We will also explore the implications and risks associated with multiple tenants using a shared controller.
Scope
The primary focus of this threat model is to identify and assess security risks associated with deploying and operating Envoy Gateway within a multi-tenant Kubernetes (K8s) cluster. This model aims to provide a comprehensive understanding of the system, its transmission points, and potential vulnerabilities to enumerated threats.
In Scope
Envoy Gateway: As the primary focus of this threat model, all aspects of Envoy Gateway, including its configuration, deployment, and operation will be analysed. This includes how the gateway manages TLS certificates, authentication, service-to-service traffic routing, and more.
Kubernetes Cluster: Configuration and operation of the underlying Kubernetes cluster, including how it manages network policies, access control, and resource isolation for different namespaces/tenants in relation to Envoy will be considered.
Tenant Workloads: Tenant workloads (and the pods they run on) will be considered, focusing on how they interact with the Envoy Gateway and potential vulnerabilities that could be exploited.
Out of Scope
This threat model will not consider security risks associated with the underlying infrastructure (e.g., EC2 compute instances and S3 buckets) or non-Envoy related components within the Kubernetes Cluster. It will focus solely on the Envoy Gateway and its interaction with the Kubernetes cluster and tenant workloads.
Implementation of Envoy Gateway as an egress traffic controller is out of scope for this threat model and will not be considered in the report’s findings.
To provide an in-depth look into both the system design and end-user deployment of Envoy Gateway, we will be focusing on the Deployment Architecture Diagram below.
The Deployment Architecture Diagram provides a high-level model of an end-user deployment of Envoy Gateway. For simplicity, we will look at different deployment topologies on a single multi-tenant Kubernetes cluster. Envoy Gateway operates as an edge proxy within this environment, handling the traffic flow between external interfaces and services within the cluster. The example will use two Envoy Gateway controllers - one dedicated controller for a single tenant, and one shared controller for two other tenants. Each Envoy Gateway controller will accept a single GatewayClass resource.
Deployment Architecture Diagram
As Envoy Gateway implements the Kubernetes GatewayAPI, this threat model will focus on the key objects in the Gateway API resource model:
GatewayClass: defines a set of gateways with a commonconfiguration and behaviour. It is a cluster scoped resource.
Gateway: requests a point where traffic can be translated to Services within the cluster.
Routes: describe how traffic coming via the Gateway maps to theServices.
At the time of writing, Envoy Gateway only supports a Kubernetes provider. As such, we will consider a reference architecture where multiple teams are working on the same Kubernetes cluster within different namespaces (Tenant A, B, & C). We will assume that some teams have similar security and performance needs, and a decision has been made to use a shared Gateway. However, we will also consider the case that some teams require dedicated Gateways, perhaps for compliance reasons or requirements driven by an internal threat model.
Infrastructure provider: The infrastructure provider (infra) is responsible for the overall environment that the cluster(s) are operating in. Examples include: the cloud provider (AWS, Azure, GCP, …) or the PaaS provider in a company.
Cluster operator: The cluster operator (ops) is responsible for administration of entire clusters. They manage policies, network access, application permissions.
Application developer: The application developer (dev) is responsible for defining their application configuration (e.g. timeouts, request matching/filter) and Service composition (e.g. path routing to backends).
Application admin: The application admin has administrative access to some namespaces within a cluster, but not the cluster as a whole.
Our threat model will be based on the high-level setup shown below, where Envoy is used in an edge-proxy scenario:
The following use cases will be considered, in line with the Envoy Gateway tasks:
Routing and controlling traffic, including:
a. HTTP b. TCP c. UDP d. gRPC e.TLS passthrough
TLS termination
Request Authentication
Rate Limiting
Key Assumptions
This section outlines the foundational premises that shape our analysis and recommendations for the deployment and management of Envoy Gateway within an organisation. The key assumptions are as follows:
1. Kubernetes Provider: For the purposes of this analysis, we assume that a K8s provider will be used to host the cluster.
2. Multi-tenant cluster: In order to produce a broad set of recommendations, it is assumed that within the single cluster, there is:
A dedicated cluster operation (ops) team responsible for maintaining the core cluster infrastructure.
Multiple application teams who wish to define their own Gateway resources, which will route traffic to their respective applications.
3. Soft multi-tenancy model: It is assumed that co-tenants will have some level of trust between themselves, and will not act in an overtly hostile manner to each other.
4. Ingress Control: It’s assumed that Envoy Gateway is the only ingress controller in the K8s cluster as multiple controllers can lead to complex routing challenges and introduce out-of-scope security vulnerabilities.
5. Container Security: This threat model focuses on evaluating the security of the Envoy Gateway and Envoy Proxy images. All other container images running in tenant clusters, not associated with the edge proxy deployment, are assumed to be secure and obtained from trusted registries such as Docker Hub or Google Container Registry (GCR).
6. Cloud Provider Security: It is assumed that the K8s cluster is running on secure cloud infrastructure provided by a trusted Cloud Service Provider (CSP) such as AWS, GCP, or Azure Cloud.
Data
Data Dictionary
Ultimately, the data of interest in a threat model is the business data processed by the system in question. However, in the case of this threat model, we are looking at a generic deployment architecture involving Envoy Gateway in order to draw out a set of generalised threats which can be considered by teams looking to adopt an implementation of Gateway API. As such, we do not know the business impacts of a compromise of confidentiality, integrity or availability that would typically be captured in a data impact assessment. Instead, will we base our threat assessment on high-level groupings of data structures used in the configuration and operation of the general use cases considered (e.g. HTTP routing, TLS termination, request authentication etc.). We will then assign a confidentiality, integrity and availability impact based on a worst-case scenario of how each compromise could potentially affect business data processed by the generic deployment.
Data Name / Type
Notes
Confidentiality
Integrity
Availability
Static Configuration Data
Static configuration data is used to configure Envoy Gateway at startup. This data structure allows for a Provider to be set, which Envoy Gateway calls to establish its runtime configuration, resolve services and persist data. Unauthorised modification of static configuration data could enable the Envoy Gateway admin interface to be configured, logging parameters to be modified, global rate limiting configuration to be misconfigured, or malicious extensions registered for the Envoy Gateway Control Plane. A compromise of confidentiality could potentially give an attacker some useful reconnaissance information. A compromise of the availability of this information at startup time would result in Envoy Gateway starting with default parameters.
Medium
High
Low
Dynamic Configuration Data
Dynamic configuration data represents the desired state of the Data Plane, and is defined through Envoy Gateway and Gateway API Kubernetes resources. Unauthorised modification of this data could lead to vulnerabilities in an organisation’s Data Plane infrastructure via misconfiguration of an EnvoyProxy custom resource. Misconfiguration of Gateway API objects such as HTTPRoutes or TLSRoutes could result in traffic being directed to incorrect backends. A compromise of confidentiality could potentially give an attacker some useful reconnaissance information. A compromise of the availability of this information could result in tenant application traffic not being routable until the configuration is recovered and reapplied.
Medium
High
Medium
TLS Private Keys
TLS Private Keys, typically in PEM format, are used to initiate secure connections and encrypt communications. In the context of this threat model, private keys will be associated with the server side of an inbound TLS connection being terminated at a secure gateway configured through Envoy Gateway. Unauthorised exposure could lead to security threats such as person-in-the-middle attacks, whereby the confidentiality or integrity of business data could be compromised. A compromise of integrity may lead to similar consequences if an attacker could insert their own key material. An availability compromise could lead to tenant services being unavailable until new key material is generated and an appropriate CSR submitted.
High
High
Medium
TLS Certificates
X.509 certificates represent the binding of a public key (associated with the private key described above) to an identity in a TLS handshake. If an attacker could compromise the integrity of a certificate, they may be able to bind the identity of a TLS termination point to a key pair under their control, enabling person-in-the middle attacks. An availability compromise could lead to tenant services being unavailable until new key material is generated and an appropriate CSR submitted.
Low
High
Medium
JWKs
JWK (JSON Web Key) containing a public key used to validate JWTs for the client authentication use case considered in this threat model. If an attacker could compromise the integrity of a JWK or JSON web key set (JWKS), they could potentially authenticate to a service maliciously. Unavailability of an endpoint exposing JWKs could lead to client requests which require authentication being denied.
Low
High
Medium
JWTs
JWTs, formatted as compact, URL-safe JSON data structures, are utilised for the client authentication use case considered in this threat model. Maintaining their confidentiality and integrity is vital to prevent unauthorised access and ensure correct user identification.
High
High
Low
OIDC credentials
In OIDC authentication scenarios, the application credentials are represented by a client ID and a client secret. A compromise of its confidentiality or integrity could allow malicious actors to impersonate the application, potentially being able to access resources on behalf of the application and request ID tokens on behalf of users. Unavailability of this data would produce a rejection of the requests coming from legitimate users.
High
High
Medium
Basic authentiation password hashes
In basic authentication scenarios, passwords are stored as Kubernetes secrets in htpasswd format, where each entry is formed by the username and the hashed password. A compromise of these credentials’ confidentiality and integrity could lead to unauthorised access to the application. Unavailability of these credentials will cause login failures from the application users.
High
High
Medium
CIA Impact Assessment
Priority
Description
Confidentiality
High
Compromise of sensitive client data
Medium
Information leaked which could be useful for attacker reconnaissance
Low
Non-sensitive information leakage
Integrity
High
Compromise of source code repositories and gateway deployments
Medium
Traffic routing fails due to misconfiguration / invalid configuration
Low
Non-critical operation is blocked due to misconfiguration / invalid configuration
Availability
High
Large scale DoS
Medium
Tenant application is blocked for a significant period
Low
Tenant application is blocked for a short period
Data Flow Diagrams
The Data Flow Diagrams (DFDs) below describe the flow of data between the various processes, entities and data stores in a system, as well as the trust boundaries between different user roles and network interfaces. The DFDs are drawn at two different levels, starting at L0 (high-level system view) and increasing in granularity (to L1).
DFD L0
DFD L1
Key Threats and Recommendations
The scope of this threat model led to us categorising threats into priorities of High, Medium or Low; notably in a production implementation some of the threats’ prioritisation may be upgraded or downgraded depending on the business context and data classification.
Risk vs. Threat
For every finding, the risk and threat are stated. Risk defines the potential for negative outcome while threat defines the event that causes the negative outcome.
Threat Categorization
Throughout this threat model, we categorised threats into different areas based on their origin and the segment of the system that they impact. Here’s an overview of each category:
Container Security (CS): These threats are general to containerised applications. Therefore, they are not associated with Envoy Gateway or the Gateway API and could occur in most containerised workloads. They can originate from misconfigurations or vulnerabilities in the orchestrator or the container.
Gateway API (GW): These are threats related to the Gateway API that could affect any of its implementations. Malicious actors could benefit from misconfigurations or excessive permissions on the Gateway API resources (e.g. xRoutes or Gateways) to compromise the confidentiality, integrity, or availability of the application.
Envoy Gateway (EG): These threats are associated with specific configurations or features from Envoy Gateway or Envoy Proxy. If not set properly, these features could be leveraged to gain unauthorised access to protected resources.
Threat Actors
In order to provide a realistic set of threats that is applicable to most organisations, we de-scoped the most advanced and hard to mitigate threat actors as described below:
In Scope Threat Actors
When considering internal threat actors, we chose to follow the security model of the Kubernetes Gateway API.
Internal Attacker
Cluster Operator: The cluster operator (ops) is responsible for administration of entire clusters. They manage policies, network access, application permissions.
Application Developer: The application developer (dev) is responsible for defining their application configuration (e.g. timeouts, request matching/filter) and Service composition (e.g. path routing to backends).
Application Administrator: The application admin has administrative access to some namespaces within a cluster, but not the cluster as a whole.
External Attacker
Vandal: Script kiddie, trespasser
Motivated Individual: Political activist, thief, terrorist
Organised Crime: Syndicates, state-affiliated groups
Out of Scope Threat Actors
External Actors
Infrastructure Provider: The infrastructure provider (infra) is responsible for the overall environment that the cluster(s) are operating in. Examples include: the cloud provider, or the PaaS provider in a company.
Cloud Service Insider: Employee, external contractor, temporary worker
Foreign Intelligence Services (FIS): Nation states
High Priority Findings
EGTM-001 Usage of self-signed certificates
ID
UID
Category
Priority
EGTM-001
EGTM-GW-001
Gateway API
High
Risk: Self-signed certificates (which do not comply with PKI best practices) could lead to unauthorised access to the private key associated with the certificate used for inbound TLS termination at Envoy Proxy, compromising the confidentiality and integrity of proxied traffic.
Threat: Compromise of the private key associated with the certificate used for inbound TLS terminating at Envoy Proxy.
Recommendation: The Envoy Gateway quickstart demonstrates how to set up a Secure Gateway using an example where a self-signed root certificate is created using openssl. As stated in the Envoy Gateway documentation, this is not a suitable configuration for Production usage. It is recommended that PKI best practices are followed, whereby certificates are signed by an Intermediary CA which sits underneath an organisational 'offline' Root CA.
PKI best practices should also apply to the management of client certificates when using mTLS. The Envoy Gateway mTLS task shows how to set up client certificates using self-signed certificates. In the same way as gateway certificates and, as mentioned in the documentation, this configuration should not be used in production environments.
EGTM-002 Private keys are stored as Kubernetes secrets
ID
UID
Category
Priority
EGTM-002
EGTM-CS-001
Container Security
High
Risk: There is a risk that a threat actor could compromise the Kubernetes secret containing the Envoy private key, allowing the attacker to decrypt Envoy Proxy traffic, compromising the confidentiality of proxied traffic.
Threat: Kubernetes secret containing the Envoy private key is compromised and used to decrypt proxied traffic.
Recommendation: Certificate management best practices mandate short-lived key material where practical, meaning that a mechanism for rotation of private keys and certificates is required, along with a way for certificates to be mounted into Envoy containers. If Kubernetes secrets are used, when a certificate expires, the associated secret must be updated, and Envoy containers must be redeployed. Instead of a manual configuration, it is recommended that cert-manager is used.
EGTM-004 Usage of ClusterRoles with wide permissions
ID
UID
Category
Priority
EGTM-004
EGTM-K8-002
Container Security
High
Risk: There is a risk that a threat actor could abuse misconfigured RBAC to access the Envoy Gateway ClusterRole (envoy-gateway-role) and use it to expose all secrets across the cluster, thus compromising the confidentiality and integrity of tenant data.
Threat: Compromised Envoy Gateway or misconfigured ClusterRoleBinding (envoy-gateway-rolebinding) to Envoy Gateway ClusterRole (envoy-gateway-role), provides access to resources and secrets in different namespaces.
Recommendation: Users should be aware that Envoy Gateway uses a ClusterRole (envoy-gateway-role) when deployed via the Helm chart, to allow management of Envoy Proxies across different namespaces. This ClusterRole is powerful and includes the ability to read secrets in namespaces which may not be within the purview of Envoy Gateway.
Kubernetes best-practices involve restriction of ClusterRoleBindings, with the use of RoleBindings where possible to limit access per namespace by specifying the namespace in metadata. Namespace isolation reduces the impact of compromise from cluster-scoped roles. Ideally, fine-grained K8s roles should be created per the principle of least privilege to ensure they have the minimum access necessary for role functions.
The pull request #1656 introduced the use of Roles and RoleBindings in namespaced mode. This feature can be leveraged to reduce the amount of permissions required by the Envoy Gateway.
EGTM-007 Misconfiguration of Envoy Gateway dynamic config
ID
UID
Category
Priority
EGTM-007
EGTM-EG-002
Envoy Gateway
High
Risk: There is a risk that a threat actor could exploit misconfigured Kubernetes RBAC to create or modify Gateway API resources with no business need, potentially leading to the compromise of the confidentiality, integrity, and availability of resources and traffic within the cluster.
Threat: Unauthorised creation or misconfiguration of Gateway API resources by a threat actor with cluster-scoped access.
Recommendation: Configure the apiGroup and resource fields in RBAC policies to restrict access to Gateway and GatewayClass resources. Enable namespace isolation by using the namespace field, preventing unauthorised access to gateways in other namespaces.
EGTM-009 Co-tenant misconfigures resource across namespaces
ID
UID
Category
Priority
EGTM-009
EGTM-GW-002
Gateway API
High
Risk: There is a risk that a co-tenant misconfigures Gateway or Route resources, compromising the confidentiality, integrity, and availability of routed traffic through Envoy Gateway.
Threat: Malicious or accidental co-tenant misconfiguration of Gateways and Routes associated with other application teams.
Recommendation: Dedicated Envoy Gateways should be provided to each tenant within their respective namespace. A one-to-one relationship should be established between GatewayClass and Gateway resources, meaning that each tenant namespace should have their own GatewayClass watched by a unique Envoy Gateway Controller as defined here in the Deployment Mode documentation.
Application Admins should have write permissions on the Gateway resource, but only in their specific namespaces, and Application Developers should only hold write permissions on Route resources. To enact this access control schema, follow the Write Permissions for Advanced 4 Tier Model described in the Kubernetes Gateway API security model. Examples of secured gateway-route topologies can be found here within the Kubernetes Gateway API docs.
Optionally, consider a GitOps model, where only the GitOps operator has the permission to deploy or modify custom resources in production.
EGTM-014 Malicious image admission
ID
UID
Category
Priority
EGTM-014
EGTM-CS-006
Container Security
High
Risk: There is a risk that a supply chain attack on Envoy Gateway results in an arbitrary compromise of the confidentiality, integrity or availability of tenant data.
Threat: Supply chain threat actor introduces malicious code into Envoy Gateway or Proxy.
Recommendation: The Envoy Gateway project should continue to work towards conformance with supply-chain security best practices throughout the project lifecycle (for example, as set out in the CNCF Software Supply Chain Best Practices Whitepaper). Adherence to Supply-chain Levels for Software Artefacts (SLSA) standards is crucial for maintaining the security of the system. Employ version control systems to monitor the source and build platforms and assign responsibility to a specific stakeholder.
Integrate a supply chain security tool such as Sigstore, which provides native capabilities for signing and verifying container images and software artefacts. Software Bill of Materials (SBOM), Vulnerability Exploitability eXchange (VEX), and signed artefacts should also be incorporated into the security protocol.
EGTM-020 Out of date or misconfigured Envoy Proxy image
ID
UID
Category
Priority
EGTM-020
EGTM-CS-009
Container Security
High
Risk: There is a risk that a threat actor exploits an Envoy Proxy vulnerability to remote code execution (RCE) due to out of date or misconfigured Envoy Proxy pod deployment, compromising the confidentiality and integrity of Envoy Proxy along with the availability of the proxy service.
Threat: Deployment of an Envoy Proxy or Gateway image containing exploitable CVEs.
Recommendation: Always use the latest version of the Envoy Proxy image. Regularly check for updates and patch the system as soon as updates become available. Implement a CI/CD pipeline that includes security checks for images and prevents deployment of insecure configurations. A suitable tool should be chosen to provide container vulnerability scanning to mitigate the risk of known vulnerabilities.
EGTM-022 Credentials are stored as Kubernetes Secrets
ID
UID
Category
Priority
EGTM-022
EGTM-CS-010
Container Security
High
Risk: There is a risk that the OIDC client secret (for OIDC authentication) and user password hashes (for basic authentication) get leaked due to misconfigured RBAC permissions.
Threat: Unauthorised access to the application due to credential leakage.
Recommendation: Ensure that only authorised users and service accounts are able to access secrets. This is especially important in namespaces where SecurityPolicy objects are configured, since those namespaces are the ones to store secrets containing the client secret (in OIDC scenarios) and user password hashes (in basic authentication scenarios).
To do so, minimise the use of ClusterRoles and Roles allowing listing and getting secrets. Perform periodic audits of RBAC permissions.
EGTM-023 Weak Authentication
ID
UID
Category
Priority
EGTM-023
EGTM-EG-007
Envoy Gateway
High
Risk: There is a risk of unauthorised access due to the use of basic authentication, which does not enforce any password restriction in terms of complexity and length. In addition, password hashes are stored in SHA1 format, which is a deprecated hashing function.
Threat: Unauthorised access to the application due to weak authentication mechanisms.
Recommendation: It is recommended to make use of stronger authentication mechanisms (i.e. JWT authentication and OIDC authentication) instead of basic authentication. These authentication mechanisms have many advantages, such as the use of short-lived credentials and a central management of security policies through the identity provider.
Medium Priority Findings
EGTM-008 Misconfiguration of Envoy Gateway static config
ID
UID
Category
Priority
EGTM-008
EGTM-EG-003
Envoy Gateway
Medium
Risk: There is a risk of a threat actor misconfiguring static config and compromising the integrity of Envoy Gateway, ultimately leading to the compromised confidentiality, integrity, or availability of tenant data and cluster resources.
Threat: Accidental or deliberate misconfiguration of static configuration leads to a misconfigured deployment of Envoy Gateway, for example logging parameters could be modified or global rate limiting configuration misconfigured.
Recommendation: Implement a GitOps model, utilising Kubernetes' Role-Based Access Control (RBAC) and adhering to the principle of least privilege to minimise human intervention on the cluster. For instance, tools like Flux and ArgoCD can be used for declarative GitOps deployments, ensuring all changes are tracked and reviewed. Additionally, configure your source control management (SCM) system to include mandatory pull request (PR) reviews, commit signing, and protected branches to ensure only authorised changes can be committed to the start-up configuration.
EGTM-010 Weak pod security contexts and policies
ID
UID
Category
Priority
EGTM-010
EGTM-CS-005
Container Security
Medium
Risk: There is a risk that a threat actor exploits a weak pod security context, compromising the CIA of a node and the resources / services which run on it.
Threat: Threat Actor who has compromised a pod exploits weak security context to escape to a node, potentially leading to the compromise of Envoy Proxy or Gateway running on the same node.
Recommendation: To mitigate this risk, apply Pod Security Standards at a minimum of Baseline level to all namespaces, especially those containing Envoy Gateway and Proxy Pods. Pod security standards are implemented through K8s Pod Security Admission to provide admission control modes (enforce, audit, and warn) for namespaces. Pod security standards can be enforced by namespace labels as shown here, to enforce a baseline level of pod security to specific namespaces.
Further enhance the security by implementing a sandboxing solution such as gVisor for Envoy Gateway and Proxy Pods to isolate the application from the host kernel. This can be set within the runtimeClassName of the Pod specification.
EGTM-012 ClusterRoles and Roles with permission to deploy ReferenceGrants
ID
UID
Category
Priority
EGTM-012
EGTM-GW-004
Gateway API
Medium
Risk: There is a risk that a threat actor could abuse excessive RBAC privileges to create ReferenceGrant resources. These resources could then be used to create cross-namespace communication, leading to unauthorised access to the application. This could compromise the confidentiality and integrity of resources and configuration in the affected namespaces and potentially disrupt the availability of services that rely on these object references.
Threat: A ReferenceGrant is created, which validates traffic to cross namespace trust boundaries without a valid business reason, such as a route in one tenant's namespace referencing a backend in another.
Recommendation: Ensure that the ability to create ReferenceGrant resources is restricted to the minimum number of people. Pay special attention to ClusterRoles that allow that action.
EGTM-018 Network Denial of Service (DoS)
ID
UID
Category
Priority
EGTM-018
EGTM-GW-006
Gateway API
Medium
Risk: There is a risk that malicious requests could lead to a Denial of Service (DoS) attack, thereby reducing API gateway availability due to misconfigurations in rate-limiting or load balancing controls, or a lack of route timeout enforcement.
Threat: Reduced API gateway availability due to an attacker's maliciously crafted request (e.g., QoD) potentially inducing a Denial of Service (DoS) attack.
Recommendation: To ensure high availability and mitigate potential security threats, follow the guidelines in the Envoy Gateway documentation for configuring local rate limit filters, global rate limit filters, and load balancing.
Path normalisation should be enabled to minimise path confusion vulnerabilities. These measures help protect against volumetric threats such as Denial of Service (DoS) attacks. Utilise custom resources to implement policy attachment, thereby exposing request limit configuration for route types.
EGTM-019 JWT-based authentication replay attacks
ID
UID
Category
Priority
EGTM-019
EGTM-DP-004
Container Security
Medium
Risk: There is a risk that replay attacks using stolen or reused JSON Web Tokens (JWTs) can compromise transmission integrity, thereby undermining the confidentiality and integrity of the data plane.
Threat: Transmission integrity is compromised due to replay attacks using stolen or reused JSON Web Tokens (JWTs).
Recommendation: Comply with JWT best practices for enhanced security, paying special attention to the use of short-lived tokens, which reduce the window of opportunity for a replay attack. The exp claim can be used to set token expiration times.
EGTM-024 Excessive privileges via extension policies
ID
UID
Category
Priority
EGTM-024
EGTM-EG-008
Envoy Gateway
Medium
Risk: There is a risk of developers getting more privileges than required due to the use of SecurityPolicy, ClientTrafficPolicy, EnvoyPatchPolicy and BackendTrafficPolicy. These resources can be attached to a Gateway resource. Therefore, a developer with permission to deploy them would be able to modify a Gateway configuration by targeting the gateway in the policy manifest. This conflicts with the Advanced 4 Tier Model, where developers do not have write permissions on Gateways.
Threat: Excessive developer permissions lead to a misconfiguration and/or unauthorised access.
Recommendation: Considering the Tenant C scenario (represented in the Architecture Diagram), if a developer can create SecurityPolicy, ClientTrafficPolicy, EnvoyPatchPolicy or BackendTrafficPolicy objects in namespace C, they would be able to modify a Gateway configuration by attaching the policy to the gateway. In such scenarios, it is recommended to either:
a. Create a separate namespace, where developers have no permissions, to host tenant C's gateway. Note that, due to design decisions, the SecurityPolicy/EnvoyPatchPolicy/ClientTrafficPolicy/BackendTrafficPolicy object can only target resources deployed in the same namespace. Therefore, having a separate namespace for the gateway would prevent developers from attaching the policy to the gateway.
b. Forbid the creation of these policies for developers in namespace C.
On the other hand, in scenarios similar to tenants A and B, where a shared gateway namespace is in place, this issue is more limited. Note that in this scenario, developers don't have access to the shared gateway namespace.
In addition, it is important to mention that EnvoyPatchPolicy resources can also be attached to GatewayClass resources. This means that, in order to comply with the Advanced 4 Tier model, individuals with the Application Administrator role should not have access to this resource either.
Low Priority Findings
EGTM-003 Misconfiguration leads to insecure TLS settings
ID
UID
Category
Priority
EGTM-003
EGTM-EG-001
Envoy Gateway
Low
Risk: There is a risk that a threat actor could downgrade the security of proxied connections by configuring a weak set of cipher suites, compromising the confidentiality and integrity of proxied traffic.
Threat: Exploit weak cipher suite configuration to downgrade security of proxied connections.
Recommendation: Users operating in highly regulated environments may need to tightly control the TLS protocol and associated cipher suites, blocking non-conforming incoming connections to the gateway.
EnvoyProxy bootstrap config can be customised as per the customise EnvoyProxy documentation. In addition, from v.1.0.0, it is possible to configure common TLS properties for a Gateway or XRoute through the ClientTrafficPolicy object.
EGTM-005 Envoy Gateway Helm chart deployment does not set AppArmor and Seccomp profiles
ID
UID
Category
Priority
EGTM-005
EGTM-CP-002
Container Security
Low
Risk: Threat actor who has obtained access to Envoy Gateway pod could exploit the lack of AppArmor and Seccomp profiles in the Envoy Gateway deployment to attempt a container breakout, given the presence of an exploitable vulnerability, potentially impacting the confidentiality and integrity node resources.
Threat: Unauthorised syscalls and malicious code running in the Envoy Gateway pod.
Recommendation: Implement AppArmor policies by setting <container_name>: <profile_ref> within container.apparmor.security.beta.kubernetes.io (note, this config is set per container). Well-defined AppArmor policies may provide greater protection from unknown threats.
Enforce Seccomp profiles by setting the seccompProfile under securityContext. Ideally, a fine-grained profile should be used to restrict access to only necessary syscalls, however the --seccomp-default flag can be set to resort to RuntimeDefault which provides a container runtime specific. Example seccomp profiles can be found here.
To further enhance pod security, consider implementing SELinux via seLinuxOptions for additional syscall attack surface reduction. Setting readOnlyRootFilesystem == true enforces an immutable root filesystem, preventing the addition of malicious binaries to the PATH and increasing the attack cost. Together, these configuration items improve the pods Security Context.
EGTM-006 Envoy Proxy pods deployed with a shell enabled
ID
UID
Category
Priority
EGTM-006
EGTM-CS-004
Container Security
Low
Risk: There is a risk that a threat actor exploits a vulnerability in Envoy Proxy to expose a reverse shell, enabling them to compromise the confidentiality, integrity and availability of tenant data via a secondary attack.
Threat: If an external attacker managed to exploit a vulnerability in Envoy, the presence of a shell would be greatly helpful for the attacker in terms of potentially pivoting, escalating, or establishing some form of persistence.
Recommendation: By default, Envoy uses a distroless image since v.0.6.0, which does not ship a shell. Therefore, ensure EnvoyProxy image is up-to-date and patched with the latest stable version.
If using private EnvoyProxy images, use a lightweight EnvoyProxy image without a shell or debugging tool(s) which may be useful for an attacker.
An AuditPolicy (audit.k8s.io/v1beta1) can be configured to record API calls made within your cluster, allowing for identification of malicious traffic and enabling incident response. Requests are recorded based on stages which delineate between the lifecycle stage of the request made (e.g., RequestReceived, ResponseStarted, & ResponseComplete).
EGTM-011 Route Bindings on custom labels
ID
UID
Category
Priority
EGTM-011
EGTM-GW-003
Gateway API
Low
Risk: There is a risk that a gateway owner (or someone with the ability to set namespace labels) maliciously or accidentally binds routes across namespace boundaries, potentially compromising the confidentiality and integrity of traffic in a multitenant scenario.
Threat: If a Route Binding within a Gateway Listener is configured based on a custom label, it could allow a malicious internal actor with the ability to label namespaces to change the set of namespaces supported by the Gateway.
Recommendation: Consider the use of custom admission control to restrict what labels can be set on namespaces through tooling such as Kubewarden, Kyverno, and OPA Gatekeeper. Route binding should follow the Kubernetes Gateway API security model, as shown here, to connect gateways in different namespaces.
EGTM-013 GatewayClass namespace validation is not configured
ID
UID
Category
Priority
EGTM-013
EGTM-GW-005
Gateway API
Low
Risk: There is a risk that an unauthorised actor deploys an unauthorised GatewayClass due to GatewayClass namespace validation not being configured, leading to non-compliance with business and security requirements.
Threat: Unauthorised deployment of Gateway resource via GatewayClass template which crosses namespace trust boundaries.
Recommendation: Leverage GatewayClass namespace validation to limit the namespaces where GatewayClasses can be run through a tool such as OPA Gatekeeper. Reference pull request #24 within gatekeeper-library which outlines how to add GatewayClass namespace validation through a GatewayClassNamespaces API resource kind within the constraints.gatekeeper.sh/v1beta1 apiGroup.
EGTM-015 ServiceAccount token authentication
ID
UID
Category
Priority
EGTM-015
EGTM-CS-007
Container Security
Low
Risk: There is a risk that threat actors could exploit ServiceAccount tokens for illegitimate authentication, thereby leading to privilege escalation and the undermining of gateway API resources' integrity, confidentiality, and availability.
Threat: The threat arises from threat actors impersonating the envoy-gateway ServiceAccount through the replay of ServiceAccount tokens, thereby achieving escalated privileges and gaining unauthorised access to Kubernetes resources.
Recommendation: Limit the creation of ServiceAccounts to only when necessary, specifically refraining from using default service account tokens, especially for high-privilege service accounts. For legacy clusters running Kubernetes version 1.21 or earlier, note that ServiceAccount tokens are long-lived by default. To disable the automatic mounting of the service account token, set automountServiceAccountToken: false in the PodSpec.
EGTM-016 Misconfiguration leads to lack of Envoy Proxy access activity visibility
ID
UID
Category
Priority
EGTM-016
EGTM-EG-004
Envoy Gateway
Low
Risk: There is a risk that threat actors establish persistence and move laterally through the cluster unnoticed due to limited visibility into access and application-level activity.
Threat: Threat actors establish persistence and move laterally through the cluster unnoticed.
Additionally, consider leveraging a central logging mechanism such as Fluentd to enhance visibility into access activity and enable effective incident response (IR).
EGTM-017 Misconfiguration leads to lack of Envoy Gateway activity visibility
ID
UID
Category
Priority
EGTM-017
EGTM-EG-005
Envoy Gateway
Low
Risk: There is a risk that an insider misconfigures an envoy gateway component and goes unnoticed due to a low-touch logging configuration (via default) which responsible stakeholders are not aptly aware of or have immediate access to.
Threat: The threat emerges from an insider misconfiguring an Envoy Gateway component without detection.
Recommendation: Configure the logging level of the Envoy Gateway using the 'level' field in EnvoyGatewayLogging. Ensure the appropriate logging levels are set for relevant components such as 'gateway-api', 'xds-translator', or 'global-ratelimit'. If left unspecified, the logging level defaults to "info", which may not provide sufficient detail for security monitoring.
Employ a centralised logging mechanism, like Fluentd, to enhance visibility into application-level activity and to enable efficient incident response.
EGTM-021 Exposed Envoy Proxy admin interface
ID
UID
Category
Priority
EGTM-021
EGTM-EG-006
Envoy Gateway
Low
Risk: There is a risk that the admin interface is exposed without valid business reason, increasing the attack surface.
Threat: Exposed admin interfaces give internal attackers the option to affect production traffic in unauthorised ways, and the option to exploit any vulnerabilities which may be present in the admin interface (e.g. by orchestrating malicious GET requests to the admin interface through CSRF, compromising Envoy Proxy global configuration or shutting off the service entirely e.g. /quitquitquit).
Recommendation: The Envoy Proxy admin interface is only exposed to localhost, meaning that it is secure by default. However, due to the risk of misconfiguration, this recommendation is included.
Due to the importance of the admin interface, it is recommended to ensure that Envoy Proxies have not been accidentally misconfigured to expose the admin interface to untrusted networks.
EGTM-025 Envoy Proxy pods deployed running as root user in the container
ID
UID
Category
Priority
EGTM-025
EGTM-CS-011
Container Security
Low
Risk: The presence of a vulnerability, be it in the kernel or another system component, when coupled with containers running as root, could enable a threat actor to escape the container, thereby compromising the confidentiality, integrity, or availability of cluster resources
Threat: The Envoy Proxy container’s root-user configuration can be leveraged by an attacker to escalate privileges, execute a container breakout, and traverse across trust boundaries.
Recommendation: By default, Envoy Gateway deployments do not use root users. Nonetheless, in case a custom image or deployment manifest is to be used, make sure Envoy Proxy pods run as a non-root user with a high UID within the container.
Set runAsUser and runAsGroup security context options to specific UIDs (e.g., runAsUser: 1000 & runAsGroup: 3000) to ensure the container operates with the stipulated non-root user and group ID. If using helm chart deployment, define the user and group ID in the values.yaml file or via the command line during helm install / upgrade.
Appendix
In Scope Threat Actor Details
Threat Actor
Capability
Personal Motivation
Envoy Gateway Attack Samples
Application Developer
Leverage internal knowledge and personal access to the Envoy Gateway infrastructure to move laterally and transit trust boundaries
Disgruntled / personal grievances.
Financial incentives
Misconfigure XRoute resources to expose internal applications.
Misconfigure SecurityPolicy objects, reducing the security posture of an application.
Application Administrator
Abuse privileged status to disrupt operations and tenant cluster services through Envoy Gateway misconfig
Disgruntled / personal grievances.
Financial incentives
Create malicious routes to internal applications.
Introduce malicious Envoy Proxy images.
Expose the Envoy Proxy Admin interface.
Cluster Operator
Alter application-level deployments by misconfiguring resource dependencies & SCM to introduce vulnerabilities
Disgruntled / personal grievances.
Financial incentives.
Notoriety
Deploy malicious resources to expose internal applications.
Access authentication secrets.
Fall victim to phishing attacks and inadvertently share authentication credentials to cloud infrastructure or Kubernetes clusters.
Vandal: Script Kiddie, Trespasser
Uses publicly available tools and applications (Nmap,Metasploit, CVE PoCs)
Curiosity.
Personal fame through defacement / denial of service of prominent public facing web resources
Exploit public-facing application services such as the bastion host to gain an initial foothold in the environment
Motivated individual: Political activist, Thief, Terrorist
Write tools and exploits required for their means if sufficiently motivated.
Tend to use these in a targeted fashion against specific organisations. May combine publicly available exploits in a targeted fashion. Tamper with open source supply chains
Personal Gain (Political or Ideological)
Phishing, DDOS, exploit known vulnerabilities.
Compromise third-party components such as Helm charts and container images to inject malicious codes to propagate access throughout the environment.
Organised crime: syndicates, state-affiliated groups
Write tools and exploits required for their means.
Tend to use these in a non-targeted fashion, unless motivation is sufficiently high.
Devotes considerable resources, writes exploits, can bribe/coerce, can launch targeted attacks
Ransom.
Mass extraction of PII / credentials / PCI data.
Financial incentives
Social Engineering, phishing, ransomware, coordinated attacks.
Intercept and replay JWT tokens (via MiTM) between tenant user(s) and envoy gateway to modify app configs in-transit
Identified Threats by Priority
ID
UID
Category
Risk
Threat
Priority
Recommendation
EGTM-001
EGTM-GW-001
Gateway API
Self-signed certificates (which do not comply with PKI best practices) could lead to unauthorised access to the private key associated with the certificate used for inbound TLS termination at Envoy Proxy, compromising the confidentiality and integrity of proxied traffic.
Compromise of the private key associated with the certificate used for inbound TLS terminating at Envoy Proxy.
High
The Envoy Gateway quickstart demonstrates how to set up a Secure Gateway using an example where a self-signed root certificate is created using openssl. As stated in the Envoy Gateway documentation, this is not a suitable configuration for Production usage. It is recommended that PKI best practices are followed, whereby certificates are signed by an Intermediary CA which sits underneath an organisational 'offline' Root CA.
PKI best practices should also apply to the management of client certificates when using mTLS. The Envoy Gateway mTLS task shows how to set up client certificates using self-signed certificates. In the same way as gateway certificates and, as mentioned in the documentation, this configuration should not be used in production environments.
EGTM-002
EGTM-CS-001
Container Security
There is a risk that a threat actor could compromise the Kubernetes secret containing the Envoy private key, allowing the attacker to decrypt Envoy Proxy traffic, compromising the confidentiality of proxied traffic.
Kubernetes secret containing the Envoy private key is compromised and used to decrypt proxied traffic.
High
Certificate management best practices mandate short-lived key material where practical, meaning that a mechanism for rotation of private keys and certificates is required, along with a way for certificates to be mounted into Envoy containers. If Kubernetes secrets are used, when a certificate expires, the associated secret must be updated, and Envoy containers must be redeployed. Instead of a manual configuration, it is recommended that cert-manager is used.
EGTM-004
EGTM-K8-002
Container Security
There is a risk that a threat actor could abuse misconfigured RBAC to access the Envoy Gateway ClusterRole (envoy-gateway-role) and use it to expose all secrets across the cluster, thus compromising the confidentiality and integrity of tenant data.
Compromised Envoy Gateway or misconfigured ClusterRoleBinding (envoy-gateway-rolebinding) to Envoy Gateway ClusterRole (envoy-gateway-role), provides access to resources and secrets in different namespaces.
High
Users should be aware that Envoy Gateway uses a ClusterRole (envoy-gateway-role) when deployed via the Helm chart, to allow management of Envoy Proxies across different namespaces. This ClusterRole is powerful and includes the ability to read secrets in namespaces which may not be within the purview of Envoy Gateway.
Kubernetes best-practices involve restriction of ClusterRoleBindings, with the use of RoleBindings where possible to limit access per namespace by specifying the namespace in metadata. Namespace isolation reduces the impact of compromise from cluster-scoped roles. Ideally, fine-grained K8s roles should be created per the principle of least privilege to ensure they have the minimum access necessary for role functions.
The pull request #1656 introduced the use of Roles and RoleBindings in namespaced mode. This feature can be leveraged to reduce the amount of permissions required by the Envoy Gateway.
EGTM-007
EGTM-EG-002
Envoy Gateway
There is a risk that a threat actor could exploit misconfigured Kubernetes RBAC to create or modify Gateway API resources with no business need, potentially leading to the compromise of the confidentiality, integrity, and availability of resources and traffic within the cluster.
Unauthorised creation or misconfiguration of Gateway API resources by a threat actor with cluster-scoped access.
High
Configure the apiGroup and resource fields in RBAC policies to restrict access to Gateway and GatewayClass resources. Enable namespace isolation by using the namespace field, preventing unauthorised access to gateways in other namespaces.
EGTM-009
EGTM-GW-002
Gateway API
There is a risk that a co-tenant misconfigures Gateway or Route resources, compromising the confidentiality, integrity, and availability of routed traffic through Envoy Gateway.
Malicious or accidental co-tenant misconfiguration of Gateways and Routes associated with other application teams.
High
Dedicated Envoy Gateways should be provided to each tenant within their respective namespace. A one-to-one relationship should be established between GatewayClass and Gateway resources, meaning that each tenant namespace should have their own GatewayClass watched by a unique Envoy Gateway Controller as defined here in the Deployment Mode documentation.
Application Admins should have write permissions on the Gateway resource, but only in their specific namespaces, and Application Developers should only hold write permissions on Route resources. To enact this access control schema, follow the Write Permissions for Advanced 4 Tier Model described in the Kubernetes Gateway API security model. Examples of secured gateway-route topologies can be found here within the Kubernetes Gateway API docs.
Optionally, consider a GitOps model, where only the GitOps operator has the permission to deploy or modify custom resources in production.
EGTM-014
EGTM-CS-006
Container Security
There is a risk that a supply chain attack on Envoy Gateway results in an arbitrary compromise of the confidentiality, integrity or availability of tenant data.
Supply chain threat actor introduces malicious code into Envoy Gateway or Proxy.
High
The Envoy Gateway project should continue to work towards conformance with supply-chain security best practices throughout the project lifecycle (for example, as set out in the CNCF Software Supply Chain Best Practices Whitepaper. Adherence to Supply-chain Levels for Software Artefacts (SLSA) standards is crucial for maintaining the security of the system. Employ version control systems to monitor the source and build platforms and assign responsibility to a specific stakeholder.
Integrate a supply chain security tool such as Sigstore, which provides native capabilities for signing and verifying container images and software artefacts. Software Bill of Materials (SBOM), Vulnerability Exploitability eXchange (VEX), and signed artefacts should also be incorporated into the security protocol.
EGTM-020
EGTM-CS-009
Container Security
There is a risk that a threat actor exploits an Envoy Proxy vulnerability to remote code execution (RCE) due to out of date or misconfigured Envoy Proxy pod deployment, compromising the confidentiality and integrity of Envoy Proxy along with the availability of the proxy service.
Deployment of an Envoy Proxy or Gateway image containing exploitable CVEs.
High
Always use the latest version of the Envoy Proxy image. Regularly check for updates and patch the system as soon as updates become available. Implement a CI/CD pipeline that includes security checks for images and prevents deployment of insecure configurations. A tool such as Snyk can be used to provide container vulnerability scanning to mitigate the risk of known vulnerabilities.
There is a risk that the OIDC client secret (for OIDC authentication) and user password hashes (for basic authentication) get leaked due to misconfigured RBAC permissions.
Unauthorised access to the application due to credential leakage.
High
Ensure that only authorised users and service accounts are able to access secrets. This is especially important in namespaces where SecurityPolicy objects are configured, since those namespaces are the ones to store secrets containing the client secret (in OIDC scenarios) and user password hashes (in basic authentication scenarios).
To do so, minimise the use of ClusterRoles and Roles allowing listing and getting secrets. Perform periodic audits of RBAC permissions.
EGTM-023
EGTM-EG-007
Envoy Gateway
There is a risk of unauthorised access due to the use of basic authentication, which does not enforce any password restriction in terms of complexity and length. In addition, password hashes are stored in SHA1 format, which is a deprecated hashing function.
Unauthorised access to the application due to weak authentication mechanisms.
High
It is recommended to make use of stronger authentication mechanisms (i.e. JWT authentication and OIDC authentication) instead of basic authentication. These authentication mechanisms have many advantages, such as the use of short-lived credentials and a central management of security policies through the identity provider.
EGTM-008
EGTM-EG-003
Envoy Gateway
There is a risk of a threat actor misconfiguring static config and compromising the integrity of Envoy Gateway, ultimately leading to the compromised confidentiality, integrity, or availability of tenant data and cluster resources.
Accidental or deliberate misconfiguration of static configuration leads to a misconfigured deployment of Envoy Gateway, for example logging parameters could be modified or global rate limiting configuration misconfigured.
Medium
Implement a GitOps model, utilising Kubernetes' Role-Based Access Control (RBAC) and adhering to the principle of least privilege to minimise human intervention on the cluster. For instance, tools like ArgoCD can be used for declarative GitOps deployments, ensuring all changes are tracked and reviewed. Additionally, configure your source control management (SCM) system to include mandatory pull request (PR) reviews, commit signing, and protected branches to ensure only authorised changes can be committed to the start-up configuration.
EGTM-010
EGTM-CS-005
Container Security
There is a risk that a threat actor exploits a weak pod security context, compromising the CIA of a node and the resources / services which run on it.
Threat Actor who has compromised a pod exploits weak security context to escape to a node, potentially leading to the compromise of Envoy Proxy or Gateway running on the same node.
Medium
To mitigate this risk, apply Pod Security Standards at a minimum of Baseline level to all namespaces, especially those containing Envoy Gateway and Proxy Pods. Pod security standards are implemented through K8s Pod Security Admission to provide admission control modes (enforce, audit, and warn) for namespaces. Pod security standards can be enforced by namespace labels as shown here, to enforce a baseline level of pod security to specific namespaces.
Further enhance the security by implementing a sandboxing solution such as gVisor for Envoy Gateway and Proxy Pods to isolate the application from the host kernel. This can be set within the runtimeClassName of the Pod specification.
EGTM-012
EGTM-GW-004
Gateway API
There is a risk that a threat actor could abuse excessive RBAC privileges to create ReferenceGrant resources. These resources could then be used to create cross-namespace communication, leading to unauthorised access to the application. This could compromise the confidentiality and integrity of resources and configuration in the affected namespaces and potentially disrupt the availability of services that rely on these object references.
A ReferenceGrant is created, which validates traffic to cross namespace trust boundaries without a valid business reason, such as a route in one tenant's namespace referencing a backend in another.
Medium
Ensure that the ability to create ReferenceGrant resources is restricted to the minimum number of people. Pay special attention to ClusterRoles that allow that action.
EGTM-018
EGTM-GW-006
Gateway API
There is a risk that malicious requests could lead to a Denial of Service (DoS) attack, thereby reducing API gateway availability due to misconfigurations in rate-limiting or load balancing controls, or a lack of route timeout enforcement.
Reduced API gateway availability due to an attacker's maliciously crafted request (e.g., QoD) potentially inducing a Denial of Service (DoS) attack.
Medium
To ensure high availability and to mitigate potential security threats, adhere to the Envoy Gateway documentation for the configuration of a rate-limiting filter and load balancing.
Path normalisation should be enabled to minimise path confusion vulnerabilities. These measures help protect against volumetric threats such as Denial of Service (DoS)nattacks. Utilise custom resources to implement policy attachment, thereby exposing request limit configuration for route types.
EGTM-019
EGTM-DP-004
Container Security
There is a risk that replay attacks using stolen or reused JSON Web Tokens (JWTs) can compromise transmission integrity, thereby undermining the confidentiality and integrity of the data plane.
Transmission integrity is compromised due to replay attacks using stolen or reused JSON Web Tokens (JWTs).
Medium
Comply with JWT best practices for enhanced security, paying special attention to the use of short-lived tokens, which reduce the window of opportunity for a replay attack. The exp claim can be used to set token expiration times.
EGTM-024
EGTM-EG-008
Envoy Gateway
There is a risk of developers getting more privileges than required due to the use of SecurityPolicy, ClientTrafficPolicy, EnvoyPatchPolicy and BackendTrafficPolicy. These resources can be attached to a Gateway resource. Therefore, a developer with permission to deploy them would be able to modify a Gateway configuration by targeting the gateway in the policy manifest. This conflicts with the Advanced 4 Tier Model, where developers do not have write permissions on Gateways.
Excessive developer permissions lead to a misconfiguration and/or unauthorised access.
Medium
Considering the Tenant C scenario (represented in the Architecture Diagram), if a developer can create SecurityPolicy, ClientTrafficPolicy, EnvoyPatchPolicy or BackendTrafficPolicy objects in namespace C, they would be able to modify a Gateway configuration by attaching the policy to the gateway. In such scenarios, it is recommended to either:
a. Create a separate namespace, where developers have no permissions, > to host tenant C's gateway. Note that, due to design decisions, > the > SecurityPolicy/EnvoyPatchPolicy/ClientTrafficPolicy/BackendTrafficPolicy > object can only target resources deployed in the same namespace. > Therefore, having a separate namespace for the gateway would > prevent developers from attaching the policy to the gateway.
b. Forbid the creation of these policies for developers in namespace C.
On the other hand, in scenarios similar to tenants A and B, where a shared gateway namespace is in place, this issue is more limited. Note that in this scenario, developers don't have access to the shared gateway namespace.
In addition, it is important to mention that EnvoyPatchPolicy resources can also be attached to GatewayClass resources. This means that, in order to comply with the Advanced 4 Tier model, individuals with the Application Administrator role should not have access to this resource either.
EGTM-003
EGTM-EG-001
Envoy Gateway
There is a risk that a threat actor could downgrade the security of proxied connections by configuring a weak set of cipher suites, compromising the confidentiality and integrity of proxied traffic.
Exploit weak cipher suite configuration to downgrade security of proxied connections.
Low
Users operating in highly regulated environments may need to tightly control the TLS protocol and associated cipher suites, blocking non-conforming incoming connections to the gateway.
EnvoyProxy bootstrap config can be customised as per the customise EnvoyProxy documentation. In addition, from v.1.0.0, it is possible to configure common TLS properties for a Gateway or XRoute through the ClientTrafficPolicy object.
EGTM-005
EGTM-CP-002
Container Security
Threat actor who has obtained access to Envoy Gateway pod could exploit the lack of AppArmor and Seccomp profiles in the Envoy Gateway deployment to attempt a container breakout, given the presence of an exploitable vulnerability, potentially impacting the confidentiality and integrity of namespace resources.
Unauthorised syscalls and malicious code running in the Envoy Gateway pod.
Low
Implement AppArmor policies by setting <container_name>: <profile_ref> within container.apparmor.security.beta.kubernetes.io (note, this config is set per container). Well-defined AppArmor policies may provide greater protection from unknown threats.
Enforce Seccomp profiles by setting the seccompProfile under securityContext. Ideally, a fine-grained profile should be used to restrict access to only necessary syscalls, however the --seccomp-default flag can be set to resort to RuntimeDefault which provides a container runtime specific. Example seccomp profiles can be found here.
To further enhance pod security, consider implementing SELinux via seLinuxOptions for additional syscall attack surface reduction. Setting readOnlyRootFilesystem == true enforces an immutable root filesystem, preventing the addition of malicious binaries to the PATH and increasing the attack cost. Together, these configuration items improve the pods Security Context.
EGTM-006
EGTM-CS-004
Container Security
There is a risk that a threat actor exploits a vulnerability in Envoy Proxy to expose a reverse shell, enabling them to compromise the confidentiality, integrity and availability of tenant data via a secondary attack.
If an external attacker managed to exploit a vulnerability in Envoy, the presence of a shell would be greatly helpful for the attacker in terms of potentially pivoting, escalating, or establishing some form of persistence.
Low
By default, Envoy uses a distroless image since v.0.6.0, which does not ship a shell. Therefore, ensure EnvoyProxy image is up-to-date and patched with the latest stable version.
If using private EnvoyProxy images, use a lightweight EnvoyProxy image without a shell or debugging tool(s) which may be useful for an attacker.
An AuditPolicy (audit.k8s.io/v1beta1) can be configured to record API calls made within your cluster, allowing for identification of malicious traffic and enabling incident response. Requests are recorded based on stages which delineate between the lifecycle stage of the request made (e.g., RequestReceived, ResponseStarted, & ResponseComplete).
EGTM-011
EGTM-GW-003
Gateway API
There is a risk that a gateway owner (or someone with the ability to set namespace labels) maliciously or accidentally binds routes across namespace boundaries, potentially compromising the confidentiality and integrity of traffic in a multitenant scenario.
If a Route Binding within a Gateway Listener is configured based on a custom label, it could allow a malicious internal actor with the ability to label namespaces to change the set of namespaces supported by the Gateway
Low
Consider the use of custom admission control to restrict what labels can be set on namespaces through tooling such as Kubewarden, Kyverno, and OPA Gatekeeper. Route binding should follow the Kubernetes Gateway API security model, as shown here, to connect gateways in different namespaces.
EGTM-013
EGTM-GW-005
Gateway API
There is a risk that an unauthorised actor deploys an unauthorised GatewayClass due to GatewayClass namespace validation not being configured, leading to non-compliance with business and security requirements.
Unauthorised deployment of Gateway resource via GatewayClass template which crosses namespace trust boundaries.
Low
Leverage GatewayClass namespace validation to limit the namespaces where GatewayClasses can be run through a tool such as using OPA Gatekeeper. Reference pull request #24 within gatekeeper-library which outlines how to add GatewayClass namespace validation through a GatewayClassNamespaces API resource kind within the constraints.gatekeeper.sh/v1beta1 apiGroup.
EGTM-015
EGTM-CS-007
Container Security
There is a risk that threat actors could exploit ServiceAccount tokens for illegitimate authentication, thereby leading to privilege escalation and the undermining of gateway API resources' integrity, confidentiality, and availability.
The threat arises from threat actors impersonating the envoy-gateway ServiceAccount through the replay of ServiceAccount tokens, thereby achieving escalated privileges and gaining unauthorised access to Kubernetes resources.
Low
Limit the creation of ServiceAccounts to only when necessary, specifically refraining from using default service account tokens, especially for high-privilege service accounts. For legacy clusters running Kubernetes version 1.21 or earlier, note that ServiceAccount tokens are long-lived by default. To disable the automatic mounting of the service account token, set automountServiceAccountToken: false in the PodSpec.
EGTM-016
EGTM-EG-004
Envoy Gateway
There is a risk that threat actors establish persistence and move laterally through the cluster unnoticed due to limited visibility into access and application-level activity.
Threat actors establish persistence and move laterally through the cluster unnoticed.
Additionally, consider leveraging a central logging mechanism such as Fluentd to enhance visibility into access activity and enable effective incident response (IR).
EGTM-017
EGTM-EG-005
Envoy Gateway
There is a risk that an insider misconfigures an envoy gateway component and goes unnoticed due to a low-touch logging configuration (via default) which responsible stakeholders are not aptly aware of or have immediate access to.
The threat emerges from an insider misconfiguring an Envoy Gateway component without detection.
Low
Configure the logging level of the Envoy Gateway using the 'level' field in EnvoyGatewayLogging. Ensure the appropriate logging levels are set for relevant components such as 'gateway-api', 'xds-translator', or 'global-ratelimit'. If left unspecified, the logging level defaults to "info", which may not provide sufficient detail for security monitoring.
Employ a centralised logging mechanism, like Fluentd, to enhance visibility into application-level activity and to enable efficient incident response.
EGTM-021
EGTM-EG-006
Envoy Gateway
There is a risk that the admin interface is exposed without valid business reason, increasing the attack surface.
Exposed admin interfaces give internal attackers the option to affect production traffic in unauthorised ways, and the option to exploit any vulnerabilities which may be present in the admin interface (e.g. by orchestrating malicious GET requests to the admin interface through CSRF, compromising Envoy Proxy global configuration or shutting off the service entirely (e.g., /quitquitquit).
Low
The Envoy Proxy admin interface is only exposed to localhost, meaning that it is secure by default. However, due to the risk of misconfiguration, this recommendation is included.
Due to the importance of the admin interface, it is recommended to ensure that Envoy Proxies have not been accidentally misconfigured to expose the admin interface to untrusted networks.
EGTM-025
EGTM-CS-011
Container Security
The presence of a vulnerability, be it in the kernel or another system component, when coupled with containers running as root, could enable a threat actor to escape the container, thereby compromising the confidentiality, integrity, or availability of cluster resources.
The Envoy Proxy container’s root-user configuration can be leveraged by an attacker to escalate privileges, execute a container breakout, and traverse across trust boundaries.
Low
By default, Envoy Gateway deployments do not use root users. Nonetheless, in case a custom image or deployment manifest is to be used, make sure Envoy Proxy pods run as a non-root user with a high UID within the container. Set runAsUser and runAsGroup security context options to specific UIDs (e.g., runAsUser: 1000 & runAsGroup: 3000) to ensure the container operates with the stipulated non-root user and group ID. If using helm chart deployment, define the user and group ID in the values.yaml file or via the command line during helm install / upgrade.
Attack Trees
Attack trees offer a methodical way of describing the security of systems, based on varying attack patterns. It’s important to approach the review of attack trees from a top-down perspective. The top node, also known as the root node, symbolises the attacker’s primary objective. This goal is then broken down into subsidiary aims, each reflecting a different strategy to attain the root objective. This deconstruction persists until reaching the lowest level objectives or ’leaf nodes’, which depict attacks that can be directly launched.
It is essential to note that attack trees presented here are speculative paths for potential exploitation. The Envoy Gateway project is in a continuous development cycle, and as the project evolves, new vulnerabilities may be exposed, or additional controls could be introduced. Therefore, the threats illustrated in the attack trees should be perceived as point-in-time reflections of the project’s current state at the time of writing this threat model.
Node ID Schema
Each node in the attack tree is assigned a unique identifier following the AT#-## schema. This allows easy reference to specific nodes in the attack trees throughout the threat model. The first part of the ID (AT#) signifies the attack tree number, while the second part (##) represents the node number within that tree.
Logical Operators
Logical AND/OR operators are used to represent the relationship between parent and child nodes. An AND operator means that all child nodes must be achieved to satisfy the parent node. An OR operator between a parent node and its child nodes means that any of the child nodes can be achieved to satisfy the parent node.
Attack Tree Node Legend
AT0
AT1
AT2
AT3
AT4
AT5
AT6
1.3.11 - TLS Passthrough
This task will walk through the steps required to configure TLS Passthrough via Envoy Gateway. Unlike configuring
Secure Gateways, where the Gateway terminates the client TLS connection, TLS Passthrough allows the application itself
to terminate the TLS connection, while the Gateway routes the requests to the application based on SNI headers.
Prerequisites
OpenSSL to generate TLS assets.
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
TLS Certificates
Generate the certificates and keys used by the Service to terminate client TLS connections.
For the application, we’ll deploy a sample echoserver app, with the certificates loaded in the application Pod.
Note: These certificates will not be used by the Gateway, but will remain in the application scope.
Create a root certificate and private key to sign certificates:
Follow the steps from the Quickstart to uninstall Envoy Gateway and the example manifest.
Delete the Secret:
kubectl delete secret/server-certs
Next Steps
Checkout the Developer Guide to get involved in the project.
1.3.12 - TLS Termination for TCP
This task will walk through the steps required to configure TLS Terminate mode for TCP traffic via Envoy Gateway.
This task uses a self-signed CA, so it should be used for testing and demonstration purposes only.
Prerequisites
OpenSSL to generate TLS assets.
Installation
Follow the steps from the Quickstart to install Envoy Gateway.
TLS Certificates
Generate the certificates and keys used by the Gateway to terminate client TLS connections.
Create a root certificate and private key to sign certificates:
This task shows how to set up cert-manager to automatically create certificates and secrets for use by Envoy Gateway.
It will first show how to enable the self-sign issuer, which is useful to test that cert-manager and Envoy Gateway can talk to each other.
Then it shows how to use Let’s Encrypt’s staging environment.
Changing to the Let’s Encrypt production environment is straight-forward after that.
Prerequisites
A Kubernetes cluster and a configured kubectl.
The helm command.
The curl command or similar for testing HTTPS requests.
For the ACME HTTP-01 challenge to work
your Gateway must be reachable on the public Internet.
the domain name you use (we use www.example.com) must point to the Gateway’s external IP(s).
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
You should now have cert-manager running with nothing to do:
$ kubectl wait --for=condition=Available deployment -n cert-manager --all
deployment.apps/cert-manager condition met
deployment.apps/cert-manager-cainjector condition met
deployment.apps/cert-manager-webhook condition met
$ kubectl get -n cert-manager deployment
NAME READY UP-TO-DATE AVAILABLE AGE
cert-manager 1/1 1 1 42m
cert-manager-cainjector 1/1 1 1 42m
cert-manager-webhook 1/1 1 1 42m
A Self-Signing Issuer
cert-manager can have any number of issuer configurations.
The simplest issuer type is SelfSigned.
It simply takes the certificate request and signs it with the private key it generates for the TLS Secret.
Self-signed certificates don't provide any help in establishing trust between certificates.
However, they are great for initial testing, due to their simplicity.
You could instead create a new Gateway serving HTTPS, if you’d prefer.
cert-manager doesn’t care, but we’ll keep it all together in this task.
Nowadays, X.509 certificates don’t use the subject Common Name for hostname matching, so you can set it to whatever you want, or leave it empty.
The important parts here are
the annotation referencing the “selfsigned” ClusterIssuer we created above,
the hostname, which is required (but see #6051 for computing it based on attached HTTPRoutes), and
the named Secret, which is what cert-manager will create for us.
In the interaction between the two, cert-manager does all the heavy lifting.
It subscribes to changes to Gateway resources (using the gateway-shim component.)
For any Gateway it finds, it looks for any TLS listeners, and the associated tls.certificateRefs.
Note that while Gateway API supports multiple refs here, Envoy Gateway only uses one.
cert-manager also looks at the hostname of the listener to figure out which hosts the certificate is expected to cover.
More than one listener can use the same certificate Secret, which means cert-manager needs to find all listeners using the same Secret before deciding what to do.
If the certificatRef points to a valid certificate, given the hostnames found in listeners, cert-manager has nothing to do.
If there is no valid certificate, or it is about to expire, cert-manager’s gateway-shim creates a Certificate resource, or updates the existing one.
cert-manager then follows the Certificate Lifecycle.
To know how to issue the certificate, an ClusterIssuer is configured, and referenced through annotations on the Gateway resource, which you did above.
Once a matching ClusterIssuer is found, that plugin does what needs to be done to acquire a signed certificate.
In the case of the ACME protocol (used by Let’s Encrypt), cert-manager can also use an HTTP Gateway to solve the HTTP-01 challenge type.
This is the other side of cert-manager’s Gateway API support:
the ACME issuer creates a temporary HTTPRoute, lets the ACME server(s) query it, and deletes it again.
cert-manager then updates the Secret that the Gateway’s listener points to in tls.certificateRefs.
Envoy Gateway picks up that the Secret has changed, and reloads the corresponding Envoy Proxy Deployments with the new private key and certificate.
As you can imagine, cert-manager requires quite broad permissions to update Secrets in any namespace, so the security-minded reader may want to look at the RBAC resources the Helm chart creates.
Using the ACME Issuer With Let’s Encrypt and HTTP-01
We will start using the Let’s Encrypt staging environment, to spare their production environment.
Our Gateway already contains an HTTP listener, so we will use that for the HTTP-01 challenges.
You probably want to set the cert-manager.io/revision-history-limit annotation on your Gateway to make cert-manager prune the CertificateRequest history.
cert-manager deletes unused Certificate resources, and they are updated in-place when possible, so there should be no need for cleaning up Certificate resources.
The deletion is based on whether a Gateway still holds a tls.certificateRefs that requires the Certificate.
If you remove a TLS listener from a Gateway, you may still have a Secret lingering.
cert-manager can clean it up using a flag.
Issuer Namespaces
We have used ClusterIssuer resources in this tutorial.
They are not bound to any namespace, and will read annotations from Gateways in any namespace.
You could also use Issuer, which is bound to a namespace.
This is useful e.g. if you want to use different ACME accounts for different namespaces.
If you change the issuer kind, you also need to change the annotation key from cert-manager.io/clusterissuer to cert-manager.io/issuer.
Using ExternalDNS
The ExternalDNS controller maintains DNS records based on Kubernetes resources.
Together with cert-manager, this can be used to fully automate hostname management.
It can use various source resources, among them Gateway Routes.
Just specify a Gateway Route resource, let ExternalDNS create the domain records, and then cert-manager the TLS certificate.
The tutorial on Gateway API uses kubectl.
They also have a Helm chart, which is easier to customize.
The only thing relevant to Envoy Gateway is to set the sources:
The Issuer has a Ready condition (though this is rather boring for the selfSigned type):
$ kubectl get issuer --all-namespaces
NAMESPACE NAME READY AGE
default selfsigned True 42m
The Gateway will say when it has an invalid certificate:
$ kubectl describe gateway/eg
...
Conditions:
Message: Secret default/eg-https does not exist.
Reason: InvalidCertificateRef
Status: False
Type: ResolvedRefs
...
Message: Listener is invalid, see other Conditions for details.
Reason: Invalid
Status: False
Type: Programmed
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BadConfig 42m cert-manager-gateway-shim Skipped a listener block: spec.listeners[1].hostname: Required value: the hostname cannot be empty
The main question is if cert-manager has picked up on the Gateway.
I.e., has it created a Certificate for it?
The above describe contains an event from cert-manager-gateway-shim telling you of one such issue.
Be aware that if you have a non-TLS listener in the Gateway, like we did, there will be events saying it is not eligible, which is of course expected.
Another option is looking at Deployment logs.
The cert-manager logs are not very verbose by default, but setting the Helm value global.logLevel to 6 will enable all debug logs (the default is 2.)
This will make them verbose enough to say why a Gateway wasn’t considered (e.g. missing hostname, or tls.mode is not Terminate.)
Simply listing Certificate resources may be useful, even if it just gives a yes/no answer:
$ kubectl get certificate --all-namespaces
NAMESPACE NAME READY SECRET AGE
default eg-https True eg-https 42m
If there is a Certificate, then the gateway-shim has recognized the Gateway.
But is there a CertificateRequest for it?
(BTW, don’t confuse this with a CertificateSigningRequest, which is a Kubernetes core resource type representing the same thing.)
$ kubectl get certificaterequest --all-namespaces
NAMESPACE NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
default eg-https-xxxxx True True selfsigned system:serviceaccount:cert-manager:cert-manager 42m
The ACME issuer also has Order and Challenge resources to watch:
$ kubectl get order --all-namespaces -o wide
NAME STATE ISSUER REASON AGE
order.acme.cert-manager.io/envoy-https-xxxxx-123456789 pending letsencrypt-staging 42m
$ kubectl get challenge --all-namespaces
NAME STATE DOMAIN AGE
challenge.acme.cert-manager.io/envoy-https-xxxxx-123456789-1234567890 pending www.example.com 42m
Using kubetctl get -o wide or kubectl describe on the Challenge will explain its state more.
$ kubectl get order --all-namespaces -o wide
NAME STATE ISSUER REASON AGE
order.acme.cert-manager.io/envoy-https-xxxxx-123456789 valid letsencrypt-staging 42m
Finally, since cert-manager creates the Secret referenced by the Gateway listener as its last step, we can also look for that:
$ kubectl get secret secret/eg-https
NAME TYPE DATA AGE
eg-https kubernetes.io/tls 3 42m
This task explains the usage of the EnvoyPatchPolicy API.
Note: This API is meant for users extremely familiar with Envoy xDS semantics.
Also before considering this API for production use cases, please be aware that this API
is unstable and the outcome may change across versions. Use at your own risk.
Introduction
The EnvoyPatchPolicy API allows user to modify the output xDS
configuration generated by Envoy Gateway intended for EnvoyProxy,
using JSON Patch semantics.
Motivation
This API was introduced to allow advanced users to be able to leverage Envoy Proxy functionality
not exposed by Envoy Gateway APIs today.
Quickstart
Prerequisites
Follow the steps from the Quickstart task to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
The default installation of Envoy Gateway installs a default EnvoyGateway configuration and attaches it
using a ConfigMap. In the next step, we will update this resource to enable EnvoyPatchPolicy.
Use EnvoyProxy’s Local Reply Modification feature to return a custom response back to the client when
the status code is 404
Apply the configuration
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
name: custom-response-patch-policy
namespace: default
spec:
targetRef:
group: gateway.networking.k8s.io
kind: Gateway
name: eg
namespace: default
type: JSONPatch
jsonPatches:
- type: "type.googleapis.com/envoy.config.listener.v3.Listener"
# The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
name: default/eg/http
operation:
op: add
path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
value:
mappers:
- filter:
status_code_filter:
comparison:
op: EQ
value:
default_value: 404
runtime_key: key_b
status_code: 406
body:
inline_string: "could not find what you are looking for"
EOF
Save and apply the following resource to your cluster:
---apiVersion:gateway.envoyproxy.io/v1alpha1kind:EnvoyPatchPolicymetadata:name:custom-response-patch-policynamespace:defaultspec:targetRef:group:gateway.networking.k8s.iokind:Gatewayname:egnamespace:defaulttype:JSONPatchjsonPatches:- type:"type.googleapis.com/envoy.config.listener.v3.Listener"# The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>name:default/eg/httpoperation:op:addpath:"/default_filter_chain/filters/0/typed_config/local_reply_config"value:mappers:- filter:status_code_filter:comparison:op:EQvalue:default_value:404runtime_key:key_bstatus_code:406body:inline_string:"could not find what you are looking for"
When mergeGateways is enabled, there will be one Envoy deployment for all Gateways in the cluster.
Then the EnvoyPatchPolicy should target a specific GatewayClass.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyPatchPolicy
metadata:
name: custom-response-patch-policy
namespace: default
spec:
targetRef:
group: gateway.networking.k8s.io
kind: GatewayClass
name: eg
namespace: default
type: JSONPatch
jsonPatches:
- type: "type.googleapis.com/envoy.config.listener.v3.Listener"
# The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>
name: default/eg/http
operation:
op: add
path: "/default_filter_chain/filters/0/typed_config/local_reply_config"
value:
mappers:
- filter:
status_code_filter:
comparison:
op: EQ
value:
default_value: 404
runtime_key: key_b
status_code: 406
body:
inline_string: "could not find what you are looking for"
EOF
Save and apply the following resource to your cluster:
---apiVersion:gateway.envoyproxy.io/v1alpha1kind:EnvoyPatchPolicymetadata:name:custom-response-patch-policynamespace:defaultspec:targetRef:group:gateway.networking.k8s.iokind:GatewayClassname:egnamespace:defaulttype:JSONPatchjsonPatches:- type:"type.googleapis.com/envoy.config.listener.v3.Listener"# The listener name is of the form <GatewayNamespace>/<GatewayName>/<GatewayListenerName>name:default/eg/httpoperation:op:addpath:"/default_filter_chain/filters/0/typed_config/local_reply_config"value:mappers:- filter:status_code_filter:comparison:op:EQvalue:default_value:404runtime_key:key_bstatus_code:406body:inline_string:"could not find what you are looking for"
Edit the HTTPRoute resource from the Quickstart to only match on paths with value /get
$ curl --header "Host: www.example.com" http://localhost:8888/find
Handling connection for 8888
could not find what you are looking for
Debugging
Runtime
The Status subresource should have information about the status of the resource. Make sure
Accepted=True and Programmed=True conditions are set to ensure that the policy has been
applied to Envoy Proxy.
apiVersion:gateway.envoyproxy.io/v1alpha1kind:EnvoyPatchPolicymetadata:annotations:kubectl.kubernetes.io/last-applied-configuration:| {"apiVersion":"gateway.envoyproxy.io/v1alpha1","kind":"EnvoyPatchPolicy","metadata":{"annotations":{},"name":"custom-response-patch-policy","namespace":"default"},"spec":{"jsonPatches":[{"name":"default/eg/http","operation":{"op":"add","path":"/default_filter_chain/filters/0/typed_config/local_reply_config","value":{"mappers":[{"body":{"inline_string":"could not find what you are looking for"},"filter":{"status_code_filter":{"comparison":{"op":"EQ","value":{"default_value":404}}}}}]}},"type":"type.googleapis.com/envoy.config.listener.v3.Listener"}],"priority":0,"targetRef":{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"eg","namespace":"default"},"type":"JSONPatch"}}creationTimestamp:"2023-07-31T21:47:53Z"generation:1name:custom-response-patch-policynamespace:defaultresourceVersion:"10265"uid:a35bda6e-a0cc-46d7-a63a-cee765174bc3spec:jsonPatches:- name:default/eg/httpoperation:op:addpath:/default_filter_chain/filters/0/typed_config/local_reply_configvalue:mappers:- body:inline_string:could not find what you are looking forfilter:status_code_filter:comparison:op:EQvalue:default_value:404type:type.googleapis.com/envoy.config.listener.v3.Listenerpriority:0targetRef:group:gateway.networking.k8s.iokind:Gatewayname:egnamespace:defaulttype:JSONPatchstatus:conditions:- lastTransitionTime:"2023-07-31T21:48:19Z"message:EnvoyPatchPolicy has been accepted.observedGeneration:1reason:Acceptedstatus:"True"type:Accepted- lastTransitionTime:"2023-07-31T21:48:19Z"message:successfully applied patches.reason:Programmedstatus:"True"type:Programmed
Offline
You can use egctl x translate to validate the translated xds output.
Caveats
This API will always be an unstable API and the same outcome cannot be garunteed
across versions for these reasons
The Envoy Proxy API might deprecate and remove API fields
Envoy Gateway might alter the xDS translation creating a different xDS output
such as changing the name field of resources.
1.5 - Observability
This section includes Observability tasks.
1.5.1 - Gateway API Metrics
Resource metrics for Gateway API objects are available using the Gateway API State Metrics project.
The project also provides example dashboard for visualising the metrics using Grafana, and example alerts using Prometheus & Alertmanager.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Run the following commands to install the metrics stack, with the Gateway API State Metrics configuration, on your kubernetes cluster:
To access the Prometheus UI, wait for the statefulset to be ready, then use the port-forward command:
# This first command may fail if the statefulset has not been created yet.# In that case, try again until you get a message like 'Waiting for 2 pods to be ready...'# or 'statefulset rolling update complete 2 pods...'kubectl -n monitoring rollout status --watch --timeout=5m statefulset/prometheus-k8s
kubectl -n monitoring port-forward service/prometheus-k8s 9090:9090 > /dev/null &
Navigate to http://localhost:9090.
Metrics can be queried from the ‘Graph’ tab e.g. gatewayapi_gateway_created
See the Gateway API State Metrics README for the full list of Gateway API metrics available.
Alerts can be seen in the ‘Alerts’ tab.
Gateway API specific alerts will be grouped under the ‘gateway-api.rules’ heading.
Note: Alerts are defined in a PrometheusRules custom resource in the ‘monitoring’ namespace. You can modify the alert rules by updating this resource.
Dashboards
To view the dashboards in Grafana, wait for the deployment to be ready, then use the port-forward command:
Navigate to http://localhost:3000 and sign in with admin/admin.
The Gateway API State dashboards will be available in the ‘Default’ folder and tagged with ‘gateway-api’.
See the Gateway API State Metrics README for further information on available dashboards.
Note: Dashboards are loaded from configmaps. You can modify the dashboards in the Grafana UI, however you will need to export them from the UI and update the json in the configmaps to persist changes.
1.5.2 - Proxy Observability
Envoy Gateway provides observability for the ControlPlane and the underlying EnvoyProxy instances.
This task show you how to config proxy observability, includes metrics, logs, and traces.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
FluentBit is used to collect logs from the EnvoyProxy instances and forward them to Loki. Install FluentBit:
LOKI_IP=$(kubectl get svc loki -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].ip}')TEMPO_IP=$(kubectl get svc tempo -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
Metrics
By default, Envoy Gateway expose metrics with prometheus endpoint.
Envoy Gateway provides observability for the RateLimit instances.
This guide show you how to config RateLimit observability, includes traces.
Prerequisites
Follow the steps from the Quickstart Guide to install Envoy Gateway and the HTTPRoute example manifest.
Before proceeding, you should be able to query the example backend using HTTP. Follow the steps from the Global Rate Limit to install RateLimit.
OpenTelemetry Collector offers a vendor-agnostic implementation of how to receive, process and export telemetry data.
By default, the Envoy Gateway does not configure RateLimit to send traces to the OpenTelemetry Sink.
You can configure the collector in the rateLimit.telemetry.tracing of the EnvoyGatewayCRD.
RateLimit uses the OpenTelemetry Exporter to export traces to the collector.
You can configure a collector that supports the OTLP protocol, which includes but is not limited to: OpenTelemetry Collector, Jaeger, Zipkin, and so on.
Note:
By default, the Envoy Gateway configures a 100% sampling rate for RateLimit, which may lead to performance issues.
Assuming the OpenTelemetry Collector is running in the observability namespace, and it has a service named otel-svc,
we only want to sample 50% of the trace data. We would configure it as follows:
Envoy Gateway provides support for exposing Envoy Proxy metrics to a Prometheus instance.
This task shows you how to visualise the metrics exposed to prometheus using grafana.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
GRAFANA_IP=$(kubectl get svc grafana -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
Connecting Grafana with Prometheus datasource
To visualise metrics from Prometheus, we have to connect Grafana with Prometheus. If you installed Grafana from the command
from prerequisites sections, the prometheus datasource should be already configured.
You can also add the data source manually by following the instructions from Grafana Docs.
Accessing Grafana
You can access the Grafana instance by visiting http://{GRAFANA_IP}, derived in prerequisites.
To log in to Grafana, use the credentials admin:admin.
Envoy Gateway has examples of dashboard for you to get started:
You can load the above dashboards in your Grafana to get started. Please refer to Grafana docs for importing dashboards.
1.6 - Operations
This section includes Operations tasks.
1.6.1 - Customize EnvoyProxy
Envoy Gateway provides an EnvoyProxy CRD that can be linked to the ParametersRef
in GatewayClass, allowing cluster admins to customize the managed EnvoyProxy Deployment and
Service. To learn more about GatewayClass and ParametersRef, please refer to Gateway API documentation.
Installation
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest.
Before proceeding, you should be able to query the example backend using HTTP.
Add GatewayClass ParametersRef
First, you need to add ParametersRef in GatewayClass, and refer to EnvoyProxy Config:
You can use egctl translate
to get the default xDS Bootstrap configuration used by Envoy Gateway.
After applying the config, the bootstrap config will be overridden by the new config you provided.
Any errors in the configuration will be surfaced as status within the GatewayClass resource.
You can also validate this configuration using egctl translate.
Customize EnvoyProxy Horizontal Pod Autoscaler
You can enable Horizontal Pod Autoscaler for EnvoyProxy Deployment. However, before enabling the HPA for EnvoyProxy, please ensure that the metrics-server component is installed in the cluster.
Once confirmed, you can apply it via EnvoyProxy Config as shown below:
After applying the config, the EnvoyProxy HPA (Horizontal Pod Autoscaler) is generated. However, upon activating the EnvoyProxy’s HPA, the Envoy Gateway will no longer reference the replicas field specified in the envoyDeployment, as outlined here.
Customize EnvoyProxy Command line options
You can customize the EnvoyProxy Command line options via spec.extraArgs in EnvoyProxy Config.
For example, the following configuration will add --disable-extensions arg in order to disable envoy.access_loggers/envoy.access_loggers.wasm extension:
After applying the configuration, you will see the change in both containers in the envoyproxy deployment.
1.6.2 - Deployment Mode
Deployment modes
One GatewayClass per Envoy Gateway Controller
An Envoy Gateway is associated with a single GatewayClass resource under one controller.
This is the simplest deployment mode and is suitable for scenarios where each Gateway needs to have its own dedicated set of resources and configurations.
Multiple GatewayClasses per Envoy Gateway Controller
An Envoy Gateway is associated with multiple GatewayClass resources under one controller.
Support for accepting multiple GatewayClasses was added here.
Separate Envoy Gateway Controllers
If you’ve instantiated multiple GatewayClasses, you can also run separate Envoy Gateway controllers in different namespaces, linking a GatewayClass to each of them for multi-tenancy.
Please follow the example Multi-tenancy.
Merged Gateways onto a single EnvoyProxy fleet
By default, each Gateway has its own dedicated set of Envoy Proxy and its configurations.
However, for some deployments, it may be more convenient to merge listeners across multiple Gateways and deploy a single Envoy Proxy fleet.
This can help to efficiently utilize the infra resources in the cluster and manage them in a centralized manner, or have a single IP address for all of the listeners.
Setting the mergeGateways field in the EnvoyProxy resource linked to GatewayClass will result in merging all Gateway listeners under one GatewayClass resource.
The tuple of port, protocol, and hostname must be unique across all Listeners.
The default deployment model is - Envoy Gateway watches for resources such a Service & HTTPRoute in all namespaces
and creates managed data plane resources such as EnvoyProxy Deployment in the namespace where Envoy Gateway is running.
Envoy Gateway also supports Namespaced deployment mode, you can watch resources in the specific namespaces by assigning
EnvoyGateway.provider.kubernetes.watch.namespaces or EnvoyGateway.provider.kubernetes.watch.namespaceSelector and creates managed data plane resources in the namespace where Envoy Gateway is running.
Support for alternate deployment modes is being tracked here.
Multi-tenancy
Kubernetes
A tenant is a group within an organization (e.g. a team or department) who shares organizational resources. We recommend
each tenant deploy their own Envoy Gateway controller in their respective namespace. Below is an example of deploying Envoy Gateway
by the marketing and product teams in separate namespaces.
Lets deploy Envoy Gateway in the marketing namespace and also watch resources only in this namespace. We are also setting the controller name to a unique string here gateway.envoyproxy.io/marketing-gatewayclass-controller.
Lets create a GatewayClass linked to the marketing team’s Envoy Gateway controller, and as well other resources linked to it, so the backend application operated by this team can be exposed to external clients.
Lets create a GatewayClass linked to the product team’s Envoy Gateway controller, and as well other resources linked to it, so the backend application operated by this team can be exposed to external clients.
With the below command you can ensure that you are no able to access the marketing team’s backend exposed using the www.marketing.example.com hostname
and the product team’s data plane.
kubectl get gateways -n default
NAMESPACE NAME CLASS ADDRESS PROGRAMMED AGE
default merged-eg-1 merged-eg 172.18.255.202 True 2m4s
default merged-eg-2 merged-eg 172.18.255.202 True 2m4s
default merged-eg-3 merged-eg 172.18.255.202 True 2m4s
Verify that HTTPRoutes are deployed
kubectl get httproute -n default
NAMESPACE NAME HOSTNAMES AGE
default hostname1-route ["www.merged1.com"] 2m4s
default hostname2-route ["www.merged2.com"] 2m4s
default hostname3-route ["www.merged3.com"] 2m4s
If you take a look at the deployed Envoy Proxy service you would notice that all of the Gateway listeners ports are added to that service.
kubectl get service -n envoy-gateway-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy-gateway ClusterIP 10.96.141.4 <none> 18000/TCP,18001/TCP 6m43s
envoy-gateway-metrics-service ClusterIP 10.96.113.191 <none> 19001/TCP 6m43s
envoy-merged-eg-668ac7ae LoadBalancer 10.96.48.255 172.18.255.202 8081:30467/TCP,8082:31793/TCP,8080:31153/TCP 3m17s
There should be also one deployment (envoy-merged-eg-668ac7ae-775f9865d-55zhs) for every Gateway and its name should reference the name of the GatewayClass.
kubectl get pods -n envoy-gateway-system
NAME READY STATUS RESTARTS AGE
envoy-gateway-5d998778f6-wr6m9 1/1 Running 0 6m43s
envoy-merged-eg-668ac7ae-775f9865d-55zhs 2/2 Running 0 3m17s
Testing the Configuration
Get the name of the merged gateways Envoy service:
exportENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gatewayclass=merged-eg -o jsonpath='{.items[0].metadata.name}')
Fetch external IP of the service:
exportGATEWAY_HOST=$(kubectl get svc/${ENVOY_SERVICE} -n envoy-gateway-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
In certain environments, the load balancer may be exposed using a hostname, instead of an IP address. If so, replace
ip in the above command with hostname.
Curl the route hostname-route2 through Envoy proxy:
You can also use the --type/-t flag to retrieve specific output types. In the below example, we will translate the Kubernetes resources (including the Gateway API resources) into xDS route resources.
You can see the output contains a EnvoyProxy resource that
can be used as a starting point to modify the xDS bootstrap resource for the managed Envoy Proxy fleet.
You can add an additional target gateway-api to show the processed Gateway API resources. For example, translating the above resources with the new argument shows that the HTTPRoute is rejected because there is no ready listener for it:
gatewayClass:metadata:creationTimestamp:nullname:egnamespace:envoy-gateway-systemspec:controllerName:gateway.envoyproxy.io/gatewayclass-controllerstatus:conditions:- lastTransitionTime:"2023-04-19T20:54:52Z"message:Valid GatewayClassreason:Acceptedstatus:"True"type:Acceptedgateways:- metadata:creationTimestamp:nullname:egnamespace:envoy-gateway-systemspec:gatewayClassName:eglisteners:- name:tlsport:8443protocol:TLSstatus:listeners:- attachedRoutes:0conditions:- lastTransitionTime:"2023-04-19T20:54:52Z"message:Listener must have TLS set when protocol is TLS.reason:Invalidstatus:"False"type:Programmedname:tlssupportedKinds:- group:gateway.networking.k8s.iokind:TLSRoutehttpRoutes:- metadata:creationTimestamp:nullname:backendnamespace:envoy-gateway-systemspec:parentRefs:- name:egrules:- backendRefs:- group:""kind:Servicename:backendport:3000weight:1matches:- path:type:PathPrefixvalue:/status:parents:- conditions:- lastTransitionTime:"2023-04-19T20:54:52Z"message:There are no ready listeners for this parent refreason:NoReadyListenersstatus:"False"type:Accepted- lastTransitionTime:"2023-04-19T20:54:52Z"message:Service envoy-gateway-system/backend not foundreason:BackendNotFoundstatus:"False"type:ResolvedRefscontrollerName:gateway.envoyproxy.io/gatewayclass-controllerparentRef:name:egxds:envoy-gateway-system-eg:'@type':type.googleapis.com/envoy.admin.v3.RoutesConfigDump
egctl experimental status
This subcommand allows users to show the summary of the status of specific or all resource types, in order to quickly find
out the status of any resources.
By default, egctl x status display all the conditions for one resource type. You can either add --quiet to only
display the latest condition, or add --verbose to display more details about current status.
Note
Currently, this subcommand only supports: GatewayClass, Gateway, HTTPRoute, GRPCRoute,
TLSRoute, TCPRoute, UDPRoute, BackendTLSPolicy,
BackendTrafficPolicy, ClientTrafficPolicy, EnvoyPatchPolicy, SecurityPolicy resource types and all.
Some examples of this command after installing Multi-tenancy example manifest:
Show the summary of GatewayClass.
~ egctl x status gatewayclass
NAME TYPE STATUS REASON
eg-marketing Accepted True Accepted
eg-product Accepted True Accepted
Show the summary of all resource types under all namespaces, the resource type with empty status will be ignored.
~ egctl x status all -A
NAME TYPE STATUS REASON
gatewayclass/eg-marketing Accepted True Accepted
gatewayclass/eg-product Accepted True Accepted
NAMESPACE NAME TYPE STATUS REASON
marketing gateway/eg Programmed True Programmed
Accepted True Accepted
product gateway/eg Programmed True Programmed
Accepted True Accepted
NAMESPACE NAME TYPE STATUS REASON
marketing httproute/backend ResolvedRefs True ResolvedRefs
Accepted True Accepted
product httproute/backend ResolvedRefs True ResolvedRefs
Accepted True Accepted
Show the summary of all the Gateways with details under all namespaces.
~ egctl x status gateway --verbose --all-namespaces
NAMESPACE NAME TYPE STATUS REASON MESSAGE OBSERVED GENERATION LAST TRANSITION TIME
marketing eg Programmed True Programmed Address assigned to the Gateway, 1/1 envoy Deployment replicas available 1 2024-02-02 18:17:14 +0800 CST
Accepted True Accepted The Gateway has been scheduled by Envoy Gateway 1 2024-02-01 17:50:39 +0800 CST
product eg Programmed True Programmed Address assigned to the Gateway, 1/1 envoy Deployment replicas available 1 2024-02-02 18:17:14 +0800 CST
Accepted True Accepted The Gateway has been scheduled by Envoy Gateway 1 2024-02-01 17:52:42 +0800 CST
Show the summary of latest Gateways condition under product namespace.
~ egctl x status gateway --quiet -n product
NAME TYPE STATUS REASON
eg Programmed True Programmed
Show the summary of latest HTTPRoutes condition under all namespaces.
~ egctl x status httproute --quiet --all-namespaces
NAMESPACE NAME TYPE STATUS REASON
marketing backend ResolvedRefs True ResolvedRefs
product backend ResolvedRefs True ResolvedRefs
egctl experimental dashboard
This subcommand streamlines the process for users to access the Envoy admin dashboard. By executing the following command:
egctl x dashboard envoy-proxy -n envoy-gateway-system envoy-engw-eg-a9c23fbb-558f94486c-82wh4
You will see the following output:
egctl x dashboard envoy-proxy -n envoy-gateway-system envoy-engw-eg-a9c23fbb-558f94486c-82wh4
http://localhost:19000
the Envoy admin dashboard will automatically open in your default web browser. This eliminates the need to manually locate and expose the admin port.
egctl experimental install
This subcommand can be used to install envoy-gateway.
egctl x install
By default, this will install both the envoy-gateway workload resource and the required gateway-api and envoy-gatewayCRDs.
We can specify to install only workload resources via --skip-crds
egctl x install --skip-crds
We can specify to install only CRDs resources via --only-crds
egctl x install --only-crds
We can specify --name and --namespace to install envoy-gateway in different places to support multi-tenant mode.
Note: If CRDs are already installed, then we need to specify --skip-crds to avoid repeated installation of CRDs resources.
egctl x install --name shop-backend --namespace shop
egctl experimental uninstall
This subcommand can be used to uninstall envoy-gateway.
egctl x uninstall
By default, this will only uninstall the envoy-gateway workload resource, if we want to also uninstall CRDs, we need to specify --with-crds
egctl x uninstall --with-crds
2 - Installation
This section includes installation related contents of Envoy Gateway.
2.1 - Install with Helm
Helm is a package manager for Kubernetes that automates the release and management of software on Kubernetes.
Envoy Gateway can be installed via a Helm chart with a few simple steps, depending on if you are deploying for the first time, upgrading Envoy Gateway from an existing installation, or migrating from Envoy Gateway.
Note: quickstart.yaml defines that Envoy Gateway will listen for
traffic on port 80 on its globally-routable IP address, to make it easy to use
browsers to test Envoy Gateway. When Envoy Gateway sees that its Listener is
using a privileged port (<1024), it will map this internally to an
unprivileged port, so that Envoy Gateway doesn’t need additional privileges.
It’s important to be aware of this mapping, since you may need to take it into
consideration when debugging.
Helm chart customizations
Some of the quick ways of using the helm install command for envoy gateway installation are below.
Note: Above are some of the ways we can directly use for customization of our installation. But if you are looking for more complex changes values.yaml comes to rescue.
Here we have made three changes to our values.yaml file. Increase the resources limit for cpu to 700m, changed the port for grpc to 18005 and for ratelimit to 18006 and also updated the logging level to debug.
You can use the below command to install the envoy gateway using values.yaml file.
If you want to know all the available fields inside the values.yaml file, please see the Helm Chart Values.
Open Ports
These are the ports used by Envoy Gateway and the managed Envoy Proxy.
Envoy Gateway
Envoy Gateway
Address
Port
Configurable
Xds EnvoyProxy Server
0.0.0.0
18000
No
Xds RateLimit Server
0.0.0.0
18001
No
Admin Server
127.0.0.1
19000
Yes
Metrics Server
0.0.0.0
19001
No
Health Check
127.0.0.1
8081
No
EnvoyProxy
Envoy Proxy
Address
Port
Admin Server
127.0.0.1
19000
Heath Check
0.0.0.0
19001
Next Steps
Envoy Gateway should now be successfully installed and running. To experience more abilities of Envoy Gateway, refer to Tasks.
2.2 - Install with Kubernetes YAML
This task walks you through installing Envoy Gateway in your Kubernetes cluster.
The manual install process does not allow for as much control over configuration
as the Helm install method, so if you need more control over your Envoy Gateway
installation, it is recommended that you use helm.
Before you begin
Envoy Gateway is designed to run in Kubernetes for production. The most essential requirements are:
Envoy Gateway should now be successfully installed and running, but in order to experience more abilities of Envoy Gateway, you can refer to Tasks.
2.3 - Install egctl
What is egctl?
egctl is a command line tool to provide additional functionality for Envoy Gateway users.
This task shows how to install the egctl CLI. egctl can be installed either from source, or from pre-built binary releases.
From The Envoy Gateway Project
The Envoy Gateway project provides two ways to fetch and install egctl. These are the official methods to get egctl releases. Installation through those methods can be found below the official methods.
From the Binary Releases
Every release of egctl provides binary releases for a variety of OSes. These binary versions can be manually downloaded and installed.
Unpack it (tar -zxvf egctl_latest_linux_amd64.tar.gz)
Find the egctl binary in the unpacked directory, and move it to its desired destination (mv bin/linux/amd64/egctl /usr/local/bin/egctl)
From there, you should be able to run: egctl help.
From Script
egctl now has an installer script that will automatically grab the latest release version of egctl and install it locally.
You can fetch that script, and then execute it locally. It’s well documented so that you can read through it and understand what it is doing before you run it.
curl -fsSL -o get-egctl.sh https://gateway.envoyproxy.io/get-egctl.sh
chmod +x get-egctl.sh
# get help info of the bash get-egctl.sh --help
# install the latest development version of egctlbash VERSION=latest get-egctl.sh
Yes, you can just use the below command if you want to live on the edge.
You can refer to the Use egctl task for more details about egctl.
2.4 - Control Plane Authentication using custom certs
Envoy Gateway establishes a secure TLS connection for control plane communication between Envoy Gateway pods and the Envoy Proxy fleet. The TLS Certificates used here are self signed and generated using a job that runs before envoy gateway is created, and these certs and mounted on to the envoy gateway and envoy proxy pods.
This task will walk you through configuring custom certs for control plane auth.
Before you begin
We use Cert-Manager to manage the certificates. You can install it by following the official guide.
Configure custom certs for control plane
First you need to set up the CA issuer, in this task, we use the selfsigned-issuer as an example.
You should not use the self-signed issuer in production, you should use a real CA issuer.
Now you can follow the helm chart installation guide to install envoy gateway with custom certs.
2.5 - Compatibility Matrix
This section includes Compatibility Matrix of Envoy Gateway.
Envoy Gateway relies on the Envoy Proxy and the Gateway API, and runs within a Kubernetes cluster. Not all versions of each of these products can function together for Envoy Gateway. Supported version combinations are listed below; bold type indicates the versions of the Envoy Proxy and the Gateway API actually compiled into each Envoy Gateway release.
HTTPProtocolVersion1_0 specifies that HTTP/1.0 should be negotiable with ALPN
http/1.1
HTTPProtocolVersion1_1 specifies that HTTP/1.1 should be negotiable with ALPN
h2
HTTPProtocolVersion2 specifies that HTTP/2 should be negotiable with ALPN
ALSEnvoyProxyAccessLog
ALSEnvoyProxyAccessLog defines the gRPC Access Log Service (ALS) sink.
The service must implement the Envoy gRPC Access Log Service streaming API:
https://www.envoyproxy.io/docs/envoy/latest/api-v3/service/accesslog/v3/als.proto
Access log format information is passed in the form of gRPC metadata when the
stream is established. Specifically, the following metadata is passed:
x-accesslog-text - The access log format string when a Text format is used.
x-accesslog-attr - JSON encoded key/value pairs when a JSON format is used.
BackendRefs references a Kubernetes object that represents the gRPC service to which the access logs will be sent. Currently only Service is supported.
logName
string
false
LogName defines the friendly name of the access log to be returned in StreamAccessLogsMessage.Identifier. This allows the access log server to differentiate between different access logs coming from the same Envoy.
MaxInterval is the maximum interval between retries. This parameter is optional, but must be greater than or equal to the base_interval if set. The default is 10 times the base_interval
BackendRef
BackendRef defines how an ObjectReference that is specific to BackendRef.
Kind is the Kubernetes resource kind of the referent. For example “Service”.
Defaults to “Service” when not specified.
ExternalName services can refer to CNAME DNS records that may live outside of the cluster and as such are difficult to reason about in terms of conformance. They also may not be safe to forward to (see CVE-2021-25740 for more information). Implementations SHOULD NOT support ExternalName Services.
Support: Core (Services with a type other than ExternalName)
Support: Implementation-specific (Services with type ExternalName)
Namespace is the namespace of the backend. When unspecified, the local namespace is inferred.
Note that when a namespace different than the local namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace’s owner to accept the reference. See the ReferenceGrant documentation for details.
Port specifies the destination port number to use for this resource. Port is required when the referent is a Kubernetes Service. In this case, the port number is the service port number, not the target port. For other resources, destination port might be derived from the referent resource or this field.
BackendTLSConfig
BackendTLSConfig describes the BackendTLS configuration for Envoy Proxy.
ClientCertificateRef defines the reference to a Kubernetes Secret that contains the client certificate and private key for Envoy to use when connecting to backend services and external services, such as ExtAuth, ALS, OpenTelemetry, etc.
Max specifies the maximal TLS protocol version to allow The default is TLS 1.3 if this is not specified.
ciphers
string array
false
Ciphers specifies the set of cipher suites supported when negotiating TLS 1.0 - 1.2. This setting has no effect for TLS 1.3. In non-FIPS Envoy Proxy builds the default cipher list is: - [ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-CHACHA20-POLY1305] - [ECDHE-RSA-AES128-GCM-SHA256|ECDHE-RSA-CHACHA20-POLY1305] - ECDHE-ECDSA-AES256-GCM-SHA384 - ECDHE-RSA-AES256-GCM-SHA384 In builds using BoringSSL FIPS the default cipher list is: - ECDHE-ECDSA-AES128-GCM-SHA256 - ECDHE-RSA-AES128-GCM-SHA256 - ECDHE-ECDSA-AES256-GCM-SHA384 - ECDHE-RSA-AES256-GCM-SHA384
ecdhCurves
string array
false
ECDHCurves specifies the set of supported ECDH curves. In non-FIPS Envoy Proxy builds the default curves are: - X25519 - P-256 In builds using BoringSSL FIPS the default curve is: - P-256
signatureAlgorithms
string array
false
SignatureAlgorithms specifies which signature algorithms the listener should support.
ALPNProtocols supplies the list of ALPN protocols that should be exposed by the listener. By default h2 and http/1.1 are enabled. Supported values are: - http/1.0 - http/1.1 - h2
BackendTrafficPolicy
BackendTrafficPolicy allows the user to configure the behavior of the connection
between the Envoy Proxy listener and the backend service.
targetRef is the name of the resource this policy is being attached to. This Policy and the TargetRef MUST be in the same namespace for this Policy to have effect and be applied to the Gateway.
FaultInjection defines the fault injection policy to be applied. This configuration can be used to inject delays and abort requests to mimic failure scenarios such as service failures and overloads
Retry provides more advanced usage, allowing users to customize the number of retries, retry fallback strategy, and retry triggering conditions. If not set, retry will be disabled.
useClientProtocol
boolean
false
UseClientProtocol configures Envoy to prefer sending requests to backends using the same HTTP protocol that the incoming request used. Defaults to false, which means that Envoy will use the protocol indicated by the attached BackendRef.
The Kubernetes secret which contains the username-password pairs in htpasswd format, used to verify user credentials in the “Authorization” header.
This is an Opaque secret. The username-password pairs should be stored in the key “.htpasswd”. As the key name indicates, the value needs to be the htpasswd format, for example: “user1:{SHA}hashed_user1_password”. Right now, only SHA hash algorithm is supported. Reference to https://httpd.apache.org/docs/2.4/programs/htpasswd.html for more details.
Note: The secret must be in the same namespace as the SecurityPolicy.
BootstrapType
Underlying type:string
BootstrapType defines the types of bootstrap supported by Envoy Gateway.
Merge merges the provided bootstrap with the default one. The provided bootstrap can add or override a value within a map, or add a new value to a list. Please note that the provided bootstrap can’t override a value within a list.
Replace
Replace replaces the default bootstrap with the provided one.
CORS
CORS defines the configuration for Cross-Origin Resource Sharing (CORS).
The maximum number of connections that Envoy will establish to the referenced backend defined within a xRoute rule.
maxPendingRequests
integer
false
The maximum number of pending requests that Envoy will queue to the referenced backend defined within a xRoute rule.
maxParallelRequests
integer
false
The maximum number of parallel requests that Envoy will make to the referenced backend defined within a xRoute rule.
maxParallelRetries
integer
false
The maximum number of parallel retries that Envoy will make to the referenced backend defined within a xRoute rule.
maxRequestsPerConnection
integer
false
The maximum number of requests that Envoy will make over a single connection to the referenced backend defined within a xRoute rule. Default: unlimited.
ClaimToHeader
ClaimToHeader defines a configuration to convert JWT claims into HTTP headers
Header defines the name of the HTTP request header that the JWT Claim will be saved into.
claim
string
true
Claim is the JWT Claim that should be saved into the header : it can be a nested claim of type (eg. “claim.nested.key”, “sub”). The nested claim name must use dot “." to separate the JSON name path.
ClientIPDetectionSettings
ClientIPDetectionSettings provides configuration for determining the original client IP address for requests.
Max specifies the maximal TLS protocol version to allow The default is TLS 1.3 if this is not specified.
ciphers
string array
false
Ciphers specifies the set of cipher suites supported when negotiating TLS 1.0 - 1.2. This setting has no effect for TLS 1.3. In non-FIPS Envoy Proxy builds the default cipher list is: - [ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-CHACHA20-POLY1305] - [ECDHE-RSA-AES128-GCM-SHA256|ECDHE-RSA-CHACHA20-POLY1305] - ECDHE-ECDSA-AES256-GCM-SHA384 - ECDHE-RSA-AES256-GCM-SHA384 In builds using BoringSSL FIPS the default cipher list is: - ECDHE-ECDSA-AES128-GCM-SHA256 - ECDHE-RSA-AES128-GCM-SHA256 - ECDHE-ECDSA-AES256-GCM-SHA384 - ECDHE-RSA-AES256-GCM-SHA384
ecdhCurves
string array
false
ECDHCurves specifies the set of supported ECDH curves. In non-FIPS Envoy Proxy builds the default curves are: - X25519 - P-256 In builds using BoringSSL FIPS the default curve is: - P-256
signatureAlgorithms
string array
false
SignatureAlgorithms specifies which signature algorithms the listener should support.
ALPNProtocols supplies the list of ALPN protocols that should be exposed by the listener. By default h2 and http/1.1 are enabled. Supported values are: - http/1.0 - http/1.1 - h2
TargetRef is the name of the Gateway resource this policy is being attached to. This Policy and the TargetRef MUST be in the same namespace for this Policy to have effect and be applied to the Gateway. TargetRef
TcpKeepalive settings associated with the downstream client connection. If defined, sets SO_KEEPALIVE on the listener socket to enable TCP Keepalives. Disabled by default.
enableProxyProtocol
boolean
false
EnableProxyProtocol interprets the ProxyProtocol header and adds the Client Address into the X-Forwarded-For header. Note Proxy Protocol must be present when this field is set, else the connection is closed.
HTTP3 provides HTTP/3 configuration on the listener.
ClientValidationContext
ClientValidationContext holds configuration that can be used to validate the client initiating the TLS connection
to the Gateway.
By default, no client specific configuration is validated.
Optional set to true accepts connections even when a client doesn’t present a certificate. Defaults to false, which rejects connections without a valid client certificate.
CACertificateRefs contains one or more references to Kubernetes objects that contain TLS certificates of the Certificate Authorities that can be used as a trust anchor to validate the certificates presented by the client.
A single reference to a Kubernetes ConfigMap or a Kubernetes Secret, with the CA certificate in a key named ca.crt is currently supported.
References to a resource in different namespace are invalid UNLESS there is a ReferenceGrant in the target namespace that allows the certificate to be attached.
Compression
Compression defines the config of enabling compression.
This can help reduce the bandwidth at the expense of higher CPU.
BufferLimit provides configuration for the maximum buffer size in bytes for each incoming connection. For example, 20Mi, 1Gi, 256Ki etc. Note that when the suffix is not provided, the value is interpreted as bytes. Default: 32768 bytes.
Value of the maximum concurrent connections limit. When the limit is reached, incoming connections will be closed after the CloseDelay duration. Default: unlimited.
Name of the header containing the original downstream remote address, if present.
failClosed
boolean
false
FailClosed is a switch used to control the flow of traffic when client IP detection fails. If set to true, the listener will respond with 403 Forbidden when the client IP address cannot be determined.
TargetRef is the name of the resource this policy is being attached to. This Policy and the TargetRef MUST be in the same namespace for this Policy to have effect and be applied to the Gateway or xRoute.
Wasm is a list of Wasm extensions to be loaded by the Gateway. Order matters, as the extensions will be loaded in the order they are defined in this list.
Provider defines the desired provider and provider-specific configuration. If unspecified, the Kubernetes provider is used with default configuration parameters.
RateLimit defines the configuration associated with the Rate Limit service deployed by Envoy Gateway required to implement the Global Rate limiting functionality. The specific rate limit service used here is the reference implementation in Envoy. For more details visit https://github.com/envoyproxy/ratelimit. This configuration is unneeded for “Local” rate limiting.
Resource defines the desired resource provider. This provider is used to specify the provider to be used to retrieve the resource configurations such as Gateway API resources
Infrastructure defines the desired infrastructure provider. This provider is used to specify the provider to be used to provide an environment to deploy the out resources like the Envoy Proxy data plane.
EnvoyGatewayFileResourceProvider
EnvoyGatewayFileResourceProvider defines configuration for the File Resource provider.
RateLimitDeployment defines the desired state of the Envoy ratelimit deployment resource. If unspecified, default settings for the managed Envoy ratelimit deployment resource are applied.
LeaderElection specifies the configuration for leader election. If it’s not set up, leader election will be active by default, using Kubernetes’ standard settings.
EnvoyGatewayLogComponent
Underlying type:string
EnvoyGatewayLogComponent defines a component that supports a configured logging level.
Level is the logging level. If unspecified, defaults to “info”. EnvoyGatewayLogComponent options: default/provider/gateway-api/xds-translator/xds-server/infrastructure/global-ratelimit. LogLevel options: debug/info/error/warn.
EnvoyGatewayMetricSink
EnvoyGatewayMetricSink defines control plane
metric sinks where metrics are sent to.
Custom defines the configuration for the Custom provider. This provider allows you to define a specific resource provider and a infrastructure provider.
EnvoyGatewayResourceProvider
EnvoyGatewayResourceProvider defines configuration for the Custom Resource provider.
Provider defines the desired provider and provider-specific configuration. If unspecified, the Kubernetes provider is used with default configuration parameters.
RateLimit defines the configuration associated with the Rate Limit service deployed by Envoy Gateway required to implement the Global Rate limiting functionality. The specific rate limit service used here is the reference implementation in Envoy. For more details visit https://github.com/envoyproxy/ratelimit. This configuration is unneeded for “Local” rate limiting.
ExtensionAPIs defines the settings related to specific Gateway API Extensions implemented by Envoy Gateway
EnvoyGatewayTelemetry
EnvoyGatewayTelemetry defines telemetry configurations for envoy gateway control plane.
Control plane will focus on metrics observability telemetry and tracing telemetry later.
TargetRef is the name of the Gateway API resource this policy is being attached to. By default attaching to Gateway is supported and when mergeGateways is enabled it should attach to GatewayClass. This Policy and the TargetRef MUST be in the same namespace for this Policy to have effect and be applied to the Gateway TargetRef
priority
integer
true
Priority of the EnvoyPatchPolicy. If multiple EnvoyPatchPolicies are applied to the same TargetRef, they will be applied in the ascending order of the priority i.e. int32.min has the highest priority and int32.max has the lowest priority. Defaults to 0.
EnvoyPatchType
Underlying type:string
EnvoyPatchType specifies the types of Envoy patching mechanisms.
JSONPatchEnvoyPatchType allows the user to patch the generated xDS resources using JSONPatch semantics. For more details on the semantics, please refer to https://datatracker.ietf.org/doc/html/rfc6902
EnvoyProxy
EnvoyProxy is the schema for the envoyproxies API.
EnvoyDeployment defines the desired state of the Envoy deployment resource. If unspecified, default settings for the managed Envoy deployment resource are applied.
EnvoyDaemonSet defines the desired state of the Envoy daemonset resource. Disabled by default, a deployment resource is used instead to provision the Envoy Proxy fleet
EnvoyService defines the desired state of the Envoy service resource. If unspecified, default settings for the managed Envoy service resource are applied.
EnvoyHpa defines the Horizontal Pod Autoscaler settings for Envoy Proxy Deployment. Once the HPA is being set, Replicas field from EnvoyDeployment will be ignored.
EnvoyProxyProvider
EnvoyProxyProvider defines the desired state of a resource provider.
Type is the type of resource provider to use. A resource provider provides infrastructure resources for running the data plane, e.g. Envoy proxy, and optional auxiliary control planes. Supported types are “Kubernetes”.
Kubernetes defines the desired state of the Kubernetes resource provider. Kubernetes provides infrastructure resources for running the data plane, e.g. Envoy proxy. If unspecified and type is “Kubernetes”, default settings for managed Kubernetes resources are applied.
EnvoyProxySpec
EnvoyProxySpec defines the desired state of EnvoyProxy.
Provider defines the desired resource provider and provider-specific configuration. If unspecified, the “Kubernetes” resource provider is used with default configuration parameters.
Bootstrap defines the Envoy Bootstrap as a YAML string. Visit https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-msg-config-bootstrap-v3-bootstrap to learn more about the syntax. If set, this is the Bootstrap configuration used for the managed Envoy Proxy fleet instead of the default Bootstrap configuration set by Envoy Gateway. Some fields within the Bootstrap that are required to communicate with the xDS Server (Envoy Gateway) and receive xDS resources from it are not configurable and will result in the EnvoyProxy resource being rejected. Backward compatibility across minor versions is not guaranteed. We strongly recommend using egctl x translate to generate a EnvoyProxy resource with the Bootstrap field set to the default Bootstrap configuration used. You can edit this configuration, and rerun egctl x translate to ensure there are no validation errors.
concurrency
integer
false
Concurrency defines the number of worker threads to run. If unset, it defaults to the number of cpuset threads on the platform.
MergeGateways defines if Gateway resources should be merged onto the same Envoy Proxy Infrastructure. Setting this field to true would merge all Gateway Listeners under the parent Gateway Class. This means that the port, protocol and hostname tuple must be unique for every listener. If a duplicate listener is detected, the newer listener (based on timestamp) will be rejected and its status will be updated with a “Accepted=False” condition.
BackendTLS is the TLS configuration for the Envoy proxy to use when connecting to backends. These settings are applied on backends for which TLS policies are specified.
EnvoyResourceType
Underlying type:string
EnvoyResourceType specifies the type URL of the Envoy resource.
HTTP defines the HTTP External Authorization service. Either GRPCService or HTTPService must be specified, and only one of them can be provided.
headersToExtAuth
string array
false
HeadersToExtAuth defines the client request headers that will be included in the request to the external authorization service. Note: If not specified, the default behavior for gRPC and HTTP external authorization services is different due to backward compatibility reasons. All headers will be included in the check request to a gRPC authorization server. Only the following headers will be included in the check request to an HTTP authorization server: Host, Method, Path, Content-Length, and Authorization. And these headers will always be included to the check request to an HTTP authorization server by default, no matter whether they are specified in HeadersToExtAuth or not.
failOpen
boolean
false
FailOpen is a switch used to control the behavior when a response from the External Authorization service cannot be obtained. If FailOpen is set to true, the system allows the traffic to pass through. Otherwise, if it is set to false or not set (defaulting to false), the system blocks the traffic and returns a HTTP 5xx error, reflecting a fail-closed approach. This setting determines whether to prioritize accessibility over strict security in case of authorization service failure.
ExtProc
ExtProc defines the configuration for External Processing filter.
MessageTimeout is the timeout for a response to be returned from the external processor Default: 200ms
failOpen
boolean
false
FailOpen defines if requests or responses that cannot be processed due to connectivity to the external processor are terminated or passed-through. Default: false
Kind is the Kubernetes resource kind of the referent. For example “Service”.
Defaults to “Service” when not specified.
ExternalName services can refer to CNAME DNS records that may live outside of the cluster and as such are difficult to reason about in terms of conformance. They also may not be safe to forward to (see CVE-2021-25740 for more information). Implementations SHOULD NOT support ExternalName Services.
Support: Core (Services with a type other than ExternalName)
Support: Implementation-specific (Services with type ExternalName)
Namespace is the namespace of the backend. When unspecified, the local namespace is inferred.
Note that when a namespace different than the local namespace is specified, a ReferenceGrant object is required in the referent namespace to allow that namespace’s owner to accept the reference. See the ReferenceGrant documentation for details.
Port specifies the destination port number to use for this resource. Port is required when the referent is a Kubernetes Service. In this case, the port number is the service port number, not the target port. For other resources, destination port might be derived from the referent resource or this field.
StreamedExtProcBodyProcessingMode will stream the body to the server in pieces as they arrive at the proxy.
Buffered
BufferedExtProcBodyProcessingMode will buffer the message body in memory and send the entire body at once. If the body exceeds the configured buffer limit, then the downstream system will receive an error.
BufferedPartial
BufferedPartialExtBodyHeaderProcessingMode will buffer the message body in memory and send the entire body in one chunk. If the body exceeds the configured buffer limit, then the body contents up to the buffer limit will be sent.
CertificateRef contains a references to objects (Kubernetes objects or otherwise) that contains a TLS certificate and private keys. These certificates are used to establish a TLS handshake to the extension server.
CertificateRef can only reference a Kubernetes Secret at this time.
FaultInjection
FaultInjection defines the fault injection policy to be applied. This configuration can be used to
inject delays and abort requests to mimic failure scenarios such as service failures and overloads
BackendRef references a Kubernetes object that represents the backend server to which the authorization request will be sent. Only service Kind is supported for now.
Gateway
Gateway defines the desired Gateway API configuration of Envoy Gateway.
Rules are a list of RateLimit selectors and limits. Each rule and its associated limit is applied in a mutually exclusive way. If a request matches multiple rules, each of their associated limits get applied, so a single request might increase the rate limit counters for multiple rules if selected. The rate limit service will return a logical OR of the individual rate limit decisions of all matching rules. For example, if a request matches two rules, one rate limited and one not, the final decision will be to rate limit the request.
GroupVersionKind
GroupVersionKind unambiguously identifies a Kind.
It can be converted to k8s.io/apimachinery/pkg/runtime/schema.GroupVersionKind
UseDefaultHost defines if the HTTP/1.0 request is missing the Host header, then the hostname associated with the listener should be injected into the request. If this is not set and an HTTP/1.0 request arrives without a host, then it will be rejected.
HTTP1Settings
HTTP1Settings provides HTTP/1 configuration on the listener.
RequestReceivedTimeout is the duration envoy waits for the complete request reception. This timer starts upon request initiation and stops when either the last byte of the request is sent upstream or when the response begins.
BackendRef references a Kubernetes object that represents the backend server to which the authorization request will be sent. Only service Kind is supported for now.
path
string
true
Path is the path of the HTTP External Authorization service. If path is specified, the authorization request will be sent to that path, or else the authorization request will be sent to the root path.
headersToBackend
string array
false
HeadersToBackend are the authorization response headers that will be added to the original client request before sending it to the backend server. Note that coexisting headers will be overridden. If not specified, no authorization response headers will be added to the original client request.
HeaderMatchType specifies the semantics of how HTTP header values should be compared.
Valid HeaderMatchType values are “Exact”, “RegularExpression”, and “Distinct”.
HeaderMatchExact matches the exact value of the Value field against the value of the specified HTTP Header.
RegularExpression
HeaderMatchRegularExpression matches a regular expression against the value of the specified HTTP Header. The regex string must adhere to the syntax documented in https://github.com/google/re2/wiki/Syntax.
Distinct
HeaderMatchDistinct matches any and all possible unique values encountered in the specified HTTP Header. Note that each unique value will receive its own rate limit bucket. Note: This is only supported for Global Rate Limits.
HeaderSettings
HeaderSettings provides configuration options for headers on the listener.
Optional determines whether a missing JWT is acceptable, defaulting to false if not specified. Note: Even if optional is set to true, JWT authentication will still fail if an invalid JWT is presented.
JWTExtractor defines a custom JWT token extraction from HTTP request.
If specified, Envoy will extract the JWT token from the listed extractors (headers, cookies, or params) and validate each of them.
If any value extracted is found to be an invalid JWT, a 401 error will be returned.
Name is the HTTP header name to retrieve the token
valuePrefix
string
false
ValuePrefix is the prefix that should be stripped before extracting the token. The format would be used by Envoy like “{ValuePrefix}”. For example, “Authorization: Bearer ”, then the ValuePrefix=“Bearer " with a space at the end.
JWTProvider
JWTProvider defines how a JSON Web Token (JWT) can be verified.
Name defines a unique name for the JWT provider. A name can have a variety of forms, including RFC1123 subdomains, RFC 1123 labels, or RFC 1035 labels.
ClaimToHeaders is a list of JWT claims that must be extracted into HTTP request headers For examples, following config: The claim must be of type; string, int, double, bool. Array type claims are not supported
recomputeRoute
boolean
false
RecomputeRoute clears the route cache and recalculates the routing decision. This field must be enabled if the headers generated from the claim are used for route matching decisions. If the recomputation selects a new route, features targeting the new matched route will be applied.
ExtractFrom defines different ways to extract the JWT token from HTTP request. If empty, it defaults to extract JWT token from the Authorization HTTP request header using Bearer schema or access_token from query parameters.
KubernetesContainerSpec
KubernetesContainerSpec defines the desired state of the Kubernetes container resource.
KubernetesHorizontalPodAutoscalerSpec defines Kubernetes Horizontal Pod Autoscaler settings of Envoy Proxy Deployment.
When HPA is enabled, it is recommended that the value in KubernetesDeploymentSpec.replicas be removed, otherwise
Envoy Gateway will revert back to this value every time reconciliation occurs.
See k8s.io.autoscaling.v2.HorizontalPodAutoScalerSpec.
metrics contains the specifications for which to use to calculate the desired replica count (the maximum replica count across all metrics will be used). If left empty, it defaults to being based on CPU utilization with average on 80% usage.
behavior configures the scaling behavior of the target in both Up and Down directions (scaleUp and scaleDown fields respectively). If not set, the default HPAScalingRules for scale up and scale down are used. See k8s.io.autoscaling.v2.HorizontalPodAutoScalerBehavior.
KubernetesPatchSpec
KubernetesPatchSpec defines how to perform the patch operation
SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field.
TopologySpreadConstraints describes how a group of pods ought to spread across topology domains. Scheduler will schedule pods in a way which abides by the constraints. All topologySpreadConstraints are ANDed.
KubernetesServiceSpec
KubernetesServiceSpec defines the desired state of the Kubernetes service resource.
Type determines how the Service is exposed. Defaults to LoadBalancer. Valid options are ClusterIP, LoadBalancer and NodePort. “LoadBalancer” means a service will be exposed via an external load balancer (if the cloud provider supports it). “ClusterIP” means a service will only be accessible inside the cluster, via the cluster IP. “NodePort” means a service will be exposed on a static Port on all Nodes of the cluster.
loadBalancerClass
string
false
LoadBalancerClass, when specified, allows for choosing the LoadBalancer provider implementation if more than one are available or is otherwise expected to be specified
allocateLoadBalancerNodePorts
boolean
false
AllocateLoadBalancerNodePorts defines if NodePorts will be automatically allocated for services with type LoadBalancer. Default is “true”. It may be set to “false” if the cluster load-balancer does not rely on NodePorts. If the caller requests specific NodePorts (by specifying a value), those requests will be respected, regardless of this field. This field may only be set for services with type LoadBalancer and will be cleared if the type is changed to any other type.
loadBalancerSourceRanges
string array
false
LoadBalancerSourceRanges defines a list of allowed IP addresses which will be configured as firewall rules on the platform providers load balancer. This is not guaranteed to be working as it happens outside of kubernetes and has to be supported and handled by the platform provider. This field may only be set for services with type LoadBalancer and will be cleared if the type is changed to any other type.
loadBalancerIP
string
false
LoadBalancerIP defines the IP Address of the underlying load balancer service. This field may be ignored if the load balancer provider does not support this feature. This field has been deprecated in Kubernetes, but it is still used for setting the IP Address in some cloud providers such as GCP.
ExternalTrafficPolicy determines the externalTrafficPolicy for the Envoy Service. Valid options are Local and Cluster. Default is “Local”. “Local” means traffic will only go to pods on the node receiving the traffic. “Cluster” means connections are loadbalanced to all pods in the cluster.
Type indicates what watch mode to use. KubernetesWatchModeTypeNamespaces and KubernetesWatchModeTypeNamespaceSelector are currently supported By default, when this field is unset or empty, Envoy Gateway will watch for input namespaced resources from all namespaces.
namespaces
string array
true
Namespaces holds the list of namespaces that Envoy Gateway will watch for namespaced scoped resources such as Gateway, HTTPRoute and Service. Note that Envoy Gateway will continue to reconcile relevant cluster scoped resources such as GatewayClass that it is linked to. Precisely one of Namespaces and NamespaceSelector must be set.
NamespaceSelector holds the label selector used to dynamically select namespaces. Envoy Gateway will watch for namespaces matching the specified label selector. Precisely one of Namespaces and NamespaceSelector must be set.
KubernetesWatchModeType
Underlying type:string
KubernetesWatchModeType defines the type of KubernetesWatchMode
LeaseDuration defines the time non-leader contenders will wait before attempting to claim leadership. It’s based on the timestamp of the last acknowledged signal. The default setting is 15 seconds.
RenewDeadline represents the time frame within which the current leader will attempt to renew its leadership status before relinquishing its position. The default setting is 10 seconds.
SlowStart defines the configuration related to the slow start load balancer policy. If set, during slow start window, traffic sent to the newly added hosts will gradually increase. Currently this is only supported for RoundRobin and LeastRequest load balancers
LoadBalancerType
Underlying type:string
LoadBalancerType specifies the types of LoadBalancer.
Rules are a list of RateLimit selectors and limits. If a request matches multiple rules, the strictest limit is applied. For example, if a request matches two rules, one with 10rps and one with 20rps, the final limit will be based on the rule with 10rps.
LogLevel
Underlying type:string
LogLevel defines a log level for Envoy Gateway and EnvoyProxy system logs.
The redirect URL to be used in the OIDC Authentication Request. If not specified, uses the default redirect URI “%REQ(x-forwarded-proto)%://%REQ(:authority)%/oauth2/callback”
logoutPath
string
true
The path to log a user out, clearing their credential cookies. If not specified, uses a default logout path “/logout”
OIDCProvider
OIDCProvider defines the OIDC Provider configuration.
The OIDC Provider’s issuer identifier. Issuer MUST be a URI RFC 3986 [RFC3986] with a scheme component that MUST be https, a host component, and optionally, port and path components and no query or fragment components.
BackendRefs references a Kubernetes object that represents the backend server to which the accesslog will be sent. Only service Kind is supported for now.
resources
object (keys:string, values:string)
false
Resources is a set of labels that describe the source of a log entry, including envoy node info. It’s recommended to follow semantic conventions.
Origin
Underlying type:string
Origin is defined by the scheme (protocol), hostname (domain), and port of
the URL used to access it. The hostname can be “precise” which is just the
domain name or “wildcard” which is a domain name prefixed with a single
wildcard label such as “*.example.com”.
In addition to that a single wildcard (with or without scheme) can be
configured to match any origin.
Interval defines the time between passive health checks.
consecutiveLocalOriginFailures
integer
false
ConsecutiveLocalOriginFailures sets the number of consecutive local origin failures triggering ejection. Parameter takes effect only when split_external_local_origin_errors is set to true.
consecutiveGatewayErrors
integer
false
ConsecutiveGatewayErrors sets the number of consecutive gateway errors triggering ejection.
consecutive5XxErrors
integer
false
Consecutive5xxErrors sets the number of consecutive 5xx errors triggering ejection.
KeepUnchangedAction keeps escaped slashes as they arrive without changes
RejectRequest
RejectRequestAction rejects client requests containing escaped slashes with a 400 status. gRPC requests will be rejected with the INTERNAL (13) error code. The “httpN.downstream_rq_failed_path_normalization” counter is incremented for each rejected request.
UnescapeAndRedirect
UnescapeAndRedirect unescapes %2F and %5C sequences and redirects to the new path if these sequences were present. Redirect occurs after path normalization and merge slashes transformations if they were configured. gRPC requests will be rejected with the INTERNAL (13) error code. This option minimizes possibility of path confusion exploits by forcing request with unescaped slashes to traverse all parties: downstream client, intermediate proxies, Envoy and upstream server. The “httpN.downstream_rq_redirected_with_normalized_path” counter is incremented for each redirected request.
UnescapeAndForward
UnescapeAndForward unescapes %2F and %5C sequences and forwards the request. Note: this option should not be enabled if intermediaries perform path based access control as it may lead to path confusion vulnerabilities.
PathSettings
PathSettings provides settings that managing how the incoming path set by clients is handled.
EscapedSlashesAction determines how %2f, %2F, %5c, or %5C sequences in the path URI should be handled. The default is UnescapeAndRedirect.
disableMergeSlashes
boolean
false
DisableMergeSlashes allows disabling the default configuration of merging adjacent slashes in the path. Note that slash merging is not part of the HTTP spec and is provided for convenience.
Text defines the text accesslog format, following Envoy accesslog formatting, It’s required when the format type is “Text”. Envoy command operators may be used in the format. The format string documentation provides more information.
json
object (keys:string, values:string)
false
JSON is additional attributes that describe the specific event occurrence. Structured format for the envoy access logs. Envoy command operators can be used as values for fields within the Struct. It’s required when the format type is “JSON”.
ProxyAccessLogSinkTypeFile defines the file accesslog sink.
OpenTelemetry
ProxyAccessLogSinkTypeOpenTelemetry defines the OpenTelemetry accesslog sink. When the provider is Kubernetes, EnvoyGateway always sends k8s.namespace.name and k8s.pod.name as additional attributes.
Level is a map of logging level per component, where the component is the key and the log level is the value. If unspecified, defaults to “default: warn”.
ProxyMetricSink
ProxyMetricSink defines the sink of metrics.
Default metrics sink is OpenTelemetry.
Matches defines configuration for selecting specific metrics instead of generating all metrics stats that are enabled by default. This helps reduce CPU and memory overhead in Envoy, but eliminating some stats may after critical functionality. Here are the stats that we strongly recommend not disabling: cluster_manager.warming_clusters, cluster.<cluster_name>.membership_total,cluster.<cluster_name>.membership_healthy, cluster.<cluster_name>.membership_degraded,reference https://github.com/envoyproxy/envoy/issues/9856, https://github.com/envoyproxy/envoy/issues/14610
enableVirtualHostStats
boolean
true
EnableVirtualHostStats enables envoy stat metrics for virtual hosts.
enablePerEndpointStats
boolean
true
EnablePerEndpointStats enables per endpoint envoy stats metrics. Please use with caution.
ProxyOpenTelemetrySink
ProxyOpenTelemetrySink defines the configuration for OpenTelemetry sink.
BackendRefs references a Kubernetes object that represents the backend server to which the metric will be sent. Only service Kind is supported for now.
Configure the compression on Prometheus endpoint. Compression is useful in situations when bandwidth is scarce and large payloads can be effectively compressed at the expense of higher CPU load.
ProxyProtocol
ProxyProtocol defines the configuration related to the proxy protocol
when communicating with the backend.
SamplingRate controls the rate at which traffic will be selected for tracing if no prior sampling decision has been made. Defaults to 100, valid values [0-100]. 100 indicates 100% sampling.
Backend holds the configuration associated with the database backend used by the rate limit service to store state associated with global ratelimiting.
Timeout specifies the timeout period for the proxy to access the ratelimit server If not set, timeout is 20ms.
failClosed
boolean
true
FailClosed is a switch used to control the flow of traffic when the response from the ratelimit server cannot be obtained. If FailClosed is false, let the traffic pass, otherwise, don’t let the traffic pass and return 500. If not set, FailClosed is False.
ClientSelectors holds the list of select conditions to select specific clients using attributes from the traffic flow. All individual select conditions must hold True for this rule and its limit to be applied.
If no client selectors are specified, the rule applies to all traffic of the targeted Route.
If the policy targets a Gateway, the rule applies to each Route of the Gateway. Please note that each Route has its own rate limit counters. For example, if a Gateway has two Routes, and the policy has a rule with limit 10rps, each Route will have its own 10rps limit.
Limit holds the rate limit values. This limit is applied for traffic flows when the selectors compute to True, causing the request to be counted towards the limit. The limit is enforced and the request is ratelimited, i.e. a response with 429 HTTP status code is sent back to the client when the selected requests have reached the limit.
RateLimitSelectCondition
RateLimitSelectCondition specifies the attributes within the traffic flow that can
be used to select a subset of clients to be ratelimited.
All the individual conditions must hold True for the overall condition to hold True.
Headers is a list of request headers to match. Multiple header values are ANDed together, meaning, a request MUST match all the specified headers. At least one of headers or sourceCIDR condition must be specified.
SamplingRate controls the rate at which traffic will be selected for tracing if no prior sampling decision has been made. Defaults to 100, valid values [0-100]. 100 indicates 100% sampling.
HttpStatusCodes specifies the http status codes to be retried. The retriable-status-codes trigger must also be configured for these status codes to trigger a retry.
SecurityPolicy
SecurityPolicy allows the user to configure various security settings for a
Gateway.
TargetRef is the name of the Gateway resource this policy is being attached to. This Policy and the TargetRef MUST be in the same namespace for this Policy to have effect and be applied to the Gateway.
ExtAuth defines the configuration for External Authorization.
ServiceExternalTrafficPolicy
Underlying type:string
ServiceExternalTrafficPolicy describes how nodes distribute service traffic they
receive on one of the Service’s “externally-facing” addresses (NodePorts, ExternalIPs,
and LoadBalancer IPs.
ServiceExternalTrafficPolicyCluster routes traffic to all endpoints.
Local
ServiceExternalTrafficPolicyLocal preserves the source IP of the traffic by routing only to endpoints on the same node as the traffic was received on (dropping the traffic if there are no local endpoints).
ServiceType
Underlying type:string
ServiceType string describes ingress methods for a service
DrainTimeout defines the graceful drain timeout. This should be less than the pod’s terminationGracePeriodSeconds. If unspecified, defaults to 600 seconds.
SourceMatchExact All IP Addresses within the specified Source IP CIDR are treated as a single client selector and share the same rate limit bucket.
Distinct
SourceMatchDistinct Each IP Address within the specified Source IP CIDR is treated as a distinct client selector and uses a separate rate limit bucket/counter. Note: This is only supported for Global Rate Limits.
StringMatch
StringMatch defines how to match any strings.
This is a general purpose match condition that can be used by other EG APIs
that need to match against a string.
Value specifies the string value that the match must have.
StringMatchType
Underlying type:string
StringMatchType specifies the semantics of how a string value should be compared.
Valid MatchType values are “Exact”, “Prefix”, “Suffix”, “RegularExpression”.
StringMatchExact :the input string must match exactly the match value.
Prefix
StringMatchPrefix :the input string must start with the match value.
Suffix
StringMatchSuffix :the input string must end with the match value.
RegularExpression
StringMatchRegularExpression :The input string must match the regular expression specified in the match value. The regex string must adhere to the syntax documented in https://github.com/google/re2/wiki/Syntax.
TCPActiveHealthChecker
TCPActiveHealthChecker defines the settings of tcp health check.
Max specifies the maximal TLS protocol version to allow The default is TLS 1.3 if this is not specified.
ciphers
string array
false
Ciphers specifies the set of cipher suites supported when negotiating TLS 1.0 - 1.2. This setting has no effect for TLS 1.3. In non-FIPS Envoy Proxy builds the default cipher list is: - [ECDHE-ECDSA-AES128-GCM-SHA256|ECDHE-ECDSA-CHACHA20-POLY1305] - [ECDHE-RSA-AES128-GCM-SHA256|ECDHE-RSA-CHACHA20-POLY1305] - ECDHE-ECDSA-AES256-GCM-SHA384 - ECDHE-RSA-AES256-GCM-SHA384 In builds using BoringSSL FIPS the default cipher list is: - ECDHE-ECDSA-AES128-GCM-SHA256 - ECDHE-RSA-AES128-GCM-SHA256 - ECDHE-ECDSA-AES256-GCM-SHA384 - ECDHE-RSA-AES256-GCM-SHA384
ecdhCurves
string array
false
ECDHCurves specifies the set of supported ECDH curves. In non-FIPS Envoy Proxy builds the default curves are: - X25519 - P-256 In builds using BoringSSL FIPS the default curve is: - P-256
signatureAlgorithms
string array
false
SignatureAlgorithms specifies which signature algorithms the listener should support.
ALPNProtocols supplies the list of ALPN protocols that should be exposed by the listener. By default h2 and http/1.1 are enabled. Supported values are: - http/1.0 - http/1.1 - h2
BackendRefs references a Kubernetes object that represents the backend server to which the accesslog will be sent. Only service Kind is supported for now.
The upstream server responds with any 5xx response code, or does not respond at all (disconnect/reset/read timeout). Includes connect-failure and refused-stream.
gateway-error
The response is a gateway error (502,503 or 504).
reset
The upstream server does not respond at all (disconnect/reset/read timeout.)
connect-failure
Connection failure to the upstream server (connect timeout, etc.). (Included in 5xx)
retriable-4xx
The upstream server responds with a retriable 4xx response code. Currently, the only response code in this category is 409.
refused-stream
The upstream server resets the stream with a REFUSED_STREAM error code.
retriable-status-codes
The upstream server responds with any response code matching one defined in the RetriableStatusCodes.
cancelled
The gRPC status code in the response headers is “cancelled”.
deadline-exceeded
The gRPC status code in the response headers is “deadline-exceeded”.
internal
The gRPC status code in the response headers is “internal”.
resource-exhausted
The gRPC status code in the response headers is “resource-exhausted”.
unavailable
The gRPC status code in the response headers is “unavailable”.
Wasm
Wasm defines a wasm extension.
Note: at the moment, Envoy Gateway does not support configuring Wasm runtime.
v8 is used as the VM runtime for the Wasm extensions.
Name is a unique name for this Wasm extension. It is used to identify the Wasm extension if multiple extensions are handled by the same vm_id and root_id. It’s also used for logging/debugging.
rootID
string
true
RootID is a unique ID for a set of extensions in a VM which will share a RootContext and Contexts if applicable (e.g., an Wasm HttpFilter and an Wasm AccessLog). If left blank, all extensions with a blank root_id with the same vm_id will share Context(s). RootID must match the root_id parameter used to register the Context in the Wasm code.
Config is the configuration for the Wasm extension. This configuration will be passed as a JSON string to the Wasm extension.
failOpen
boolean
false
FailOpen is a switch used to control the behavior when a fatal error occurs during the initialization or the execution of the Wasm extension. If FailOpen is set to true, the system bypasses the Wasm extension and allows the traffic to pass through. Otherwise, if it is set to false or not set (defaulting to false), the system blocks the traffic and returns an HTTP 5xx error.
WasmCodeSource
WasmCodeSource defines the source of the wasm code.
WithUnderscoresActionAllow allows headers with underscores to be passed through.
RejectRequest
WithUnderscoresActionRejectRequest rejects the client request. HTTP/1 requests are rejected with the 400 status. HTTP/2 requests end with the stream reset.
DropHeader
WithUnderscoresActionDropHeader drops the client header with name containing underscores. The header is dropped before the filter chain is invoked and as such filters will not see dropped headers.
XDSTranslatorHook
Underlying type:string
XDSTranslatorHook defines the types of hooks that an Envoy Gateway extension may support
for the xds-translator
Fixed certgen to support creating the hmac secret during an upgrade
Translator
Fixed nil secret in resourceversiontable
Add missing http filters to the http filter chain when ClientTrafficPolicy and MergeGateways is enabled
Allow websockets when url rewrite is enabled
Set the Host header for http health checker
Fixed double slashes in redirect URL
Allow ClientTrafficPolicy to attach to multiple http (non https) listeners within the same Gateway
Set path prefix for the http ext auth service
Set the route matching precedence order to Exact > RegularExpression > PathPrefix
Fixed infraIR duplicate port translation for merged gateways
Set SpawnUpstreamSpan to true
Allow rate limit to work with multiple listeners
Infra-manager
Skip creating infra resources when the InfraIR has empty listeners
4.2 - v1.0.0
Date: March 13, 2024
Documentation
Added User Guide for Local Ratelimit
Added User Guide for Circuit Breaker
Added User Guide for fault injection
Added User Guide for EnvoyProxy extraArgs
Added User Guide for Timeouts in ClientTrafficPolicy
Added User Guide for JWT claim base routing
Added User Guide for HTTP Timeout
Added User Guide for Retry in BackendTrafficPolicy
Added User Guide for Basic Auth
Added User Guide for OIDC
Added User Guide for ClientTrafficPolicy
Added User Guide for BackendTrafficPolicy
Added User Guide for Basic Auth using HTTPS
Added User Guide for External Authorization
Added User Guide for Routing Outside Kubernetes
Added User Guide for BackendTLSPolicy
Added User Guide for Mutual TLS from External Clients to the Gateway
Added User Guide for Control Plane Authentication using custom certs
Added User Guide for Multiple Gatewayclass and Merge Gateways Deployment Mode
Added Type and required for CRD API doc
Refactored Structure of User Guide docs
Refactored Move Design docs under “Get Involved”
Updated crd-ref-docs to 0.0.10
Updated Envoy proxy image to envoy:distroless-dev in main
Installation
Added Support for Pulling envoyGateway image from a private registry
Added Support for Configuring resources for certgen job
Added Support for Configuring affinity for EnvoyGateway pod
API
Added Support for Downstream QUIC/HTTP3 in ClientTrafficPolicy CRD
Added Support for Downstream MTLS in ClientTrafficPolicy CRD
Added Support for Enabling EnvoyHeaders in ClientTrafficPolicy CRD
Added Support for DisableMergeSlash and escapedSlashesAction in ClientTrafficPolicy CRD
Added Support for EnableTrailers in HTTP/1.1 in ClientTrafficPolicy CRD
Added Support for Preserving header letter-case on HTTP/1 in ClientTrafficPolicy CRD
Added Support for Enabling HTTP/1.0 and HTTP/0.9 in ClientTrafficPolicy CRD
Added Support for Client IP Detection using XFF in ClientTrafficPolicy CRD
Added Support for Client IP Detection using Custom Header in ClientTrafficPolicy CRD
Added Support for Connection Timeouts in ClientTrafficPolicy CRD
Added Support for Common TLS configuration properties in ClientTrafficPolicy CRD
Added Support for Proxy protocol in ClientTrafficPolicy CRD
Added Support for TCPKeepAlive in ClientTrafficPolicy CRD
Added Support for Local rate limit in BackendTrafficPolicy CRD
Added Support for CircuitBreaker in BackendTrafficPolicy CRD
Added Support for Fault injection in BackendTrafficPolicy CRD
Added Support for Passive Health Checks in BackendTrafficPolicy CRD
Added Support for Active Health Checks in BackendTrafficPolicy CRD
Added Support for Connection Timeouts in BackendTrafficPolicy CRD
Added Support for Compressor/Decompressor in BackendTrafficPolicy CRD
Added Support for Retry in BackendTrafficPolicy CRD
Added Support for Slow start mode in BackendTrafficPolicy CRD
Added Support for Proxy protocol in BackendTrafficPolicy CRD
Added Support for TCPKeepAlive in BackendTrafficPolicy CRD
Added Support for PolicyStatus in BackendTrafficPolicy CRD
Added Support for PolicyStatus in ClientTrafficPolicy CRD
Added Support for PolicyStatus in SecurityPolicy CRD
Added Support for OIDC in SecurityPolicy CRD
Added Support for Basic Auth in SecurityPolicy CRD
Added Support for RedirectURL and signoutPath to OIDC in SecurityPolicy CRD
Added Support for ExtractFrom headers and params to JWT in SecurityPolicy CRD
Added Support for External Authorization in SecurityPolicy CRD
Added Support for RecomputeRoute field to JWT in SecurityPolicy CRD
Added Support for AllowCredentials knob to CORS setting in SecurityPolicy CRD
Added Support for Extract from different identifier to JWT in SecurityPolicy CRD
Added Support for Secret resource in EnvoyPatchPolicy CRD
Added Support for Making the value optional for JSONPatchOperation in EnvoyPatchPolicy CRD
Added Support for From field to JSONPatchOperation in EnvoyPatchPolicy CRD
Added Support for MergeGateways in EnvoyPatchPolicy CRD
Added Support for Upstream TLS by implementing BackendTLSPolicy CRD
Added Support for LabelSelector type for NamespaceSelectors in EnvoyGateway Configuration
Added Support for Ratelimit prometheus in EnvoyGateway Configuration
Added Support for Gracefully drain listeners before envoy shutdown on pod termination in EnvoyProxy CRD
Added Support for Configuring externalTrafficPolicy to the envoy service in EnvoyProxy CRD
Added Support for Envoy extra args in EnvoyProxy CRD
Added Support for Mergepatch to envoyproxy/ratelimit deployment in EnvoyProxy CRD
Added Support for Mergepatch to envoyproxy service in EnvoyProxy CRD
Added Support for NodeSelector to PodSpec in EnvoyProxy CRD
Added Support for HorizontalPodAutoscaler in EnvoyProxy CRD
Added Support for TopologySpreadConstraints to PodSpec in EnvoyProxy CRD
Added Support for ImagePullSecrets to PodSpec in EnvoyProxy CRD
Breaking Changes
Use wildcard to match AllowOrigins to CORS in SecurityPolicy CRD
Remove Hostnetwork support in EnvoyProxy CRD
Conformance
Replaced backend image from gcr.io/k8s-staging-ingressconformance/echoserver to gcr.io/k8s-staging-gateway-api/echo-basic
Testing
Added e2e test for Header Case-Preserving
Added e2e test for Timeout in ClientTrafficPolicy
Added e2e test for JWT claim base routing
Added e2e test for OIDC
Added e2e test for BackendTrafficPolicy Retry
Added e2e test for Backend Upgrade
Added e2e test for External Authorization
Added e2e test for Backend TLS policy
Added e2e test for Envoy Gateway Release Upgrade
Added e2e test for Weighted backend
Added validation for LoadBalancerIP to prevent trailing period
Translator
Fixed Prefix match to prevent mismatching routes with the same prefix
Fixed Multiple reconciling by implementing comparable interface for ir.Infra
Fixed EndpointSlice with empty conditions {}
Fixed Error handling when parsing the http request timeout
Fixed No status when EnvoyPatchPolicy is disabled
Fixed Printable for xds and infra IRs
Fixed Skip backendRefs with weight set to 0
Fixed AND Header matches in ratelimiting not working
Fixed Deletion logics when no gatewayclasses exist
Fixed Match mergedGateways irKey for ClientTrafficPolicy
Fixed Policies should apply only to gateways they were attached to when mergeGateways is true
Fixed Listener status is not surfaced for gateways when MergeGateways enabled
Fixed GRPCroute websocket not working by moving web socket upgrade config from hcm to route
Fixed Configure idle timeout when timeout is set on HTTPRoute
Fixed Relaxing HTTPS restriction for OIDC token endpoint
Fixed Panic when translating routes with empty backends
Fixed Xds translation should be done in a best-effort manner
Fixed Delete unused status keys from watchable
Fixed Ignoring finalizers when comparing envoy proxy service
Fixed Don’t override the ALPN array if HTTP/3 is enabled
Fixed Add h3 ALPN by default if HTTP/3 is enabled
Fixed Change the Merge behavior to Replace for SecurityPolicy/BackendTrafficPolicy
Fixed Use service port in alt-svc header if HTTP/3 is enabled
Fixed Prevent policies targeting non-TLS listeners on the same port from conflicting
Fixed Skip the ReasonTargetNotFound for all policies
Fixed Skip publishing empty status for all policies
Added Support for validating regex before sending to Envoy
Added Support for setting spec.addresses.value into ClusterIP when Service Type is ClusterIP
Added Unsupported status condition for filters within BackendRef
Added List instead of map for Provider Resources for order stability
Added Suffix for oauth cookies to prevent multiple oauth filters from overwriting each other’s cookies
Added Support for overriding condition to BackendTrafficPolicy and SecurityPolicy
Added Support for default retry budget and retry host predicate
Added Support for implementing gateway.spec.infrastructure
Added Support for Upstream TLS to multiple Backends
Added Validation for CA Cert in ClientTrafficPolicy
Providers
Added Support for multiple GatewayClass per controller
Added SecurityPolicyIndexers in Kubernetes Provider
Added Support for generating HMAC secret in CertGen Job
Fixed Finalizer logic when deleting Gatewayclasses
Fixed MergeGateways panics when restarting control plane
xDS
Added Support for EDS cache
Added Support for ADS cache to ensure the rule order
Fixed Deprecated field error when using RequestHeaderModifier filter
Fixed Envoy rejects XDS at runtime losing all routes on restart
Fixed Requests not matching defined routes trigger per-route filters
Bumped go-control-plane to v0.12.0
Cli
Added Support for egctl x status
Added Support for egctl experimental dashboard envoy-proxy
Added Support for egctl config ratelimit
Added Support for egctl translate from gateway-api resources to IR
4.3 - v0.6.0
Date: Nov 1, 2023
Documentation
Introduced a new website based on Hugo
Added Grafana dashboards and integration docs for EnvoyProxy metrics
Added Grafana integration docs for Gateway API metrics
Installation
Updated EnvoyProxy image to be a distroless variant.
Removed resources around kube-rbac-proxy
API
Upgraded to Gateway API v1.0.0
Added the ClientTrafficPolicy CRD with Keep Alive Support
Added the BackendTrafficPolicy CRD with RateLimit and LoadBalancer Support
Added the SecurityPolicy CRD with CORS and JWT Support
Added EnvoyGateway Metrics with Prometheus and OpenTelemetry support
Added Support for InitContainers in EnvoyProxy CRD
Added Support for LoadBalancerIP in EnvoyProxy CRD
Added Support for AllocateLoadBalancerNodePorts in EnvoyProxy CRD
Added Support for LoadBalancerClass in EnvoyProxy CRD
Added Support for selecting EnvoyProxy stats to be generated
Added Support for enabling EnvoyProxy Virtual Host metrics
Added Support for Merging Gateway resources onto the same infrastructure
Breaking changes
Removed the AuthenticationFilter CRD
Removed the RateLimitFilter CRD
Moved EnvoyProxy CRD from config.gateway.envoyproxy.io to gateway.envoyproxy.io
Enabled EnvoyProxy Prometheus Endpoint by default with an option to disable it
Updated the Bootstrap field within the EnvoyProxy CRD with an additional value
field to specify bootstrap config
Conformance
Added Support for HTTPRouteBackendProtocolH2C Test
Added Support for HTTPRouteBackendProtocolWebSocket Test
Added Support for HTTPRouteRequestMultipleMirrors Test
Added Support for HTTPRouteTimeoutRequest Test
Added Support for HTTPRouteTimeoutBackendRequest Test
Added Support for HTTPRouteRedirectPortAndScheme Test
Watchable
Improved caching of resource by implementing a compare function agnostic of resource order
Translator
Added support for routing to EndpointSlice endpoints
Added support for HTTPRoute Timeouts
Added support for multiple RequestMirror filters per HTTPRoute rule
Use / instead of - in IR Route Names
Added Support to ignore ports in Host header
Providers
Added the generationChangedPredicate to most resources to limit resource reconiliation
Improved reconiliation by using the same enqueue request for all resources
Added support for reconciling ServiceImport CRD
Added support for selectively watching resources based on Namespace Selector
XDS
Fixed Layered Runtime warnings
Upgraded to the latest version of go-control-plane that fixed xDS Resource ordering issues for ADS.
Added HTTP2 Keep Alives to the xds connection
Cli
Added Support for egctl stats command
4.4 - v0.6.0-rc.1
Date: Oct 27, 2023
Documentation
Introduced a new website based on Hugo
Added Grafana dashboards and integration docs for EnvoyProxy metrics
Added Grafana integration docs for Gateway API metrics
Installation
Added Support for configuring Envoy Gateway Label and Annotations using Helm
Increased default Resource defaults for Envoy Gateway to 100m CPU and 256Mi Memory
Fixes Helm values for EnvoyGateway startup configuration
Added opt-in field to skip creating control plane TLS Certificates allowing users to bring their own certificates.
API
Upgraded to Gateway API v1.0.0
Added the ClientTrafficPolicy CRD with Keep Alive Support
Added the BackendTrafficPolicy CRD with RateLimit and LoadBalancer Support
Added the SecurityPolicy CRD with CORS and JWT Support
Added EnvoyGateway Metrics with Prometheus and OpenTelemetry support
Added Support for InitContainers in EnvoyProxy CRD
Added Support for LoadBalancerIP in EnvoyProxy CRD
Added Support for AllocateLoadBalancerNodePorts in EnvoyProxy CRD
Added Support for LoadBalancerClass in EnvoyProxy CRD
Added Support for selecting EnvoyProxy stats to be generated
Added Support for enabling EnvoyProxy Virtual Host metrics
Added Support for Merging Gateway resources onto the same infrastructure
Breaking changes
Removed the AuthenticationFilter CRD
Removed the RateLimitFilter CRD
Enabled EnvoyProxy Prometheus Endpoint by default with an option to disable it
Updated the Bootstrap field within the EnvoyProxy CRD with an additional value
field to specify bootstrap config
watchable
Improved caching of resource by implementing a compare function agnostic of resource order
Translator
Breaking changes
Added support for routing to EndpointSlice endpoints
Added support for HTTPRoute Timeouts
Added support for multiple RequestMirror filters per HTTPRoute rule
Use / instead of - in IR Route Names
Added Support to ignore ports in Host header
Providers
Added the generationChangedPredicate to most resources to limit resource reconiliation
Improved reconiliation by using the same enqueue request for all resources
Added support for reconciling ServiceImport CRD
Added support for selectively watching resources based on Namespace Selector
XDS
Fixed Layered Runtime warnings
Upgraded to the latest version of go-control-plane that fixed xDS Resource ordering issues for ADS.
Added HTTP2 Keep Alives to the xds connection
Cli
Added Support for egctl stats command
4.5 - v0.5.0
Date: July 26, 2023
Documentation
Added Docs for Installation page using Helm
Added Docs for Cert Manager Integration
Added Docs for Presentation Links
Added Docs for configuring multiple TLS Certificates per Listener
Installation
Added Support for configuring Envoy Gateway Label and Annotations using Helm
Increased default Resource defaults for Envoy Gateway to 100m CPU and 256Mi Memory
Fixes Helm values for EnvoyGateway startup configuration
Added opt-in field to skip creating control plane TLS Certificates allowing users to bring their own certificates.
API
Upgraded to Gateway API v0.7.1
Added Support for EnvoyPatchPolicy
Added Support for EnvoyProxy Telemetry - Access Logging, Traces and Metrics
Added Support for configuring EnvoyProxy Pod Labels
Added Support for configuring EnvoyProxy Deployment Strategy Settings, Volumes and Volume Mounts
Added Support for configuring EnvoyProxy as a NodePort Type Service
Added Support for Distinct RateLimiting for IP Addresses
Added Support for converting JWT Claims to Headers, to be used for RateLimiting
Added Admin Server for Envoy Gateway
Added Pprof Debug Support for Envoy Gateway
Added Support to Watch for Resources in Select Namespaces
Breaking Changes
Renamed field in EnvoyGateway API from Extension to ExtensionManager
CI Tooling Testing
Added Retest Github Action
Added CherryPick Github Action
Added E2E Step in Github CI Workflow
Added RateLimit E2E Tests
Added JWT Claim based RateLimit E2E Tests
Added Access Logging E2E tests
Added Metrics E2E tests
Added Tracing E2E tests
Conformance
Enabled GatewayWithAttachedRoutes Test
Enabled HttpRouteRequestMirror Test
Skipped HTTPRouteRedirectPortAndScheme Test
Translator
Breaking Changes
Renamed IR resources from - to /
which also affects generated Xds Resources
Providers
Reconcile Node resources to be able to compute Status Addresses for Gateway
Discard Status before publishing Provider resources to reduce memory consumption
xDS
Fix Init Race in Xds Runner when starting Xds Server and receiving Xds Input
Switched to Xds SOTW Server for RateLimit Service Configuration
Added Control Plane TLS between EnvoyProxy and RateLimit Service
Enabled adding RateLimit Headers when RateLimit is set
Allowed GRPCRoute and HTTPRoute to be linked to the same HTTPS Listener
Set ALPN in the Xds Listener with TLS enabled.
Added Best Practices Default Edge Settings to Xds Resources
Compute and Publish EnvoyPatchPolicy status from xds-translator runner
Cli
Added egctl x translate Support to generate default missing Resources
Added egctl x translate Support for AuthenticationFilter and EnvoyPatchPolicy
4.6 - v0.5.0-rc.1
Date: July 26, 2023
Documentation
Added Docs for Installation page using Helm
Added Docs for Cert Manager Integration
Added Docs for Presentation Links
Added Docs for configuring multiple TLS Certificates per Listener
Installation
Added Support for configuring Envoy Gateway Label and Annotations using Helm
Increased default Resource defaults for Envoy Gateway to 100m CPU and 256Mi Memory
Fixes Helm values for EnvoyGateway startup configuration
Added opt-in field to skip creating control plane TLS Certificates allowing users to bring their own certificates.
API
Upgraded to Gateway API v0.7.1
Added Support for EnvoyPatchPolicy
Added Support for EnvoyProxy Telemetry - Access Logging, Traces and Metrics
Added Support for configuring EnvoyProxy Pod Labels
Added Support for configuring EnvoyProxy Deployment Strategy Settings, Volumes and Volume Mounts
Added Support for configuring EnvoyProxy as a NodePort Type Service
Added Support for Distinct RateLimiting for IP Addresses
Added Support for converting JWT Claims to Headers, to be used for RateLimiting
Added Admin Server for Envoy Gateway
Added Pprof Debug Support for Envoy Gateway
Added Support to Watch for Resources in Select Namespaces
Breaking Changes
Renamed field in EnvoyGateway API from Extension to ExtensionManager
CI Tooling Testing
Added Retest Github Action
Added CherryPick Github Action
Added E2E Step in Github CI Workflow
Added RateLimit E2E Tests
Added JWT Claim based RateLimit E2E Tests
Added Access Logging E2E tests
Added Metrics E2E tests
Added Tracing E2E tests
Conformance
Enabled GatewayWithAttachedRoutes Test
Enabled HttpRouteRequestMirror Test
Skipped HTTPRouteRedirectPortAndScheme Test
Translator
Breaking changes
Renamed IR resources from - to /
which also affects generated Xds Resources
Providers
Reconcile Node resources to be able to compute Status Addresses for Gateway
Discard Status before publishing Provider resources to reduce memory consumption
xDS
Fix Init Race in Xds Runner when starting Xds Server and receiving Xds Input
Switched to Xds SOTW Server for RateLimit Service Configuration
Added Control Plane TLS between EnvoyProxy and RateLimit Service
Enabled adding RateLimit Headers when RateLimit is set
Allowed GRPCRoute and HTTPRoute to be linked to the same HTTPS Listener
Set ALPN in the Xds Listener with TLS enabled.
Added Best Practices Default Edge Settings to Xds Resources
Compute and Publish EnvoyPatchPolicy status from xds-translator runner
Cli
Added egctl x translate Support to generate default missing Resources
Added egctl x translate Support for AuthenticationFilter and EnvoyPatchPolicy
4.7 - v0.4.0
Date: April 24, 2023
Documentation
Added Docs for Installing and Using egctl
Installation
Added Helm Installation Support
Added Support for Ratelimiting Based On IP Subnet
Added Gateway API Support Doc
Added Namespace Resource to Helm Templates
Updated Installation Yaml to Use the envoy-gateway-system Namespace
API
Upgraded to Gateway API v0.6.2
Added Support for Custom Envoy Proxy Bootstrap Config
Added Support for Configuring the Envoy Proxy Image and Service
Added Support for Configuring Annotations, Resources, and Securitycontext Settings on Ratelimit Infra and Envoy Proxy
Added Support for Using Multiple Certificates on a Single Fully Qualified Domain Name
Gateway Status Address is now Populated for ClusterIP type Envoy Services
Envoy Proxy Pod and Container SecurityContext is now Configurable
Added Custom Envoy Gateway Extensions Framework
Added Support for Service Method Match in GRPCRoute
Fixed a Bug in the Extension Hooks for xDS Virtual Hosts and Routes
CI Tooling Testing
Fixed CI Flakes During Helm Install
Added Test To Ensure Static xDS Cluster Has Same Field Values as Dynamic Cluster
Added egctl to Build and Test CI Workflow
Code Coverage Thresholds are now Enforced by CI
Fixed latest-release-check CI Job Failures
Added Auto Release Tooling for Charts
Conformance
Enabled GatewayWithAttachedRoutes Test
Enabled Enable HTTPRouteInvalidParentRefNotMatchingSectionName Test
Enabled Enable HTTPRouteDisallowedKind Test
Re-Enabled Gateway/HTTPRouteObservedGenerationBump Test
Translator
Added Support for Dynamic GatewayControllerName in Route Status
Providers
Update GatewayClass Status Based on EnvoyProxy Config Validation
xDS
Added EDS Support
Fixed PathSeparatedPrefix and Optimized Logic for Prefixes Ending With Trailing Slash
Updated Deprecated RegexMatcher
Refactored Authn and Ratelimit Features to Reuse buildXdsCluster
Cli
Added egctl CLI Tool
Added egctl Support for Dry Runs of Gateway API Config
Added egctl Support for Dumping Envoy Proxy xDS Resources
4.8 - v0.4.0-rc.1
Date: April 13, 2023
Documentation
Added Docs for Installing and Using egctl
Installation
Added Helm Installation Support
Added Support for Ratelimiting Based On IP Subnet
Added Gateway API Support Doc
API
Upgraded to Gateway API v0.6.2
Added Support for Custom Envoy Proxy Bootstrap Config
Added Support for Configuring the Envoy Proxy Image and Service
Added Support for Configuring Annotations, Resources, and Securitycontext Settings on Ratelimit Infra and Envoy Proxy
Added Support for Using Multiple Certificates on a Single Fully Qualified Domain Name
Gateway Status Address is now Populated for ClusterIP type Envoy Services
Envoy Proxy Pod and Container SecurityContext is now Configurable
Added Custom Envoy Gateway Extensions Framework
Added Support for Service Method Match in GRPCRoute
CI Tooling Testing
Fixed CI Flakes During Helm Install
Added Test To Ensure Static xDS Cluster Has Same Field Values as Dynamic Cluster
Added egctl to Build and Test CI Workflow
Code Coverage Thresholds are now Enforced by CI
Fixed latest-release-check CI Job Failures
Added Auto Release Tooling for Charts
Conformance
Enabled GatewayWithAttachedRoutes Test
Enabled Enable HTTPRouteInvalidParentRefNotMatchingSectionName Test
Enabled Enable HTTPRouteDisallowedKind Test
Re-Enabled Gateway/HTTPRouteObservedGenerationBump Test
Translator
Added Support for Dynamic GatewayControllerName in Route Status
Providers
Update GatewayClass Status Based on EnvoyProxy Config Validation
xDS
Added EDS Support
Fixed PathSeparatedPrefix and Optimized Logic for Prefixes Ending With Trailing Slash
Updated Deprecated RegexMatcher
Refactored Authn and Ratelimit Features to Reuse buildXdsCluster
Cli
Added egctl CLI Tool
Added egctl Support for Dry Runs of Gateway API Config
Added egctl Support for Dumping Envoy Proxy xDS Resources
4.9 - v0.3.0
Date: February 09, 2023
Documentation
Added Global Rate Limit User Docs
Added Request Authentication User Docs
Added TCP Routing User Docs
Added UDP Routing User Docs
Added GRPC Routing User Docs
Added HTTP Response Headers User Docs
Added TCP and UDP Proxy Design Docs
Added egctl Design Docs
Added Rate Limit Design Docs
Added Request Authentication Design Docs
Added Support for Versioned Docs
Added Support for Multiple Release Versions
Added Release Details Docs
Added API Docs Generating Tooling
Refactored Layout for User Docs
API
Upgraded to v0.6.1 Gateway API
Added Support for the TCPRoute API
Added Support for the UDPRoute API
Added Support for the GRPCRoute API
Added Support for HTTPRoute URLRewrite Filter
Added Support for HTTPRoute RequestMirror Filter
Added Support for HTTPRoute ResponseHeaderModifier Filter
Added Support for Request Authentication
Added Support for Global Rate Limiting
Added Support for Routes ReferenceGrant
Added Support for Namespace Server Config Type
Added initial management of Envoy Proxy deployment via EnvoyProxy API
CI Tooling Testing
Fixed Make Image Failed in Darwin
Fixed Wait for Job Succeeded before conformance test
Upgraded Echoserver Image Tag
Added Support for User-Facing Version
Added Support for Testing EG against Multiple Kubernetes Versions
Conformance
Enabled GatewayClassObservedGenerationBump conformance test
Enabled GatewayInvalidTLSConfiguration conformance test
Enabled GatewayInvalidRouteKind conformance test
Enabled HTTPRouteReferenceGrant conformance test
Enabled HTTPRouteMethodMatching conformance test
Enabled HTTPRoutePartiallyInvalidViaInvalidReferenceGrant conformance test
Enabled HTTPRouteInvalidParentRefNotMatchingListenerPort conformance test
(Currently EG passes all conformance tests except redirect and gateway/httproute ObservedGenerationBump tests. Redirect tests are failing due to a possible issue with the way upstream conformance tests have made assumptions. Skip them for now until below issues #992 #993 #994 are resolved)
IR
Added TCP Listener per TLSRoute
Translator
Fixes Remove Stale Listener Condition
Added Support for Suffix Matches for Headers
Added Support for HTTP Method Matching to HTTPRoute
Added Support for Regex Match Type
Added Support for HTTPQueryParamMatch
Providers
Refactored Kubernetes Provider to Single Reconciler
Upgraded Kube Provider Test Data Manifests to v0.6.1
Removed Duplicate Settings from Bootstrap Config
Updated Certgen to Use EG Namespace Env
Added EnvoyProxy to Translator and Kube Infra Manager
Upgraded Envoyproxy Image to envoy-dev latest in Main