This is the multi-page printable view of this section. Click here to print.
Design
1 - Goals
The high-level goal of the Envoy Gateway project is to attract more users to Envoy by lowering barriers to adoption through expressive, extensible, role-oriented APIs that support a multitude of ingress and L7/L4 traffic routing use cases; and provide a common foundation for vendors to build value-added products without having to re-engineer fundamental interactions.
Objectives
Expressive API
The Envoy Gateway project will expose a simple and expressive API, with defaults set for many capabilities.
The API will be the Kubernetes-native Gateway API, plus Envoy-specific extensions and extension points. This expressive and familiar API will make Envoy accessible to more users, especially application developers, and make Envoy a stronger option for “getting started” as compared to other proxies. Application developers will use the API out of the box without needing to understand in-depth concepts of Envoy Proxy or use OSS wrappers. The API will use familiar nouns that users understand.
The core full-featured Envoy xDS APIs will remain available for those who need more capability and for those who add functionality on top of Envoy Gateway, such as commercial API gateway products.
This expressive API will not be implemented by Envoy Proxy, but rather an officially supported translation layer on top.
Batteries included
Envoy Gateway will simplify how Envoy is deployed and managed, allowing application developers to focus on delivering core business value.
The project plans to include additional infrastructure components required by users to fulfill their Ingress and API gateway needs: It will handle Envoy infrastructure provisioning (e.g. Kubernetes Service, Deployment, et cetera), and possibly infrastructure provisioning of related sidecar services. It will include sensible defaults with the ability to override. It will include channels for improving ops by exposing status through API conditions and Kubernetes status sub-resources.
Making an application accessible needs to be a trivial task for any developer. Similarly, infrastructure administrators will enjoy a simplified management model that doesn’t require extensive knowledge of the solution’s architecture to operate.
All environments
Envoy Gateway will support running natively in Kubernetes environments as well as non-Kubernetes deployments.
Initially, Kubernetes will receive the most focus, with the aim of having Envoy Gateway become the de facto standard for Kubernetes ingress supporting the Gateway API. Additional goals include multi-cluster support and various runtime environments.
Extensibility
Vendors will have the ability to provide value-added products built on the Envoy Gateway foundation.
It will remain easy for end-users to leverage common Envoy Proxy extension points such as providing an implementation for authentication methods and rate-limiting. For advanced use cases, users will have the ability to use the full power of xDS.
Since a general-purpose API cannot address all use cases, Envoy Gateway will provide additional extension points for flexibility. As such, Envoy Gateway will form the base of vendor-provided managed control plane solutions, allowing vendors to shift to a higher management plane layer.
Non-objectives
Cannibalize vendor models
Vendors need to have the ability to drive commercial value, so the goal is not to cannibalize any existing vendor monetization model, though some vendors may be affected by it.
Disrupt current Envoy usage patterns
Envoy Gateway is purely an additive convenience layer and is not meant to disrupt any usage pattern of any user with Envoy Proxy, xDS, or go-control-plane.
Personas
In order of priority
1. Application developer
The application developer spends the majority of their time developing business logic code. They require the ability to manage access to their application.
2. Infrastructure administrators
The infrastructure administrators are responsible for the installation, maintenance, and operation of API gateways appliances in infrastructure, such as CRDs, roles, service accounts, certificates, etc. Infrastructure administrators support the needs of application developers by managing instances of Envoy Gateway.
2 - System Design
Goals
- Define the system components needed to satisfy the requirements of Envoy Gateway.
Non-Goals
- Create a detailed design and interface specification for each system component.
Terminology
- Control Plane- A collection of inter-related software components for providing application gateway and routing functionality. The control plane is implemented by Envoy Gateway and provides services for managing the data plane. These services are detailed in the components section.
- Data Plane- Provides intelligent application-level traffic routing and is implemented as one or more Envoy proxies.
Architecture
Configuration
Envoy Gateway is configured statically at startup and the managed data plane is configured dynamically through Kubernetes resources, primarily Gateway API objects.
Static Configuration
Static configuration is used to configure Envoy Gateway at startup, i.e. change the GatewayClass controllerName, configure a Provider, etc. Currently, Envoy Gateway only supports configuration through a configuration file. If the configuration file is not provided, Envoy Gateway starts-up with default configuration parameters.
Dynamic Configuration
Dynamic configuration is based on the concept of a declaring the desired state of the data plane and using reconciliation loops to drive the actual state toward the desired state. The desired state of the data plane is defined as Kubernetes resources that provide the following services:
- Infrastructure Management- Manage the data plane infrastructure, i.e. deploy, upgrade, etc. This configuration is
expressed through GatewayClass and Gateway resources. The
EnvoyProxyCustom Resource can be referenced bygatewayclass.spec.parametersRefto modify data plane infrastructure default parameters, e.g. expose Envoy network endpoints using aClusterIPservice instead of aLoadBalancerservice. - Traffic Routing- Define how to handle application-level requests to backend services. For example, route all HTTP requests for “www.example.com” to a backend service running a web server. This configuration is expressed through HTTPRoute and TLSRoute resources that match, filter, and route traffic to a backend. Although a backend can be any valid Kubernetes Group/Kind resource, Envoy Gateway only supports a Service reference.
Components
Envoy Gateway is made up of several components that communicate in-process; how this communication happens is described in the Watching Components Design.
Provider
A Provider is an infrastructure component that Envoy Gateway calls to establish its runtime configuration, resolve services, persist data, etc. As of v0.2, Kubernetes is the only implemented provider. A file provider is on the roadmap via Issue #37. Other providers can be added in the future as Envoy Gateway use cases are better understood. A provider is configured at start up through Envoy Gateway’s static configuration.
Kubernetes Provider
- Uses Kubernetes-style controllers to reconcile Kubernetes resources that comprise the dynamic configuration.
- Manages the data plane through Kubernetes API CRUD operations.
- Uses Kubernetes for Service discovery.
- Uses etcd (via Kubernetes API) to persist data.
File Provider
- Uses a file watcher to watch files in a directory that define the data plane configuration.
- Manages the data plane by calling internal APIs, e.g.
CreateDataPlane(). - Uses the host’s DNS for Service discovery.
- If needed, the local filesystem is used to persist data.
Resource Watcher
The Resource Watcher watches resources used to establish and maintain Envoy Gateway’s dynamic configuration. The mechanics for watching resources is provider-specific, e.g. informers, caches, etc. are used for the Kubernetes provider. The Resource Watcher uses the configured provider for input and provides resources to the Resource Translator as output.
Resource Translator
The Resource Translator translates external resources, e.g. GatewayClass, from the Resource Watcher to the Intermediate Representation (IR). It is responsible for:
- Translating infrastructure-specific resources/fields from the Resource Watcher to the Infra IR.
- Translating proxy configuration resources/fields from the Resource Watcher to the xDS IR.
Note: The Resource Translator is implemented as the Translator API type in the gatewayapi package.
Intermediate Representation (IR)
The Intermediate Representation defines internal data models that external resources are translated into. This allows Envoy Gateway to be decoupled from the external resources used for dynamic configuration. The IR consists of an Infra IR used as input for the Infra Manager and an xDS IR used as input for the xDS Translator.
- Infra IR- Used as the internal definition of the managed data plane infrastructure.
- xDS IR- Used as the internal definition of the managed data plane xDS configuration.
xDS Translator
The xDS Translator translates the xDS IR into xDS Resources that are consumed by the xDS server.
xDS Server
The xDS Server is a xDS gRPC Server based on Go Control Plane. Go Control Plane implements the Delta xDS Server Protocol and is responsible for using xDS to configure the data plane.
Infra Manager
The Infra Manager is a provider-specific component responsible for managing the following infrastructure:
- Data Plane - Manages all the infrastructure required to run the managed Envoy proxies. For example, CRUD Deployment, Service, etc. resources to run Envoy in a Kubernetes cluster.
- Auxiliary Control Planes - Optional infrastructure needed to implement application Gateway features that require external integrations with the managed Envoy proxies. For example, Global Rate Limiting requires provisioning and configuring the Envoy Rate Limit Service and the Rate Limit filter. Such features are exposed to users through the Custom Route Filters extension.
The Infra Manager consumes the Infra IR as input to manage the data plane infrastructure.
Design Decisions
- Envoy Gateway consumes one GatewayClass by comparing its configured controller name with
spec.controllerNameof a GatewayClass. If multiple GatewayClasses exist with the samespec.controllerName, Envoy Gateway follows Gateway API guidelines to resolve the conflict.gatewayclass.spec.parametersRefrefers to theEnvoyProxycustom resource for configuring the managed proxy infrastructure. If unspecified, default configuration parameters are used for the managed proxy infrastructure. - Envoy Gateway manages Gateways that reference its GatewayClass.
- A Gateway resource causes Envoy Gateway to provision managed Envoy proxy infrastructure.
- Envoy Gateway groups Listeners by Port and collapses each group of Listeners into a single Listener if the Listeners
in the group are compatible. Envoy Gateway considers Listeners to be compatible if all the following conditions are
met:
- Either each Listener within the group specifies the “HTTP” Protocol or each Listener within the group specifies either the “HTTPS” or “TLS” Protocol.
- Each Listener within the group specifies a unique “Hostname”.
- As a special case, one Listener within a group may omit “Hostname”, in which case this Listener matches when no other Listener matches.
- Envoy Gateway does not merge listeners across multiple Gateways.
- Envoy Gateway follows Gateway API guidelines to resolve any conflicts.
- A Gateway
listenercorresponds to an Envoy proxy Listener.
- A Gateway
- An HTTPRoute resource corresponds to an Envoy proxy Route.
- Each backendRef corresponds to an Envoy proxy Cluster.
- The goal is to make Envoy Gateway components extensible in the future. See the roadmap for additional details.
The draft for this document is here.
3 - Watching Components Design
Envoy Gateway is made up of several components that communicate in-process. Some of them (namely Providers) watch external resources, and “publish” what they see for other components to consume; others watch what another publishes and act on it (such as the resource translator watches what the providers publish, and then publishes its own results that are watched by another component). Some of these internally published results are consumed by multiple components.
To facilitate this communication use the watchable library. The watchable.Map type is very similar to the
standard library’s sync.Map type, but supports a .Subscribe (and .SubscribeSubset) method that promotes a pub/sub
pattern.
Pub
Many of the things we communicate around are naturally named, either by a bare “name” string or by a “name”/“namespace”
tuple. And because watchable.Map is typed, it makes sense to have one map for each type of thing (very similar to if
we were using native Go maps). For example, a struct that might be written to by the Kubernetes provider, and read by
the IR translator:
type ResourceTable struct {
// gateway classes are cluster-scoped; no namespace
GatewayClasses watchable.Map[string, *gwapiv1b1.GatewayClass]
// gateways are namespace-scoped, so use a k8s.io/apimachinery/pkg/types.NamespacedName as the map key.
Gateways watchable.Map[types.NamespacedName, *gwapiv1b1.Gateway]
HTTPRoutes watchable.Map[types.NamespacedName, *gwapiv1b1.HTTPRoute]
}
The Kubernetes provider updates the table by calling table.Thing.Store(name, val) and table.Thing.Delete(name);
updating a map key with a value that is deep-equal (usually reflect.DeepEqual, but you can implement your own .Equal
method) the current value is a no-op; it won’t trigger an event for subscribers. This is handy so that the publisher
doesn’t have as much state to keep track of; it doesn’t need to know “did I already publish this thing”, it can just
.Store its data and watchable will do the right thing.
Sub
Meanwhile, the translator and other interested components subscribe to it with table.Thing.Subscribe (or
table.Thing.SubscribeSubset if they only care about a few “Thing"s). So the translator goroutine might look like:
func(ctx context.Context) error {
for snapshot := range k8sTable.HTTPRoutes.Subscribe(ctx) {
fullState := irInput{
GatewayClasses: k8sTable.GatewayClasses.LoadAll(),
Gateways: k8sTable.Gateways.LoadAll(),
HTTPRoutes: snapshot.State,
}
translate(irInput)
}
}
Or, to watch multiple maps in the same loop:
func worker(ctx context.Context) error {
classCh := k8sTable.GatewayClasses.Subscribe(ctx)
gwCh := k8sTable.Gateways.Subscribe(ctx)
routeCh := k8sTable.HTTPRoutes.Subscribe(ctx)
for ctx.Err() == nil {
var arg irInput
select {
case snapshot := <-classCh:
arg.GatewayClasses = snapshot.State
case snapshot := <-gwCh:
arg.Gateways = snapshot.State
case snapshot := <-routeCh:
arg.Routes = snapshot.State
}
if arg.GateWayClasses == nil {
arg.GatewayClasses = k8sTable.GateWayClasses.LoadAll()
}
if arg.GateWays == nil {
arg.Gateways = k8sTable.GateWays.LoadAll()
}
if arg.HTTPRoutes == nil {
arg.HTTPRoutes = k8sTable.HTTPRoutes.LoadAll()
}
translate(irInput)
}
}
From the updates it gets from .Subscribe, it can get a full view of the map being subscribed to via snapshot.State;
but it must read the other maps explicitly. Like sync.Map, watchable.Maps are thread-safe; while .Subscribe is a
handy way to know when to run, .Load and friends can be used without subscribing.
There can be any number of subscribers. For that matter, there can be any number of publishers .Storeing things, but
it’s probably wise to just have one publisher for each map.
The channel returned from .Subscribe is immediately readable with a snapshot of the map as it existed when
.Subscribe was called; and becomes readable again whenever .Store or .Delete mutates the map. If multiple
mutations happen between reads (or if mutations happen between .Subscribe and the first read), they are coalesced in
to one snapshot to be read; the snapshot.State is the most-recent full state, and snapshot.Updates is a listing of
each of the mutations that cause this snapshot to be different than the last-read one. This way subscribers don’t need
to worry about a backlog accumulating if they can’t keep up with the rate of changes from the publisher.
If the map contains anything before .Subscribe is called, that very first read won’t include snapshot.Updates
entries for those pre-existing items; if you are working with snapshot.Update instead of snapshot.State, then you
must add special handling for your first read. We have a utility function ./internal/message.HandleSubscription to
help with this.
Other Notes
The common pattern will likely be that the entrypoint that launches the goroutines for each component instantiates the
map, and passes them to the appropriate publishers and subscribers; same as if they were communicating via a dumb
chan.
A limitation of watchable.Map is that in order to ensure safety between goroutines, it does require that value types
be deep-copiable; either by having a DeepCopy method, being a proto.Message, or by containing no reference types and
so can be deep-copied by naive assignment. Fortunately, we’re using controller-gen anyway, and controller-gen can
generate DeepCopy methods for us: just stick a // +k8s:deepcopy-gen=true on the types that you want it to generate
methods for.
4 - Gateway API Translator Design
The Gateway API translates external resources, e.g. GatewayClass, from the configured Provider to the Intermediate Representation (IR).
Assumptions
Initially target core conformance features only, to be followed by extended conformance features.
Inputs and Outputs
The main inputs to the Gateway API translator are:
- GatewayClass, Gateway, HTTPRoute, TLSRoute, Service, ReferenceGrant, Namespace, and Secret resources.
Note: ReferenceGrant is not fully implemented as of v0.2.
The outputs of the Gateway API translator are:
- Xds and Infra Internal Representations (IRs).
- Status updates for GatewayClass, Gateways, HTTPRoutes
Listener Compatibility
Envoy Gateway follows Gateway API listener compatibility spec:
Each listener in a Gateway must have a unique combination of Hostname, Port, and Protocol. An implementation MAY group Listeners by Port and then collapse each group of Listeners into a single Listener if the implementation determines that the Listeners in the group are “compatible”.
Note: Envoy Gateway does not collapse listeners across multiple Gateways.
Listener Compatibility Examples
Example 1: Gateway with compatible Listeners (same port & protocol, different hostnames)
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: gateway-1
namespace: envoy-gateway
spec:
gatewayClassName: envoy-gateway
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
hostname: "*.envoygateway.io"
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
hostname: whales.envoygateway.io
Example 2: Gateway with compatible Listeners (same port & protocol, one hostname specified, one not)
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: gateway-1
namespace: envoy-gateway
spec:
gatewayClassName: envoy-gateway
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
hostname: "*.envoygateway.io"
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
Example 3: Gateway with incompatible Listeners (same port, protocol and hostname)
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: gateway-1
namespace: envoy-gateway
spec:
gatewayClassName: envoy-gateway
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
hostname: whales.envoygateway.io
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
hostname: whales.envoygateway.io
Example 4: Gateway with incompatible Listeners (neither specify a hostname)
kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
name: gateway-1
namespace: envoy-gateway
spec:
gatewayClassName: envoy-gateway
listeners:
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
- name: http
protocol: HTTP
port: 80
allowedRoutes:
namespaces:
from: All
Computing Status
Gateway API specifies a rich set of status fields & conditions for each resource. To achieve conformance, Envoy Gateway must compute the appropriate status fields and conditions for managed resources.
Status is computed and set for:
- The managed GatewayClass (
gatewayclass.status.conditions). - Each managed Gateway, based on its Listeners’ status (
gateway.status.conditions). For the Kubernetes provider, the Envoy Deployment and Service status are also included to calculate Gateway status. - Listeners for each Gateway (
gateway.status.listeners). - The ParentRef for each Route (
route.status.parents).
The Gateway API translator is responsible for calculating status conditions while translating Gateway API resources to the IR and publishing status over the message bus. The Status Manager subscribes to these status messages and updates the resource status using the configured provider. For example, the Status Manager uses a Kubernetes client to update resource status on the Kubernetes API server.
Outline
The following roughly outlines the translation process. Each step may produce (1) IR; and (2) status updates on Gateway API resources.
Process Gateway Listeners
- Validate unique hostnames, ports, and protocols.
- Validate and compute supported kinds.
- Validate allowed namespaces (validate selector if specified).
- Validate TLS fields if specified, including resolving referenced Secrets.
Process HTTPRoutes
- foreach route rule:
- compute matches
- [core] path exact, path prefix
- [core] header exact
- [extended] query param exact
- [extended] HTTP method
- compute filters
- [core] request header modifier (set/add/remove)
- [core] request redirect (hostname, statuscode)
- [extended] request mirror
- compute backends
- [core] Kubernetes services
- compute matches
- foreach route parent ref:
- get matching listeners (check Gateway, section name, listener validation status, listener allowed routes, hostname intersection)
- foreach matching listener:
- foreach hostname intersection with route:
- add each computed route rule to host
- foreach hostname intersection with route:
- foreach route rule:
Context Structs
To help store, access and manipulate information as it’s processed during the translation process, a set of context structs are used. These structs wrap a given Gateway API type, and add additional fields and methods to support processing.
GatewayContext wraps a Gateway and provides helper methods for setting conditions, accessing Listeners, etc.
type GatewayContext struct {
// The managed Gateway
*v1beta1.Gateway
// A list of Gateway ListenerContexts.
listeners []*ListenerContext
}
ListenerContext wraps a Listener and provides helper methods for setting conditions and other status information on
the associated Gateway.
type ListenerContext struct {
// The Gateway listener.
*v1beta1.Listener
// The Gateway this Listener belongs to.
gateway *v1beta1.Gateway
// An index used for managing this listener in the list of Gateway listeners.
listenerStatusIdx int
// Only Routes in namespaces selected by the selector may be attached
// to the Gateway this listener belongs to.
namespaceSelector labels.Selector
// The TLS Secret for this Listener, if applicable.
tlsSecret *v1.Secret
}
RouteContext represents a generic Route object (HTTPRoute, TLSRoute, etc.) that can reference Gateway objects.
type RouteContext interface {
client.Object
// GetRouteType returns the Kind of the Route object, HTTPRoute,
// TLSRoute, TCPRoute, UDPRoute etc.
GetRouteType() string
// GetHostnames returns the hosts targeted by the Route object.
GetHostnames() []string
// GetParentReferences returns the ParentReference of the Route object.
GetParentReferences() []v1beta1.ParentReference
// GetRouteParentContext returns RouteParentContext by using the Route
// objects' ParentReference.
GetRouteParentContext(forParentRef v1beta1.ParentReference) *RouteParentContext
}
5 - Bootstrap Design
Overview
Issue 31 specifies the need for allowing advanced users to specify their custom Envoy Bootstrap configuration rather than using the default Bootstrap configuration defined in Envoy Gateway. This allows advanced users to extend Envoy Gateway and support their custom use cases such setting up tracing and stats configuration that is not supported by Envoy Gateway.
Goals
- Define an API field to allow a user to specify a custom Bootstrap
- Provide tooling to allow the user to generate the default Bootstrap configuration as well as validate their custom Bootstrap.
Non Goals
- Allow user to configure only a section of the Bootstrap
API
Leverage the existing EnvoyProxy resource which can be attached to the GatewayClass using
the parametersRef field, and define a Bootstrap field within the resource. If this field is set,
the value is used as the Bootstrap configuration for all managed Envoy Proxies created by Envoy Gateway.
// EnvoyProxySpec defines the desired state of EnvoyProxy.
type EnvoyProxySpec struct {
......
// Bootstrap defines the Envoy Bootstrap as a YAML string.
// Visit https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto#envoy-v3-api-msg-config-bootstrap-v3-bootstrap
// to learn more about the syntax.
// If set, this is the Bootstrap configuration used for the managed Envoy Proxy fleet instead of the default Bootstrap configuration
// set by Envoy Gateway.
// Some fields within the Bootstrap that are required to communicate with the xDS Server (Envoy Gateway) and receive xDS resources
// from it are not configurable and will result in the `EnvoyProxy` resource being rejected.
// Backward compatibility across minor versions is not guaranteed.
// We strongly recommend using `egctl x translate` to generate a `EnvoyProxy` resource with the `Bootstrap` field set to the default
// Bootstrap configuration used. You can edit this configuration, and rerun `egctl x translate` to ensure there are no validation errors.
//
// +optional
Bootstrap *string `json:"bootstrap,omitempty"`
}
Tooling
A CLI tool egctl x translate will be provided to the user to help generate a working Bootstrap configuration.
Here is an example where a user inputs a GatewayClass and the CLI generates the EnvoyProxy resource with the Bootstrap field populated.
cat <<EOF | egctl x translate --from gateway-api --to gateway-api -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
name: with-bootstrap-config
---
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: with-bootstrap-config
spec:
bootstrap: |
admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 19000
dynamic_resources:
cds_config:
resource_api_version: V3
api_config_source:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
resource_api_version: V3
api_config_source:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
static_resources:
clusters:
- connect_timeout: 1s
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
typed_extension_protocol_options:
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
"explicit_http_config":
"http2_protocol_options": {}
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: "/sds/xds-certificate.json"
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: "/sds/xds-trusted-ca.json"
resource_api_version: V3
layered_runtime:
layers:
- name: runtime-0
rtds_layer:
rtds_config:
resource_api_version: V3
api_config_source:
transport_api_version: V3
api_type: DELTA_GRPC
grpc_services:
envoy_grpc:
cluster_name: xds_cluster
name: runtime-0
The user can now modify the output, for their use case. Lets say for this example, the user wants to change the admin server port
from 19000 to 18000, they can do so by editing the previous output and running egctl x translate again to see if there any validation
errors. Validation errors should be surfaced in the Status subresource. The internal validator will ensure that the Bootstrap string can be
unmarshalled into the Bootstrap object as well as ensure the user can override certain fields within the Bootstrap configuration such as the
address and tls context within the xds_cluster which are essential for xDS communication between Envoy Gateway and Envoy Proxy.
cat <<EOF | egctl x translate --from gateway-api --to gateway-api -f -
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
name: with-bootstrap-config
---
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: with-bootstrap-config
spec:
bootstrap: |
admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 18000
dynamic_resources:
cds_config:
resource_api_version: V3
api_config_source:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
resource_api_version: V3
api_config_source:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
static_resources:
clusters:
- connect_timeout: 1s
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
typed_extension_protocol_options:
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
"explicit_http_config":
"http2_protocol_options": {}
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: "/sds/xds-certificate.json"
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: "/sds/xds-trusted-ca.json"
resource_api_version: V3
layered_runtime:
layers:
- name: runtime-0
rtds_layer:
rtds_config:
resource_api_version: V3
api_config_source:
transport_api_version: V3
api_type: DELTA_GRPC
grpc_services:
envoy_grpc:
cluster_name: xds_cluster
name: runtime-0
EOF
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
name: with-bootstrap-config
---
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: with-bootstrap-config
spec:
bootstrap: |
admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 18000
dynamic_resources:
cds_config:
resource_api_version: V3
api_config_source:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
resource_api_version: V3
api_config_source:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
static_resources:
clusters:
- connect_timeout: 1s
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
typed_extension_protocol_options:
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
"explicit_http_config":
"http2_protocol_options": {}
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: "/sds/xds-certificate.json"
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: "/sds/xds-trusted-ca.json"
resource_api_version: V3
layered_runtime:
layers:
- name: runtime-0
rtds_layer:
rtds_config:
resource_api_version: V3
api_config_source:
transport_api_version: V3
api_type: DELTA_GRPC
grpc_services:
envoy_grpc:
cluster_name: xds_cluster
name: runtime-0
6 - Configuration API Design
Motivation
Issue 51 specifies the need to design an API for configuring Envoy Gateway. The control plane is configured statically at startup and the data plane is configured dynamically through Kubernetes resources, primarily Gateway API objects. Refer to the Envoy Gateway design doc for additional details regarding Envoy Gateway terminology and configuration.
Goals
- Define an initial API to configure Envoy Gateway at startup.
- Define an initial API for configuring the managed data plane, e.g. Envoy proxies.
Non-Goals
- Implementation of the configuration APIs.
- Define the
statussubresource of the configuration APIs. - Define a complete set of APIs for configuring Envoy Gateway. As stated in the Goals, this document defines the initial configuration APIs.
- Define an API for deploying/provisioning/operating Envoy Gateway. If needed, a future Envoy Gateway operator would be responsible for designing and implementing this type of API.
- Specify tooling for managing the API, e.g. generate protos, CRDs, controller RBAC, etc.
Control Plane API
The EnvoyGateway API defines the control plane configuration, e.g. Envoy Gateway. Key points of this API are:
- It will define Envoy Gateway’s startup configuration file. If the file does not exist, Envoy Gateway will start up with default configuration parameters.
- EnvoyGateway inlines the
TypeMetaAPI. This allows EnvoyGateway to be versioned and managed as a GroupVersionKind scheme. - EnvoyGateway does not contain a metadata field since it’s currently represented as a static configuration file instead of a Kubernetes resource.
- Since EnvoyGateway does not surface status, EnvoyGatewaySpec is inlined.
- If data plane static configuration is required in the future, Envoy Gateway will use a separate file for this purpose.
The v1alpha1 version and config.gateway.envoyproxy.io API group get generated:
// gateway/api/config/v1alpha1/doc.go
// Package v1alpha1 contains API Schema definitions for the config.gateway.envoyproxy.io API group.
//
// +groupName=config.gateway.envoyproxy.io
package v1alpha1
The initial EnvoyGateway API:
// gateway/api/config/v1alpha1/envoygateway.go
package valpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// EnvoyGateway is the Schema for the envoygateways API
type EnvoyGateway struct {
metav1.TypeMeta `json:",inline"`
// EnvoyGatewaySpec defines the desired state of Envoy Gateway.
EnvoyGatewaySpec `json:",inline"`
}
// EnvoyGatewaySpec defines the desired state of Envoy Gateway configuration.
type EnvoyGatewaySpec struct {
// Gateway defines Gateway-API specific configuration. If unset, default
// configuration parameters will apply.
//
// +optional
Gateway *Gateway `json:"gateway,omitempty"`
// Provider defines the desired provider configuration. If unspecified,
// the Kubernetes provider is used with default parameters.
//
// +optional
Provider *EnvoyGatewayProvider `json:"provider,omitempty"`
}
// Gateway defines desired Gateway API configuration of Envoy Gateway.
type Gateway struct {
// ControllerName defines the name of the Gateway API controller. If unspecified,
// defaults to "gateway.envoyproxy.io/gatewayclass-controller". See the following
// for additional details:
//
// https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1alpha2.GatewayClass
//
// +optional
ControllerName string `json:"controllerName,omitempty"`
}
// EnvoyGatewayProvider defines the desired configuration of a provider.
// +union
type EnvoyGatewayProvider struct {
// Type is the type of provider to use. If unset, the Kubernetes provider is used.
//
// +unionDiscriminator
Type ProviderType `json:"type,omitempty"`
// Kubernetes defines the configuration of the Kubernetes provider. Kubernetes
// provides runtime configuration via the Kubernetes API.
//
// +optional
Kubernetes *EnvoyGatewayKubernetesProvider `json:"kubernetes,omitempty"`
// File defines the configuration of the File provider. File provides runtime
// configuration defined by one or more files.
//
// +optional
File *EnvoyGatewayFileProvider `json:"file,omitempty"`
}
// ProviderType defines the types of providers supported by Envoy Gateway.
type ProviderType string
const (
// KubernetesProviderType defines the "Kubernetes" provider.
KubernetesProviderType ProviderType = "Kubernetes"
// FileProviderType defines the "File" provider.
FileProviderType ProviderType = "File"
)
// EnvoyGatewayKubernetesProvider defines configuration for the Kubernetes provider.
type EnvoyGatewayKubernetesProvider struct {
// TODO: Add config as use cases are better understood.
}
// EnvoyGatewayFileProvider defines configuration for the File provider.
type EnvoyGatewayFileProvider struct {
// TODO: Add config as use cases are better understood.
}
Note: Provider-specific configuration is defined in the {$PROVIDER_NAME}Provider API.
Gateway
Gateway defines desired configuration of Gateway API controllers that reconcile and translate Gateway API resources into the Intermediate Representation (IR). Refer to the Envoy Gateway design doc for additional details.
Provider
Provider defines the desired configuration of an Envoy Gateway provider. A provider is an infrastructure component that
Envoy Gateway calls to establish its runtime configuration. Provider is a union type. Therefore, Envoy Gateway
can be configured with only one provider based on the type discriminator field. Refer to the Envoy Gateway
design doc for additional details.
Control Plane Configuration
The configuration file is defined by the EnvoyGateway API type. At startup, Envoy Gateway searches for the configuration at “/etc/envoy-gateway/config.yaml”.
Start Envoy Gateway:
$ ./envoy-gateway
Since the configuration file does not exist, Envoy Gateway will start with default configuration parameters.
The Kubernetes provider can be configured explicitly using provider.kubernetes:
$ cat << EOF > /etc/envoy-gateway/config.yaml
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
provider:
type: Kubernetes
kubernetes: {}
EOF
This configuration will cause Envoy Gateway to use the Kubernetes provider with default configuration parameters.
The Kubernetes provider can be configured using the provider field. For example, the foo field can be set to “bar”:
$ cat << EOF > /etc/envoy-gateway/config.yaml
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
provider:
type: Kubernetes
kubernetes:
foo: bar
EOF
Note: The Provider API from the Kubernetes package is currently undefined and foo: bar is provided for
illustration purposes only.
The same API structure is followed for each supported provider. The following example causes Envoy Gateway to use the File provider:
$ cat << EOF > /etc/envoy-gateway/config.yaml
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
provider:
type: File
file:
foo: bar
EOF
Note: The Provider API from the File package is currently undefined and foo: bar is provided for illustration
purposes only.
Gateway API-related configuration is expressed through the gateway field. If unspecified, Envoy Gateway will use
default configuration parameters for gateway. The following example causes the GatewayClass controller to
manage GatewayClasses with controllerName foo instead of the default gateway.envoyproxy.io/gatewayclass-controller:
$ cat << EOF > /etc/envoy-gateway/config.yaml
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
gateway:
controllerName: foo
With any of the above configuration examples, Envoy Gateway can be started without any additional arguments:
$ ./envoy-gateway
Data Plane API
The data plane is configured dynamically through Kubernetes resources, primarily Gateway API objects.
Optionally, the data plane infrastructure can be configured by referencing a custom resource (CR) through
spec.parametersRef of the managed GatewayClass. The EnvoyProxy API defines the data plane infrastructure
configuration and is represented as the CR referenced by the managed GatewayClass. Key points of this API are:
- If unreferenced by
gatewayclass.spec.parametersRef, default parameters will be used to configure the data plane infrastructure, e.g. expose Envoy network endpoints using a LoadBalancer service. - Envoy Gateway will follow Gateway API recommendations regarding updates to the EnvoyProxy CR:
It is recommended that this resource be used as a template for Gateways. This means that a Gateway is based on the state of the GatewayClass at the time it was created and changes to the GatewayClass or associated parameters are not propagated down to existing Gateways.
The initial EnvoyProxy API:
// gateway/api/config/v1alpha1/envoyproxy.go
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
// EnvoyProxy is the Schema for the envoyproxies API.
type EnvoyProxy struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec EnvoyProxySpec `json:"spec,omitempty"`
Status EnvoyProxyStatus `json:"status,omitempty"`
}
// EnvoyProxySpec defines the desired state of Envoy Proxy infrastructure
// configuration.
type EnvoyProxySpec struct {
// Undefined by this design spec.
}
// EnvoyProxyStatus defines the observed state of EnvoyProxy.
type EnvoyProxyStatus struct {
// Undefined by this design spec.
}
The EnvoyProxySpec and EnvoyProxyStatus fields will be defined in the future as proxy infrastructure configuration use cases are better understood.
Data Plane Configuration
GatewayClass and Gateway resources define the data plane infrastructure. Note that all examples assume Envoy Gateway is running with the Kubernetes provider.
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
name: example-class
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: example-gateway
spec:
gatewayClassName: example-class
listeners:
- name: http
protocol: HTTP
port: 80
Since the GatewayClass does not define spec.parametersRef, the data plane is provisioned using default configuration
parameters. The Envoy proxies will be configured with a http listener and a Kubernetes LoadBalancer service listening
on port 80.
The following example will configure the data plane to use a ClusterIP service instead of the default LoadBalancer service:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
name: example-class
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
name: example-config
group: config.gateway.envoyproxy.io
kind: EnvoyProxy
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
name: example-gateway
spec:
gatewayClassName: example-class
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: example-config
spec:
networkPublishing:
type: ClusterIPService
Note: The NetworkPublishing API is currently undefined and is provided here for illustration purposes only.
7 - egctl Design
Motivation
EG should provide a command line tool with following capabilities:
- Collect configuration from envoy proxy and gateway
- Analyse system configuration to diagnose any issues in envoy gateway
This tool is named egctl.
Syntax
Use the following syntax to run egctl commands from your terminal window:
egctl [command] [entity] [name] [flags]
where command, name, and flags are:
command: Specifies the operation that you want to perform on one or more resources, for exampleconfig,version.entity: Specifies the entity the operation is being performed on such asenvoy-proxyorenvoy-gateway.name: Specifies the name of the specified instance.flags: Specifies optional flags. For example, you can use the-cor--configflags to specify the values for installing.
If you need help, run egctl help from the terminal window.
Operation
The following table includes short descriptions and the general syntax for all the egctl operations:
| Operation | Syntax | Description |
|---|---|---|
version | egctl version | Prints out build version information. |
config | egctl config ENTITY | Retrieve information about proxy configuration from envoy proxy and gateway |
analyze | egctl analyze | Analyze EG configuration and print validation messages |
experimental | egctl experimental | Subcommand for experimental features. These do not guarantee backwards compatibility |
Examples
Use the following set of examples to help you familiarize yourself with running the commonly used egctl operations:
# Retrieve all information about proxy configuration from envoy
egctl config envoy-proxy all <instance_name>
# Retrieve listener information about proxy configuration from envoy
egctl config envoy-proxy listener <instance_name>
# Retrieve information about envoy gateway
egctl config envoy-gateway
8 - Envoy Gateway Extensions Design
As outlined in the official goals for the Envoy Gateway project, one of the main goals is to “provide a common foundation for vendors to build value-added products without having to re-engineer fundamental interactions”. Development of the Envoy Gateway project has been focused on developing the core features for the project and Kubernetes Gateway API conformance. This system focuses on the “common foundation for vendors” component by introducing a way for vendors to extend Envoy Gateway.
To meaningfully extend Envoy Gateway and provide additional features, Extensions need to be able to introduce their own custom resources and have a high level of control over the configuration generated by Envoy Gateway. Simply applying some static xDS configuration patches or relying on the existing Gateway API resources are both insufficient on their own as means to add larger features that require dynamic user-configuration.
As an example, an extension developer may wish to provide their own out-of-the-box authentication filters that require configuration from the end-user. This is a scenario where the ability to introduce custom resources and attach them to HTTPRoutes as an ExtensionRef is necessary. Providing the same feature through a series of xDS patch resources would be too cumbersome for many end-users that want to avoid that level of complexity when managing their clusters.
Goals
- Provide a foundation for extending the Envoy Gateway control plane
- Allow Extension Developers to introduce their own custom resources for extending the Gateway-API via ExtensionRefs, policyAttachments (future) and backendRefs (future).
- Extension developers should NOT have to maintain a custom fork of Envoy Gateway
- Provide a system for extending Envoy Gateway which allows extension projects to ship updates independent of Envoy Gateway’s release schedule
- Modify the generated Envoy xDS config
- Setup a foundation for the initial iteration of Extending Envoy Gateway
- Allow an Extension to hook into the infra manager pipeline (future)
Non-Goals
- The initial design does not capture every hook that Envoy Gateway will eventually support.
- Extend Gateway API Policy Attachments. At some point, these will be addressed using this extension system, but the initial implementation omits these.
- Support multiple extensions at the same time. Due to the fact that extensions will be modifying xDS resources after they are generated, handling the order of extension execution for each individual hook point is a challenge. Additionally, there is no real way to prevent one extension from overwriting or breaking modifications to xDS resources that were made by another extension that was executed first.
Overview
Envoy Gateway can be extended by vendors by means of an extension server developed by the vendor and deployed alongside Envoy Gateway. An extension server can make use of one or more pre/post hooks inside Envoy Gateway before and after its major components (translator, etc.) to allow the extension to modify the data going into or coming out of these components. An extension can be created external to Envoy Gateway as its own Kubernetes deployment or loaded as a sidecar. gRPC is used for the calls between Envoy Gateway and an extension. In the hook call, Envoy Gateway sends data as well as context information to the extension and expects a reply with a modified version of the data that was sent to the extension. Since extensions fundamentally alter the logic and data that Envoy Gateway provides, Extension projects assume responsibility for any bugs and issues they create as a direct result of their modification of Envoy Gateway.
Diagram
Registering Extensions in Envoy Gateway
Information about the extension that Envoy Gateway needs to load is configured in the Envoy Gateway config.
An example configuration:
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
extension:
resources:
- group: example.myextension.io
version: v2
kind: OAuth2Filter
hooks:
post:
- Route
- VirtualHost
- HTTPListener
- Translation
service:
host: my-extension.example
port: 443
tls:
certificateRef:
name: my-secret
namespace: default
An extension must supply connection information in the extension.service field so that Envoy Gateway can communicate with the extension. The tls configuration is optional.
If the extension wants Envoy Gateway to watch resources for it then the extension must configure the optional extension.resources field and supply a list of:
group: the API group of the resourceversion: the API version of the resourcekind: the Kind of resource
The extension can configure the extension.hooks field to specify which hook points it would like to support. If a given hook is not listed here then it will not be executed even
if the extension is configured properly. This allows extension developers to only opt-in to the hook points they want to make use of.
This configuration is required to be provided at bootstrap and modifying the registered extension during runtime is not currently supported.
Envoy Gateway will keep track of the registered extension and its API groups and kinds when processing Gateway API resources.
Extending Gateway API and the Data Plane
Envoy Gateway manages Envoy deployments, which act as the data plane that handles actual user traffic. Users configure the data plane using the K8s Gateway API resources which Envoy Gateway converts into Envoy specific configuration (xDS) to send over to Envoy.
Gateway API offers ExtensionRef filters and Policy Attachments as extension points for implementers to use. Envoy Gateway extends the Gateway API using these extension points to provide support for rate limiting and authentication native to the project. The initial design of Envoy Gateway extensions will primarily focus on ExtensionRef filters so that extension developers can reference their own resources as HTTP Filters in the same way that Envoy Gateway has native support for rate limiting and authentication filters.
When Envoy Gateway encounters an HTTPRoute or GRPCRoute that has an ExtensionRef filter with a group and kind that Envoy Gateway does not support, it will first
check the registered extension to determine if it supports the referenced object before considering it a configuration error.
This allows users to be able to reference additional filters provided by their Envoy Gateway Extension, in their HTTPRoutes / GRPCRoutes:
apiVersion: example.myextension.io/v1alpha1
kind: OAuth2Filter
metadata:
name: oauth2-filter
spec:
...
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: example
spec:
parentRefs:
- name: eg
hostnames:
- www.example.com
rules:
- clientSelectors:
- path:
type: PathPrefix
value: /
filters:
- type: ExtensionRef
extensionRef:
group: example.myextension.io
kind: OAuth2Filter
name: oauth2-filter
backendRefs:
- name: backend
port: 3000
In order to enable the usage of new resources introduced by an extension for translation and xDS modification, Envoy Gateway provides hook points within the translation pipeline, where it calls out to the extension service registered in the EnvoyGateway config
if they specify an group that matches the group of an ExtensionRef filter. The extension will then be able to modify the xDS that Envoy Gateway generated and send back the
modified configuration. If an extension is not registered or if the registered extension does not specify support for the group of an ExtensionRef filter then Envoy Gateway will treat it as an unknown resource
and provide an error to the user.
Note: Currently (as of v1beta1) Gateway API does not provide a means to specify the namespace or version of an object referenced as an ExtensionRef. The extension mechanism will assume that
the namespace of any ExtensionRef is the same as the namespace of the HTTPRoute or GRPCRoute it is attached to rather than treating the name field of an ExtensionRef as a name.namespace string.
If Gateway API adds support for these fields then the design of the Envoy Gateway extensions will be updated to support them.
Watching New Resources
Envoy Gateway will dynamically create new watches on resources introduced by the registered Extension. It does so by using the controller-runtime to create new watches on Unstructured resources that match the versions, groups, and kinds that the
registered extension configured. When communicating with an extension, Envoy Gateway sends these Unstructured resources over to the extension. This eliminates the need for the extension to create its own watches which would have a strong chance of creating race conditions and reconciliation loops when resources change. When an extension receives the Unstructured resources from Envoy Gateway it can perform its own type validation on them. Currently we make the simplifying assumption that the registered extension’s Kinds are filters referenced by extensionRef in HTTPRouteFilters . Support for Policy attachments will be introduced at a later time.
xDS Hooks API
Envoy Gateway supports the following hooks as the initial foundation of the Extension system. Additional hooks can be developed using this extension system at a later point as new use-cases and needs are discovered. The primary iteration of the extension hooks focuses solely on the modification of xDS resources.
Route Modification Hook
The Route level Hook provides a way for extensions to modify a route generated by Envoy Gateway before it is finalized.
Doing so allows extensions to configure/modify route fields configured by Envoy Gateway and also to configure the
Route’s TypedPerFilterConfig which may be desirable to do things such as pass settings and information to ext_authz filters.
The Post Route Modify hook also passes a list of Unstructured data for the externalRefs owned by the extension on the HTTPRoute that created this xDS route
This hook is always executed when an extension is loaded that has added Route to the EnvoyProxy.extensions.hooks.post, and only on Routes which were generated from an HTTPRoute that uses extension resources as externalRef filters.
// PostRouteModifyRequest sends a Route that was generated by Envoy Gateway along with context information to an extension so that the Route can be modified
message PostRouteModifyRequest {
envoy.config.route.v3.Route route = 1;
PostRouteExtensionContext post_route_context = 2;
}
// RouteExtensionContext provides resources introduced by an extension and watched by Envoy Gateway
// additional context information can be added to this message as more use-cases are discovered
message PostRouteExtensionContext {
// Resources introduced by the extension that were used as extensionRefs in an HTTPRoute/GRPCRoute
repeated ExtensionResource extension_resources = 1;
// hostnames are the fully qualified domain names attached to the HTTPRoute
repeated string hostnames = 2;
}
// ExtensionResource stores the data for a K8s API object referenced in an HTTPRouteFilter
// extensionRef. It is constructed from an unstructured.Unstructured marshalled to JSON. An extension
// can marshal the bytes from this resource back into an unstructured.Unstructured and then
// perform type checking to obtain the resource.
message ExtensionResource {
bytes unstructured_bytes = 1;
}
// PostRouteModifyResponse is the expected response from an extension and contains a modified version of the Route that was sent
// If an extension returns a nil Route then it will not be modified
message PostRouteModifyResponse {
envoy.config.route.v3.Route route = 1;
}
VirtualHost Modification Hook
The VirtualHost Hook provides a way for extensions to modify a VirtualHost generated by Envoy Gateway before it is finalized.
An extension can also make use of this hook to generate and insert entirely new Routes not generated by Envoy Gateway.
This hook is always executed when an extension is loaded that has added VirtualHost to the EnvoyProxy.extensions.hooks.post.
An extension may return nil to not make any changes to the VirtualHost.
// PostVirtualHostModifyRequest sends a VirtualHost that was generated by Envoy Gateway along with context information to an extension so that the VirtualHost can be modified
message PostVirtualHostModifyRequest {
envoy.config.route.v3.VirtualHost virtual_host = 1;
PostVirtualHostExtensionContext post_virtual_host_context = 2;
}
// Empty for now but we can add fields to the context as use-cases are discovered without
// breaking any clients that use the API
// additional context information can be added to this message as more use-cases are discovered
message PostVirtualHostExtensionContext {}
// PostVirtualHostModifyResponse is the expected response from an extension and contains a modified version of the VirtualHost that was sent
// If an extension returns a nil Virtual Host then it will not be modified
message PostVirtualHostModifyResponse {
envoy.config.route.v3.VirtualHost virtual_host = 1;
}
HTTP Listener Modification Hook
The HTTP Listener modification hook is the broadest xDS modification Hook available and allows an extension to make changes to a Listener generated by Envoy Gateway before it is finalized.
This hook is always executed when an extension is loaded that has added HTTPListener to the EnvoyProxy.extensions.hooks.post. An extension may return nil
in order to not make any changes to the Listener.
// PostVirtualHostModifyRequest sends a Listener that was generated by Envoy Gateway along with context information to an extension so that the Listener can be modified
message PostHTTPListenerModifyRequest {
envoy.config.listener.v3.Listener listener = 1;
PostHTTPListenerExtensionContext post_listener_context = 2;
}
// Empty for now but we can add fields to the context as use-cases are discovered without
// breaking any clients that use the API
// additional context information can be added to this message as more use-cases are discovered
message PostHTTPListenerExtensionContext {}
// PostHTTPListenerModifyResponse is the expected response from an extension and contains a modified version of the Listener that was sent
// If an extension returns a nil Listener then it will not be modified
message PostHTTPListenerModifyResponse {
envoy.config.listener.v3.Listener listener = 1;
}
Post xDS Translation Modify Hook
The Post Translate Modify hook allows an extension to modify the clusters and secrets in the xDS config.
This allows for inserting clusters that may change along with extension specific configuration to be dynamically created rather than
using custom bootstrap config which would be sufficient for clusters that are static and not prone to have their configurations changed.
An example of how this may be used is to inject a cluster that will be used by an ext_authz http filter created by the extension.
The list of clusters and secrets returned by the extension are used as the final list of all clusters and secrets
This hook is always executed when an extension is loaded that has added Translation to the EnvoyProxy.extensions.hooks.post.
// PostTranslateModifyRequest currently sends only clusters and secrets to an extension.
// The extension is free to add/modify/remove the resources it received.
message PostTranslateModifyRequest {
PostTranslateExtensionContext post_translate_context = 1;
repeated envoy.config.cluster.v3.Cluster clusters = 2;
repeated envoy.extensions.transport_sockets.tls.v3.Secret secrets = 3;
}
// PostTranslateModifyResponse is the expected response from an extension and contains
// the full list of xDS clusters and secrets to be used for the xDS config.
message PostTranslateModifyResponse {
repeated envoy.config.cluster.v3.Cluster clusters = 1;
repeated envoy.extensions.transport_sockets.tls.v3.Secret secrets = 2;
}
Extension Service
Currently, an extension must implement all of the following hooks although it may return the input(s) it received if no modification of the resource is desired. A future expansion of the extension hooks will allow an Extension to specify with config which Hooks it would like to “subscribe” to and which Hooks it does not wish to support. These specific Hooks were chosen in order to provide extensions with the ability to have both broad and specific control over xDS resources and to minimize the amount of data being sent.
service EnvoyGatewayExtension {
rpc PostRouteModify (PostRouteModifyRequest) returns (PostRouteModifyResponse) {};
rpc PostVirtualHostModify(PostVirtualHostModifyRequest) returns (PostVirtualHostModifyResponse) {};
rpc PostHTTPListenerModify(PostHTTPListenerModifyRequest) returns (PostHTTPListenerModifyResponse) {};
rpc PostTranslateModify(PostTranslateModifyRequest) returns (PostTranslateModifyResponse) {};
}
Design Decisions
- Envoy Gateway watches new custom resources introduced by a loaded extension and passes the resources back to the extension when they are used.
- This decision was made to solve the problem about how resources introduced by an extension get watched. If an extension server watches its own resources then it would need some way to trigger an Envoy Gateway reconfigure when a resource that Envoy Gateway is not watching gets updated. Having Envoy Gateway watch all resources removes any concern about creating race confitions or reconcile loops that would result from Envoy Gateway and the extension server both having so much separate state that needs to be synchronized.
- The Extension Server takes ownership of producing the correct xDS configuration in the hook responses
- The Extension Server will be responsible for ensuring the performance of the hook processing time
- The Post xDS level gRPC hooks all currently send a context field even though it contains nothing for several hooks. These fields exist so that they can be updadated in the future to pass additional information to extensions as new use-cases and needs are discovered.
- The initial design supplies the scaffolding for both “pre xDS” and “post xDS” hooks. Only the post hooks are currently implemented which operate on xDS resources after they have been generated. The pre hooks will be implemented at a later date along with one or more hooks in the infra manager. The infra manager level hook(s) will exist to power use-cases such as dynamically creating Deployments/Services for the extension the whenever Envoy Gateway creates an instance of Envoy Proxy. An extension developer might want to take advantage of this functionality to inject a new authorization service as a sidecar on the Envoy Proxy deployment for reduced latency.
- Multiple extensions are not be supported at the same time. Preventing conflict between multiple extensions that are mangling xDS resources is too difficult to ensure compatibility with and is likely to only generate issues.
Known Challenges
Extending Envoy Gateway by using an external extension server which makes use of hook points in Envoy Gateway does comes with a few trade-offs. One known trade-off is the impact of the time that it takes for the hook calls to be executed. Since an extension would make use of hook points in Envoy Gateway that use gRPC for communication, the time it takes to perform these requests could become a concern for some extension developers. One way to minimize the request time of the hook calls is to load the extension server as a sidecar to Envoy Gateway to minimize the impact of networking on the hook calls.
9 - Rate Limit Design
Overview
Rate limit is a feature that allows the user to limit the number of incoming requests to a predefined value based on attributes within the traffic flow.
Here are some reasons why a user may want to implements Rate limits
- To prevent malicious activity such as DDoS attacks.
- To prevent applications and its resources (such as a database) from getting overloaded.
- To create API limits based on user entitlements.
Scope Types
The rate limit type here describes the scope of rate limits.
Global - In this case, the rate limit is common across all the instances of Envoy proxies where its applied i.e. if the data plane has 2 replicas of Envoy running, and the rate limit is 10 requests/second, this limit is common and will be hit if 5 requests pass through the first replica and 5 requests pass through the second replica within the same second.
Local - In this case, the rate limits are specific to each instance/replica of Envoy running. Note - This is not part of the initial design and will be added as a future enhancement.
Match Types
Rate limit a specific traffic flow
- Here is an example of a ratelimit implemented by the application developer to limit a specific user
by matching on a custom
x-user-idheader with a value set toone
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: RateLimitFilter
metadata:
name: ratelimit-specific-user
spec:
type: Global
global:
rules:
- clientSelectors:
- headers:
- name: x-user-id
value: one
limit:
requests: 10
unit: Hour
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: example
spec:
parentRefs:
- name: eg
hostnames:
- www.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /foo
filters:
- type: ExtensionRef
extensionRef:
group: gateway.envoyproxy.io
kind: RateLimitFilter
name: ratelimit-specific-user
backendRefs:
- name: backend
port: 3000
Rate limit all traffic flows
- Here is an example of a rate limit implemented by the application developer that limits the total requests made
to a specific route to safeguard health of internal application components. In this case, no specific
headersmatch is specified, and the rate limit is applied to all traffic flows accepted by thisHTTPRoute.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: RateLimitFilter
metadata:
name: ratelimit-all-requests
spec:
type: Global
global:
rules:
- limit:
requests: 1000
unit: Second
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: example
spec:
parentRefs:
- name: eg
hostnames:
- www.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /foo
filters:
- type: ExtensionRef
extensionRef:
group: gateway.envoyproxy.io
kind: RateLimitFilter
name: ratelimit-all-requests
backendRefs:
- name: backend
port: 3000
Rate limit per distinct value
- Here is an example of a rate limit implemented by the application developer to limit any unique user
by matching on a custom
x-user-idheader. Here, user A (recognised from the traffic flow using the headerx-user-idand valuea) will be rate limited at 10 requests/hour and so will user B (recognised from the traffic flow using the headerx-user-idand valueb).
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: RateLimitFilter
metadata:
name: ratelimit-per-user
spec:
type: Global
global:
rules:
- clientSelectors:
- headers:
- type: Distinct
name: x-user-id
limit:
requests: 10
unit: Hour
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: example
spec:
parentRefs:
- name: eg
hostnames:
- www.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /foo
filters:
- type: ExtensionRef
extensionRef:
group: gateway.envoyproxy.io
kind: RateLimitFilter
name: ratelimit-per-user
backendRefs:
- name: backend
port: 3000
Rate limit per source IP
- Here is an example of a rate limit implemented by the application developer that limits the total requests made
to a specific route by matching on source IP. In this case, requests from
x.x.x.xwill be rate limited at 10 requests/hour.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: RateLimitFilter
metadata:
name: ratelimit-per-ip
spec:
type: Global
global:
rules:
- clientSelectors:
- sourceIP: x.x.x.x/32
limit:
requests: 10
unit: Hour
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: example
spec:
parentRefs:
- name: eg
hostnames:
- www.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /foo
filters:
- type: ExtensionRef
extensionRef:
group: gateway.envoyproxy.io
kind: RateLimitFilter
name: ratelimit-per-user
backendRefs:
- name: backend
port: 3000
Multiple RateLimitFilters, rules and clientSelectors
- Users can create multiple
RateLimitFilters and apply it to the sameHTTPRoute. In such a case eachRateLimitFilterwill be applied to the route and matched (and limited) in a mutually exclusive way, independent of each other. - Rate limits are applied for each
RateLimitFilterrulewhen ALL the conditions underclientSelectorshold true.
Here’s an example highlighting this -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: RateLimitFilter
metadata:
name: ratelimit-all-safeguard-app
spec:
type: Global
global:
rules:
- limit:
requests: 100
unit: Second
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: RateLimitFilter
metadata:
name: ratelimit-per-user
spec:
type: Global
global:
rules:
- clientSelectors:
- headers:
- type: Distinct
name: x-user-id
limit:
requests: 1000
unit: Hour
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: example
spec:
parentRefs:
- name: eg
hostnames:
- www.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /foo
filters:
- type: ExtensionRef
extensionRef:
group: gateway.envoyproxy.io
kind: RateLimitFilter
name: ratelimit-per-user
- type: ExtensionRef
extensionRef:
group: gateway.envoyproxy.io
kind: RateLimitFilter
name: ratelimit-all-safeguard-app
backendRefs:
- name: backend
port: 3000
- The user has created two
RateLimitFilters and has attached it to aHTTPRoute- one(ratelimit-all-safeguard-app) to ensure that the backend does not get overwhelmed with requests, any excess requests are rate limited irrespective of the attributes within the traffic flow, and another(ratelimit-per-user) to rate limit each distinct user client who can be differentiated using thex-user-idheader, to ensure that each client does not make exessive requests to the backend. - If user
baz(identified with the header and value ofx-user-id: baz) sends 90 requests within the first second, and userbarsends 11 more requests during that same interval of 1 second, and userbarsends the 101th request within that second, the rule defined inratelimit-all-safeguard-appgets activated and Envoy Gateway will ratelimit the request sent bybar(and any other request sent within that 1 second). After 1 second, the rate limit counter associated with theratelimit-all-safeguard-apprule is reset and again evaluated. - If user
baralso ends up sending 90 more requests within the hour, summing upbar’s total request count to 101, the rate limit rule defined withinratelimit-per-userwill get activated, andbar’s requests will be rate limited again until the hour interval ends. - Within the same above hour, if
bazsends 991 more requests, summing upbaz’s total request count to 1001, the rate limit rule defined withinratelimit-per-userwill get activated forbaz, andbaz’s requests will also be rate limited until the hour interval ends.
Design Decisions
- The initial design uses an Extension filter to apply the Rate Limit functionality on a specific HTTPRoute. This was preferred over the PolicyAttachment extension mechanism, because it is unclear whether Rate Limit will be required to be enforced or overridden by the platform administrator or not.
- The RateLimitFilter can only be applied as a filter to a [HTTPRouteRule[], applying it across all backends within a HTTPRoute and cannot be applied a filter within a HTTPBackendRef for a specific backend.
- The HTTPRoute API has a matches field within each rule to select a specific traffic flow to be routed to
the destination backend. The RateLimitFilter API that can be attached to an HTTPRoute via an extensionRef filter,
also has a
clientSelectorsfield within eachruleto select attributes within the traffic flow to rate limit specific clients. The two levels of selectors/matches allow for flexibility and aim to hold match information specific to its use, allowing the author/owner of each configuration to be different. It also allows theclientSelectorsfield within the RateLimitFilter to be enhanced with other matchable attribute such as IP subnet in the future that are not relevant in the HTTPRoute API.
Implementation Details
Global Rate limiting
- Global rate limiting in Envoy Proxy can be achieved using the following -
- Actions can be configured per xDS Route.
- If the match criteria defined within these actions is met for a specific HTTP Request, a set of key value pairs called descriptors defined within the above actions is sent to a remote rate limit service, whose configuration (such as the URL for the rate limit service) is defined using a rate limit filter.
- Based on information received by the rate limit service and its programmed configuration, a decision is computed, whether to rate limit the HTTP Request or not, and is sent back to Envoy, which enforces this decision on the data plane.
- Envoy Gateway will leverage this Envoy Proxy feature by -
- Translating the user facing RateLimitFilter API into Rate limit Actions as well as Rate limit service configuration to implement the desired API intent.
- Envoy Gateway will use the existing reference implementation of the rate limit service.
- The Infrastructure administrator will need to enable the rate limit service using new settings that will be defined in the EnvoyGateway config API.
- The xDS IR will be enhanced to hold the user facing rate limit intent.
- The xDS Translator will be enhanced to translate the rate limit field within the xDS IR into Rate limit Actions as well as instantiate the rate limit filter.
- A new runner called
rate-limitwill be added that subscribes to the xDS IR messages and translates it into a new Rate Limit Infra IR which contains the rate limit service configuration as well as other information needed to deploy the rate limit service. - The infrastructure service will be enhanced to subscribe to the Rate Limit Infra IR and deploy a provider specific rate limit service runnable entity.
- A Status field within the RateLimitFilter API will be added to reflect whether the specific configuration was programmed correctly in these multiple locations or not.
10 - Request Authentication Design
Overview
Issue 336 specifies the need for exposing a user-facing API to configure request authentication. Request
authentication is defined as an authentication mechanism to be enforced by Envoy on a per-request basis. A connection
will be rejected if it contains invalid authentication information, based on the AuthenticationFilter API type
proposed in this design document.
Envoy Gateway leverages Gateway API for configuring managed Envoy proxies. Gateway API defines core, extended, and
implementation-specific API support levels for implementers such as Envoy Gateway to expose features. Since
implementing request authentication is not covered by Core or Extended APIs, an Implementation-specific API will
be created for this purpose.
Goals
- Define an API for configuring request authentication.
- Implement JWT as the first supported authentication type.
- Allow users that manage routes, e.g. HTTPRoute, to authenticate matching requests before forwarding to a backend service.
- Support HTTPRoutes as an Authentication API referent. HTTPRoute provides multiple extension points. The HTTPRouteFilter is the extension point supported by the Authentication API.
Non-Goals
- Allow infrastructure administrators to override or establish default authentication policies.
- Support referents other than HTTPRoute.
- Support Gateway API extension points other than HTTPRouteFilter.
Use-Cases
These use-cases are presented as an aid for how users may attempt to utilize the outputs of the design. They are not an exhaustive list of features for authentication support in Envoy Gateway.
As a Service Producer, I need the ability to:
- Authenticate a request before forwarding it to a backend service.
- Have different authentication mechanisms per route rule.
- Choose from different authentication mechanisms supported by Envoy, e.g. OIDC.
Authentication API Type
The Authentication API type defines authentication configuration for authenticating requests through managed Envoy proxies.
package v1alpha1
import (
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)
type AuthenticationFilter struct {
metav1.TypeMeta
metav1.ObjectMeta
// Spec defines the desired state of the Authentication type.
Spec AuthenticationFilterSpec
// Note: The status sub-resource has been excluded but may be added in the future.
}
// AuthenticationFilterSpec defines the desired state of the AuthenticationFilter type.
// +union
type AuthenticationFilterSpec struct {
// Type defines the type of authentication provider to use. Supported provider types are:
//
// * JWT: A provider that uses JSON Web Token (JWT) for authenticating requests.
//
// +unionDiscriminator
Type AuthenticationFilterType
// JWT defines the JSON Web Token (JWT) authentication provider type. When multiple
// jwtProviders are specified, the JWT is considered valid if any of the providers
// successfully validate the JWT.
JwtProviders []JwtAuthenticationFilterProvider
}
...
Refer to PR 773 for the detailed AuthenticationFilter API spec.
The status subresource is not included in the AuthenticationFilter API. Status will be surfaced by an HTTPRoute that
references an AuthenticationFilter. For example, an HTTPRoute will surface the ResolvedRefs=False status condition if it
references an AuthenticationFilter that does not exist. It may be beneficial to add AuthenticationFilter status fields in the future
based on defined use-cases. For example, a remote JWKS can be validated based on the specified URI and have an
appropriate status condition surfaced.
AuthenticationFilter Example
The following is an AuthenticationFilter example with one JWT authentication provider:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: AuthenticationFilter
metadata:
name: example
spec:
type: JWT
jwtProviders:
- name: example
issuer: https://www.example.com
audiences:
- foo.com
remoteJwks:
uri: https://foo.com/jwt/public-key/jwks.json
<TBD>
Note: type is a union type, allowing only one of any supported provider type such as jwtProviders to be
specified.
The following is an example HTTPRoute configured to use the above JWT authentication provider:
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: example
spec:
parentRefs:
- name: eg
hostnames:
- www.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /foo
filters:
- type: ExtensionRef
extensionRef:
group: gateway.envoyproxy.io
kind: AuthenticationFilter
name: example
backendRefs:
- name: backend
port: 3000
Requests for www.example.com/foo will be authenticated using the referenced JWT provider before being forwarded to the
backend service named “backend”.
Implementation Details
The JWT authentication type is translated to an Envoy JWT authentication filter and a cluster is created for each remote JWKS. The following examples provide additional details on how Gateway API and AuthenticationFilter resources are translated into Envoy configuration.
Example 1: One Route with One JWT Provider
The following cluster is created from the above HTTPRoute and AuthenticationFilter:
dynamic_clusters:
- name: foo.com|443
load_assignment:
cluster_name: foo.com|443
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: foo.com
port_value: 443
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
sni: foo.com
common_tls_context:
validation_context:
match_subject_alt_names:
- exact: "*.foo.com"
trusted_ca:
filename: /etc/ssl/certs/ca-certificates.crt
A JWT authentication HTTP filter is added to the HTTP Connection Manager. For example:
dynamic_resources:
dynamic_listeners:
- name: example_listener
address:
socket_address:
address: 1.2.3.4
port_value: 80
filter_chains:
- filters:
- name: envoy.http_connection_manager
http_filters:
- name: envoy.filters.http.jwt_authn
typed_config:
"@type": type.googleapis.com/envoy.config.filter.http.jwt_authn.v2alpha.JwtAuthentication
This JWT authentication HTTP filter contains two fields:
- The
providersfield specifies how a JWT should be verified, such as where to extract the token, where to fetch the public key (JWKS) and where to output its payload. This field is built from the source resourcenamespace-name, and the JWT provider name of an AuthenticationFilter. - The
rulesfield specifies matching rules and their requirements. If a request matches a rule, its requirement applies. The requirement specifies which JWT providers should be used. This field is built from a HTTPRoutematchesrule that references the AuthenticationFilter. When a referenced Authentication specifies multiplejwtProviders, the JWT is considered valid if any of the providers successfully validate the JWT.
The following JWT authentication HTTP filter providers configuration is created from the above AuthenticationFilter.
providers:
example:
issuer: https://www.example.com
audiences:
- foo.com
remote_jwks:
http_uri:
uri: https://foo.com/jwt/public-key/jwks.json
cluster: example_jwks_cluster
timeout: 1s
The following JWT authentication HTTP filter rules configuration is created from the above HTTPRoute.
rules:
- match:
prefix: /foo
requires:
provider_name: default-example-example
Example 2: Two HTTPRoutes with Different AuthenticationFilters
The following example contains:
- Two HTTPRoutes with different hostnames.
- Each HTTPRoute references a different AuthenticationFilter.
- Each AuthenticationFilter contains a different JWT provider.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: AuthenticationFilter
metadata:
name: example1
spec:
type: JWT
jwtProviders:
- name: example1
issuer: https://www.example1.com
audiences:
- foo.com
remoteJwks:
uri: https://foo.com/jwt/public-key/jwks.json
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: AuthenticationFilter
metadata:
name: example2
spec:
type: JWT
jwtProviders:
- name: example2
issuer: https://www.example2.com
audiences:
- bar.com
remoteJwks:
uri: https://bar.com/jwt/public-key/jwks.json
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: example1
spec:
hostnames:
- www.example1.com
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: eg
rules:
- matches:
- path:
type: PathPrefix
value: /foo
filters:
- type: ExtensionRef
extensionRef:
group: gateway.envoyproxy.io
kind: AuthenticationFilter
name: example1
backendRefs:
- name: backend
port: 3000
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
name: example2
spec:
hostnames:
- www.example2.com
parentRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: eg
rules:
- matches:
- path:
type: PathPrefix
value: /bar
filters:
- type: ExtensionRef
extensionRef:
group: gateway.envoyproxy.io
kind: AuthenticationFilter
name: example2
backendRefs:
- name: backend2
port: 3000
The following xDS configuration is created from the above example resources:
configs:
...
dynamic_listeners:
- name: default-eg-http
...
default_filter_chain:
filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
'@type': >-
type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: http
rds:
config_source:
...
route_config_name: default-eg-http
http_filters:
- name: envoy.filters.http.jwt_authn
typed_config:
'@type': >-
type.googleapis.com/envoy.config.filter.http.jwt_authn.v2alpha.JwtAuthentication
providers:
default-example1-example1:
issuer: https://www.example1.com
audiences:
- foo.com
remote_jwks:
http_uri:
uri: https://foo.com/jwt/public-key/jwks.json
cluster: default-example1-example1-jwt
default-example2-example2:
issuer: https://www.example2.com
audiences:
- bar.com
remote_jwks:
http_uri:
uri: https://bar.com/jwt/public-key/jwks.json
cluster: default-example2-example2-jwt
rules:
- match:
exact: /foo
requires:
provider_name: default-example1-example1
- match:
exact: /bar
requires:
provider_name: default-example2-example2
- name: envoy.filters.http.router
typed_config:
'@type': >-
type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
dynamic_route_configs:
- route_config:
'@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
name: default-eg-http
virtual_hosts:
- name: default-eg-http
domains:
- '*'
routes:
- match:
prefix: /foo
headers:
- name: ':authority'
string_match:
exact: www.example1.com
route:
cluster: default-backend-rule-0-match-0-www.example1.com
- match:
prefix: /bar
headers:
- name: ':authority'
string_match:
exact: www.example2.com
route:
cluster: default-backend2-rule-0-match-0-www.example2.com
dynamic_active_clusters:
- cluster:
name: default-backend-rule-0-match-0-www.example.com
...
endpoints:
- locality: {}
lb_endpoints:
- endpoint:
address:
socket_address:
address: $BACKEND_SERVICE1_IP
port_value: 3000
- cluster:
'@type': type.googleapis.com/envoy.config.cluster.v3.Cluster
name: default-backend-rule-1-match-0-www.example.com
...
endpoints:
- locality: {}
lb_endpoints:
- endpoint:
address:
socket_address:
address: $BACKEND_SERVICE2_IP
port_value: 3000
...
Note: The JWT provider cluster and route is omitted from the above example for brevity.
Implementation Outline
- Update the Kubernetes provider to get/watch AuthenticationFilter resources that are referenced by managed HTTPRoutes. Add the referenced AuthenticationFilter object to the resource map and publish it.
- Update the resource translator to include the AuthenticationFilter API in HTTPRoute processing.
- Update the xDS translator to translate an AuthenticationFilter into xDS resources. The translator should perform the
following:
- Convert a list of JWT rules from the xds IR into an Envoy JWT filter config.
- Create a JWT authentication HTTP filter.
- Build the HTTP Connection Manager (HCM) HTTP filters.
- Build the HCM.
- When building the Listener, create an HCM for each filter-chain.
Adding Authentication Types
Additional authentication types can be added in the future through the AuthenticationFilterType API. For
example, to add the Foo authentication type:
Define the Foo authentication provider:
package v1alpha1
// FooAuthenticationFilterProvider defines the "Foo" authentication filter provider type.
type FooAuthenticationFilterProvider struct {
// TODO: Define fields of the Foo authentication filter provider type.
}
Add the FooAuthenticationFilterProvider type to AuthenticationFilterSpec:
package v1alpha1
type AuthenticationFilterSpec struct {
...
// Foo defines the Foo authentication type. For additional
// details, see:
//
// <INSERT_LINK>
//
// +optional
Foo *FooAuthenticationFilterProvider
}
Lastly, add the type to the AuthenticationType enum:
// AuthenticationType is a type of authentication provider.
// +kubebuilder:validation:Enum=JWT,FOO
type AuthenticationFilterType string
const (
// JwtAuthenticationProviderType is the JWT authentication provider type.
FooAuthenticationFilterProviderType AuthenticationFilterType = "FOO"
)
The AuthenticationFilter API should support additional authentication types in the future, for example:
- OAuth2
- OIDC
Outstanding Questions
- If Envoy Gateway owns the AuthenticationFilter API, is an xDS IR equivalent needed?
- Should local JWKS be implemented before remote JWKS?
- How should Envoy obtain the trusted CA for a remote JWKS?
- Should HTTPS be the only supported scheme for remote JWKS?
- Should OR’ing JWT providers be supported?
- Should Authentication provide status?
- Are the API field validation rules acceptable?
11 - TCP and UDP Proxy Design
Even though most of the use cases for Envoy Gateway are at Layer-7, Envoy Gateway can also work at Layer-4 to proxy TCP and UDP traffic. This document will explore the options we have when operating Envoy Gateway at Layer-4 and explain the design decision.
Envoy can work as a non-transparent proxy or a transparent proxy for both TCP and UDP , so ideally, Envoy Gateway should also be able to work in these two modes:
Non-transparent Proxy Mode
For TCP, Envoy terminates the downstream connection, connects the upstream with its own IP address, and proxies the TCP traffic from the downstream to the upstream.
For UDP, Envoy receives UDP datagrams from the downstream, and uses its own IP address as the sender IP address when proxying the UDP datagrams to the upstream.
In this mode, the upstream will see Envoy’s IP address and port.
Transparent Proxy Mode
For TCP, Envoy terminates the downstream connection, connects the upstream with the downstream IP address, and proxies the TCP traffic from the downstream to the upstream.
For UDP, Envoy receives UDP datagrams from the downstream, and uses the downstream IP address as the sender IP address when proxying the UDP datagrams to the upstream.
In this mode, the upstream will see the original downstream IP address and Envoy’s mac address.
Note: Even in transparent mode, the upstream can’t see the port number of the downstream because Envoy doesn’t forward the port number.
The Implications of Transparent Proxy Mode
Escalated Privilege
Envoy needs to bind to the downstream IP when connecting to the upstream, which means Envoy requires escalated CAP_NET_ADMIN privileges. This is often considered as a bad security practice and not allowed in some sensitive deployments.
Routing
The upstream can see the original source IP, but the original port number won’t be passed, so the return traffic from the upstream must be routed back to Envoy because only Envoy knows how to send the return traffic back to the right port number of the downstream, which requires routing at the upstream side to be set up. In a Kubernetes cluster, Envoy Gateway will have to carefully cooperate with CNI plugins to get the routing right.
The Design Decision (For Now)
The implementation will only support proxying in non-transparent mode i.e. the backend will see the source IP and port of the deployed Envoy instance instead of the client.