This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Design

This section includes Designs of Envoy Gateway.

1: Goals
2: System Design
3: Watching Components Design
4: Gateway API Translator Design
5: Configuration API Design

1 - Goals

The high-level goal of the Envoy Gateway project is to attract more users to Envoy by lowering barriers to adoption through expressive, extensible, role-oriented APIs that support a multitude of ingress and L7/L4 traffic routing use cases; and provide a common foundation for vendors to build value-added products without having to re-engineer fundamental interactions.

Objectives

Expressive API

The Envoy Gateway project will expose a simple and expressive API, with defaults set for many capabilities.

The API will be the Kubernetes-native Gateway API, plus Envoy-specific extensions and extension points. This expressive and familiar API will make Envoy accessible to more users, especially application developers, and make Envoy a stronger option for “getting started” as compared to other proxies. Application developers will use the API out of the box without needing to understand in-depth concepts of Envoy Proxy or use OSS wrappers. The API will use familiar nouns that users understand.

The core full-featured Envoy xDS APIs will remain available for those who need more capability and for those who add functionality on top of Envoy Gateway, such as commercial API gateway products.

This expressive API will not be implemented by Envoy Proxy, but rather an officially supported translation layer on top.

Batteries included

Envoy Gateway will simplify how Envoy is deployed and managed, allowing application developers to focus on delivering core business value.

The project plans to include additional infrastructure components required by users to fulfill their Ingress and API gateway needs: It will handle Envoy infrastructure provisioning (e.g. Kubernetes Service, Deployment, et cetera), and possibly infrastructure provisioning of related sidecar services. It will include sensible defaults with the ability to override. It will include channels for improving ops by exposing status through API conditions and Kubernetes status sub-resources.

Making an application accessible needs to be a trivial task for any developer. Similarly, infrastructure administrators will enjoy a simplified management model that doesn’t require extensive knowledge of the solution’s architecture to operate.

All environments

Envoy Gateway will support running natively in Kubernetes environments as well as non-Kubernetes deployments.

Initially, Kubernetes will receive the most focus, with the aim of having Envoy Gateway become the de facto standard for Kubernetes ingress supporting the Gateway API. Additional goals include multi-cluster support and various runtime environments.

Extensibility

Vendors will have the ability to provide value-added products built on the Envoy Gateway foundation.

It will remain easy for end-users to leverage common Envoy Proxy extension points such as providing an implementation for authentication methods and rate-limiting. For advanced use cases, users will have the ability to use the full power of xDS.

Since a general-purpose API cannot address all use cases, Envoy Gateway will provide additional extension points for flexibility. As such, Envoy Gateway will form the base of vendor-provided managed control plane solutions, allowing vendors to shift to a higher management plane layer.

Non-objectives

Cannibalize vendor models

Vendors need to have the ability to drive commercial value, so the goal is not to cannibalize any existing vendor monetization model, though some vendors may be affected by it.

Disrupt current Envoy usage patterns

Envoy Gateway is purely an additive convenience layer and is not meant to disrupt any usage pattern of any user with Envoy Proxy, xDS, or go-control-plane.

Personas

In order of priority

1. Application developer

The application developer spends the majority of their time developing business logic code. They require the ability to manage access to their application.

2. Infrastructure administrators

The infrastructure administrators are responsible for the installation, maintenance, and operation of API gateways appliances in infrastructure, such as CRDs, roles, service accounts, certificates, etc. Infrastructure administrators support the needs of application developers by managing instances of Envoy Gateway.

2 - System Design

Goals

Define the system components needed to satisfy the requirements of Envoy Gateway.

Non-Goals

Create a detailed design and interface specification for each system component.

Terminology

Control Plane- A collection of inter-related software components for providing application gateway and routing functionality. The control plane is implemented by Envoy Gateway and provides services for managing the data plane. These services are detailed in the components section.
Data Plane- Provides intelligent application-level traffic routing and is implemented as one or more Envoy proxies.

Architecture

Configuration

Envoy Gateway is configured statically at startup and the managed data plane is configured dynamically through Kubernetes resources, primarily Gateway API objects.

Static Configuration

Static configuration is used to configure Envoy Gateway at startup, i.e. change the GatewayClass controllerName, configure a Provider, etc. Currently, Envoy Gateway only supports configuration through a configuration file. If the configuration file is not provided, Envoy Gateway starts-up with default configuration parameters.

Dynamic Configuration

Dynamic configuration is based on the concept of a declaring the desired state of the data plane and using reconciliation loops to drive the actual state toward the desired state. The desired state of the data plane is defined as Kubernetes resources that provide the following services:

Infrastructure Management- Manage the data plane infrastructure, i.e. deploy, upgrade, etc. This configuration is expressed through GatewayClass and Gateway resources. The EnvoyProxy Custom Resource can be referenced by gatewayclass.spec.parametersRef to modify data plane infrastructure default parameters, e.g. expose Envoy network endpoints using a NodePort service instead of a LoadBalancer service.
Traffic Routing- Define how to handle application-level requests to backend services. For example, route all HTTP requests for “www.example.com” to a backend service running a web server. This configuration is expressed through HTTPRoute and TLSRoute resources that match, filter, and route traffic to a backend. Although a backend can be any valid Kubernetes Group/Kind resource, Envoy Gateway only supports a Service reference.

Components

Envoy Gateway is made up of several components that communicate in-process; how this communication happens is described in the Watching Components Design.

Provider

A Provider is an infrastructure component that Envoy Gateway calls to establish its runtime configuration, resolve services, persist data, etc. As of v0.2, Kubernetes is the only implemented provider. A file provider is on the roadmap via Issue #37. Other providers can be added in the future as Envoy Gateway use cases are better understood. A provider is configured at start up through Envoy Gateway’s static configuration.

Kubernetes Provider

Uses Kubernetes-style controllers to reconcile Kubernetes resources that comprise the dynamic configuration.
Manages the data plane through Kubernetes API CRUD operations.
Uses Kubernetes for Service discovery.
Uses etcd (via Kubernetes API) to persist data.

File Provider

Uses a file watcher to watch files in a directory that define the data plane configuration.
Manages the data plane by calling internal APIs, e.g. CreateDataPlane().
Uses the host’s DNS for Service discovery.
If needed, the local filesystem is used to persist data.

Resource Watcher

The Resource Watcher watches resources used to establish and maintain Envoy Gateway’s dynamic configuration. The mechanics for watching resources is provider-specific, e.g. informers, caches, etc. are used for the Kubernetes provider. The Resource Watcher uses the configured provider for input and provides resources to the Resource Translator as output.

Resource Translator

The Resource Translator translates external resources, e.g. GatewayClass, from the Resource Watcher to the Intermediate Representation (IR). It is responsible for:

Translating infrastructure-specific resources/fields from the Resource Watcher to the Infra IR.
Translating proxy configuration resources/fields from the Resource Watcher to the xDS IR.

Note: The Resource Translator is implemented as the Translator API type in the gatewayapi package.

Intermediate Representation (IR)

The Intermediate Representation defines internal data models that external resources are translated into. This allows Envoy Gateway to be decoupled from the external resources used for dynamic configuration. The IR consists of an Infra IR used as input for the Infra Manager and an xDS IR used as input for the xDS Translator.

Infra IR- Used as the internal definition of the managed data plane infrastructure.
xDS IR- Used as the internal definition of the managed data plane xDS configuration.

xDS Translator

The xDS Translator translates the xDS IR into xDS Resources that are consumed by the xDS server.

xDS Server

The xDS Server is a xDS gRPC Server based on Go Control Plane. Go Control Plane implements the Delta xDS Server Protocol and is responsible for using xDS to configure the data plane.

Infra Manager

The Infra Manager is a provider-specific component responsible for managing the following infrastructure:

Data Plane - Manages all the infrastructure required to run the managed Envoy proxies. For example, CRUD Deployment, Service, etc. resources to run Envoy in a Kubernetes cluster.
Auxiliary Control Planes - Optional infrastructure needed to implement application Gateway features that require external integrations with the managed Envoy proxies. For example, Global Rate Limiting requires provisioning and configuring the Envoy Rate Limit Service and the Rate Limit filter. Such features are exposed to users through the Custom Route Filters extension.

The Infra Manager consumes the Infra IR as input to manage the data plane infrastructure.

Design Decisions

Envoy Gateway consumes one GatewayClass by comparing its configured controller name with spec.controllerName of a GatewayClass. If multiple GatewayClasses exist with the same spec.controllerName, Envoy Gateway follows Gateway API guidelines to resolve the conflict. gatewayclass.spec.parametersRef refers to the EnvoyProxy custom resource for configuring the managed proxy infrastructure. If unspecified, default configuration parameters are used for the managed proxy infrastructure.
Envoy Gateway manages Gateways that reference its GatewayClass.
- A Gateway resource causes Envoy Gateway to provision managed Envoy proxy infrastructure.
- Envoy Gateway groups Listeners by Port and collapses each group of Listeners into a single Listener if the Listeners in the group are compatible. Envoy Gateway considers Listeners to be compatible if all the following conditions are met:
  - Either each Listener within the group specifies the “HTTP” Protocol or each Listener within the group specifies either the “HTTPS” or “TLS” Protocol.
  - Each Listener within the group specifies a unique “Hostname”.
  - As a special case, one Listener within a group may omit “Hostname”, in which case this Listener matches when no other Listener matches.
- Envoy Gateway does not merge listeners across multiple Gateways.
Envoy Gateway follows Gateway API guidelines to resolve any conflicts.
- A Gateway listener corresponds to an Envoy proxy Listener.
An HTTPRoute resource corresponds to an Envoy proxy Route.
- Each backendRef corresponds to an Envoy proxy Cluster.
The goal is to make Envoy Gateway components extensible in the future. See the roadmap for additional details.

The draft for this document is here.

3 - Watching Components Design

Envoy Gateway is made up of several components that communicate in-process. Some of them (namely Providers) watch external resources, and “publish” what they see for other components to consume; others watch what another publishes and act on it (such as the resource translator watches what the providers publish, and then publishes its own results that are watched by another component). Some of these internally published results are consumed by multiple components.

To facilitate this communication use the watchable library. The watchable.Map type is very similar to the standard library’s sync.Map type, but supports a .Subscribe (and .SubscribeSubset) method that promotes a pub/sub pattern.

Pub

Many of the things we communicate around are naturally named, either by a bare “name” string or by a “name”/“namespace” tuple. And because watchable.Map is typed, it makes sense to have one map for each type of thing (very similar to if we were using native Go maps). For example, a struct that might be written to by the Kubernetes provider, and read by the IR translator:

type ResourceTable struct {
    // gateway classes are cluster-scoped; no namespace
    GatewayClasses watchable.Map[string, *gwapiv1b1.GatewayClass]

    // gateways are namespace-scoped, so use a k8s.io/apimachinery/pkg/types.NamespacedName as the map key.
    Gateways watchable.Map[types.NamespacedName, *gwapiv1b1.Gateway]

    HTTPRoutes watchable.Map[types.NamespacedName, *gwapiv1b1.HTTPRoute]
}

The Kubernetes provider updates the table by calling table.Thing.Store(name, val) and table.Thing.Delete(name); updating a map key with a value that is deep-equal (usually reflect.DeepEqual, but you can implement your own .Equal method) the current value is a no-op; it won’t trigger an event for subscribers. This is handy so that the publisher doesn’t have as much state to keep track of; it doesn’t need to know “did I already publish this thing”, it can just .Store its data and watchable will do the right thing.

Sub

Meanwhile, the translator and other interested components subscribe to it with table.Thing.Subscribe (or table.Thing.SubscribeSubset if they only care about a few “Thing"s). So the translator goroutine might look like:

func(ctx context.Context) error {
    for snapshot := range k8sTable.HTTPRoutes.Subscribe(ctx) {
        fullState := irInput{
           GatewayClasses: k8sTable.GatewayClasses.LoadAll(),
           Gateways:       k8sTable.Gateways.LoadAll(),
           HTTPRoutes:     snapshot.State,
        }
        translate(irInput)
    }
}

Or, to watch multiple maps in the same loop:

func worker(ctx context.Context) error {
    classCh := k8sTable.GatewayClasses.Subscribe(ctx)
    gwCh := k8sTable.Gateways.Subscribe(ctx)
    routeCh := k8sTable.HTTPRoutes.Subscribe(ctx)
    for ctx.Err() == nil {
        var arg irInput
        select {
        case snapshot := <-classCh:
            arg.GatewayClasses = snapshot.State
        case snapshot := <-gwCh:
            arg.Gateways = snapshot.State
        case snapshot := <-routeCh:
            arg.Routes = snapshot.State
        }
        if arg.GateWayClasses == nil {
            arg.GatewayClasses = k8sTable.GateWayClasses.LoadAll()
        }
        if arg.GateWays == nil {
            arg.Gateways = k8sTable.GateWays.LoadAll()
        }
        if arg.HTTPRoutes == nil {
            arg.HTTPRoutes = k8sTable.HTTPRoutes.LoadAll()
        }
        translate(irInput)
    }
}

From the updates it gets from .Subscribe, it can get a full view of the map being subscribed to via snapshot.State; but it must read the other maps explicitly. Like sync.Map, watchable.Maps are thread-safe; while .Subscribe is a handy way to know when to run, .Load and friends can be used without subscribing.

There can be any number of subscribers. For that matter, there can be any number of publishers .Storeing things, but it’s probably wise to just have one publisher for each map.

The channel returned from .Subscribe is immediately readable with a snapshot of the map as it existed when .Subscribe was called; and becomes readable again whenever .Store or .Delete mutates the map. If multiple mutations happen between reads (or if mutations happen between .Subscribe and the first read), they are coalesced in to one snapshot to be read; the snapshot.State is the most-recent full state, and snapshot.Updates is a listing of each of the mutations that cause this snapshot to be different than the last-read one. This way subscribers don’t need to worry about a backlog accumulating if they can’t keep up with the rate of changes from the publisher.

If the map contains anything before .Subscribe is called, that very first read won’t include snapshot.Updates entries for those pre-existing items; if you are working with snapshot.Update instead of snapshot.State, then you must add special handling for your first read. We have a utility function ./internal/message.HandleSubscription to help with this.

Other Notes

The common pattern will likely be that the entrypoint that launches the goroutines for each component instantiates the map, and passes them to the appropriate publishers and subscribers; same as if they were communicating via a dumb chan.

A limitation of watchable.Map is that in order to ensure safety between goroutines, it does require that value types be deep-copiable; either by having a DeepCopy method, being a proto.Message, or by containing no reference types and so can be deep-copied by naive assignment. Fortunately, we’re using controller-gen anyway, and controller-gen can generate DeepCopy methods for us: just stick a // +k8s:deepcopy-gen=true on the types that you want it to generate methods for.

4 - Gateway API Translator Design

The Gateway API translates external resources, e.g. GatewayClass, from the configured Provider to the Intermediate Representation (IR).

Assumptions

Initially target core conformance features only, to be followed by extended conformance features.

Inputs and Outputs

The main inputs to the Gateway API translator are:

GatewayClass, Gateway, HTTPRoute, TLSRoute, Service, ReferenceGrant, Namespace, and Secret resources.

Note: ReferenceGrant is not fully implemented as of v0.2.

The outputs of the Gateway API translator are:

Xds and Infra Internal Representations (IRs).
Status updates for GatewayClass, Gateways, HTTPRoutes

Listener Compatibility

Envoy Gateway follows Gateway API listener compatibility spec:

Each listener in a Gateway must have a unique combination of Hostname, Port, and Protocol. An implementation MAY group Listeners by Port and then collapse each group of Listeners into a single Listener if the implementation determines that the Listeners in the group are “compatible”.

Note: Envoy Gateway does not collapse listeners across multiple Gateways.

Listener Compatibility Examples

Example 1: Gateway with compatible Listeners (same port & protocol, different hostnames)

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: gateway-1
  namespace: envoy-gateway
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
      hostname: "*.envoygateway.io"
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
      hostname: whales.envoygateway.io

Example 2: Gateway with compatible Listeners (same port & protocol, one hostname specified, one not)

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: gateway-1
  namespace: envoy-gateway
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
      hostname: "*.envoygateway.io"
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All

Example 3: Gateway with incompatible Listeners (same port, protocol and hostname)

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: gateway-1
  namespace: envoy-gateway
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
      hostname: whales.envoygateway.io
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
      hostname: whales.envoygateway.io

Example 4: Gateway with incompatible Listeners (neither specify a hostname)

kind: Gateway
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: gateway-1
  namespace: envoy-gateway
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All

Computing Status

Gateway API specifies a rich set of status fields & conditions for each resource. To achieve conformance, Envoy Gateway must compute the appropriate status fields and conditions for managed resources.

Status is computed and set for:

The managed GatewayClass (gatewayclass.status.conditions).
Each managed Gateway, based on its Listeners’ status (gateway.status.conditions). For the Kubernetes provider, the Envoy Deployment and Service status are also included to calculate Gateway status.
Listeners for each Gateway (gateway.status.listeners).
The ParentRef for each Route (route.status.parents).

The Gateway API translator is responsible for calculating status conditions while translating Gateway API resources to the IR and publishing status over the message bus. The Status Manager subscribes to these status messages and updates the resource status using the configured provider. For example, the Status Manager uses a Kubernetes client to update resource status on the Kubernetes API server.

Outline

The following roughly outlines the translation process. Each step may produce (1) IR; and (2) status updates on Gateway API resources.

Process Gateway Listeners
- Validate unique hostnames, ports, and protocols.
- Validate and compute supported kinds.
- Validate allowed namespaces (validate selector if specified).
- Validate TLS fields if specified, including resolving referenced Secrets.
Process HTTPRoutes
- foreach route rule:
  - compute matches
    - [core] path exact, path prefix
    - [core] header exact
    - [extended] query param exact
    - [extended] HTTP method
  - compute filters
    - [core] request header modifier (set/add/remove)
    - [core] request redirect (hostname, statuscode)
    - [extended] request mirror
  - compute backends
    - [core] Kubernetes services
- foreach route parent ref:
  - get matching listeners (check Gateway, section name, listener validation status, listener allowed routes, hostname intersection)
  - foreach matching listener:
    - foreach hostname intersection with route:
      - add each computed route rule to host

Context Structs

To help store, access and manipulate information as it’s processed during the translation process, a set of context structs are used. These structs wrap a given Gateway API type, and add additional fields and methods to support processing.

GatewayContext wraps a Gateway and provides helper methods for setting conditions, accessing Listeners, etc.

type GatewayContext struct {
	// The managed Gateway
	*v1beta1.Gateway

	// A list of Gateway ListenerContexts.
	listeners []*ListenerContext
}

ListenerContext wraps a Listener and provides helper methods for setting conditions and other status information on the associated Gateway.

type ListenerContext struct {
    // The Gateway listener.
	*v1beta1.Listener

	// The Gateway this Listener belongs to.
	gateway           *v1beta1.Gateway

	// An index used for managing this listener in the list of Gateway listeners.
	listenerStatusIdx int

	// Only Routes in namespaces selected by the selector may be attached
	// to the Gateway this listener belongs to.
	namespaceSelector labels.Selector

	// The TLS Secret for this Listener, if applicable.
	tlsSecret         *v1.Secret
}

RouteContext represents a generic Route object (HTTPRoute, TLSRoute, etc.) that can reference Gateway objects.

type RouteContext interface {
	client.Object

	// GetRouteType returns the Kind of the Route object, HTTPRoute,
	// TLSRoute, TCPRoute, UDPRoute etc.
	GetRouteType() string

	// GetHostnames returns the hosts targeted by the Route object.
	GetHostnames() []string

	// GetParentReferences returns the ParentReference of the Route object.
	GetParentReferences() []v1beta1.ParentReference

	// GetRouteParentContext returns RouteParentContext by using the Route
	// objects' ParentReference.
	GetRouteParentContext(forParentRef v1beta1.ParentReference) *RouteParentContext
}

5 - Configuration API Design

Motivation

Issue 51 specifies the need to design an API for configuring Envoy Gateway. The control plane is configured statically at startup and the data plane is configured dynamically through Kubernetes resources, primarily Gateway API objects. Refer to the Envoy Gateway design doc for additional details regarding Envoy Gateway terminology and configuration.

Goals

Define an initial API to configure Envoy Gateway at startup.
Define an initial API for configuring the managed data plane, e.g. Envoy proxies.

Non-Goals

Implementation of the configuration APIs.
Define the status subresource of the configuration APIs.
Define a complete set of APIs for configuring Envoy Gateway. As stated in the Goals, this document defines the initial configuration APIs.
Define an API for deploying/provisioning/operating Envoy Gateway. If needed, a future Envoy Gateway operator would be responsible for designing and implementing this type of API.
Specify tooling for managing the API, e.g. generate protos, CRDs, controller RBAC, etc.

Control Plane API

The EnvoyGateway API defines the control plane configuration, e.g. Envoy Gateway. Key points of this API are:

It will define Envoy Gateway’s startup configuration file. If the file does not exist, Envoy Gateway will start up with default configuration parameters.
EnvoyGateway inlines the TypeMeta API. This allows EnvoyGateway to be versioned and managed as a GroupVersionKind scheme.
EnvoyGateway does not contain a metadata field since it’s currently represented as a static configuration file instead of a Kubernetes resource.
Since EnvoyGateway does not surface status, EnvoyGatewaySpec is inlined.
If data plane static configuration is required in the future, Envoy Gateway will use a separate file for this purpose.

The v1alpha1 version and config.gateway.envoyproxy.io API group get generated:

// gateway/api/config/v1alpha1/doc.go

// Package v1alpha1 contains API Schema definitions for the config.gateway.envoyproxy.io API group.
//
// +groupName=config.gateway.envoyproxy.io
package v1alpha1

The initial EnvoyGateway API:

// gateway/api/config/v1alpha1/envoygateway.go

package valpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EnvoyGateway is the Schema for the envoygateways API
type EnvoyGateway struct {
	metav1.TypeMeta `json:",inline"`

	// EnvoyGatewaySpec defines the desired state of Envoy Gateway.
	EnvoyGatewaySpec `json:",inline"`
}

// EnvoyGatewaySpec defines the desired state of Envoy Gateway configuration.
type EnvoyGatewaySpec struct {
	// Gateway defines Gateway-API specific configuration. If unset, default
	// configuration parameters will apply.
	//
	// +optional
	Gateway *Gateway `json:"gateway,omitempty"`

	// Provider defines the desired provider configuration. If unspecified,
	// the Kubernetes provider is used with default parameters.
	//
	// +optional
	Provider *Provider `json:"provider,omitempty"`
}

// Gateway defines desired Gateway API configuration of Envoy Gateway.
type Gateway struct {
	// ControllerName defines the name of the Gateway API controller. If unspecified,
	// defaults to "gateway.envoyproxy.io/gatewayclass-controller". See the following
	// for additional details:
	//
	// https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1alpha2.GatewayClass
	//
	// +optional
	ControllerName string `json:"controllerName,omitempty"`
}

// Provider defines the desired configuration of a provider.
// +union
type Provider struct {
	// Type is the type of provider to use. If unset, the Kubernetes provider is used.
	//
	// +unionDiscriminator
	Type ProviderType `json:"type,omitempty"`
	// Kubernetes defines the configuration of the Kubernetes provider. Kubernetes
	// provides runtime configuration via the Kubernetes API.
	//
	// +optional
	Kubernetes *KubernetesProvider `json:"kubernetes,omitempty"`

	// File defines the configuration of the File provider. File provides runtime
	// configuration defined by one or more files.
	//
	// +optional
	File *FileProvider `json:"file,omitempty"`
}

// ProviderType defines the types of providers supported by Envoy Gateway.
type ProviderType string

const (
	// KubernetesProviderType defines the "Kubernetes" provider.
	KubernetesProviderType ProviderType = "Kubernetes"

	// FileProviderType defines the "File" provider.
	FileProviderType ProviderType = "File"
)

// KubernetesProvider defines configuration for the Kubernetes provider.
type KubernetesProvider struct {
	// TODO: Add config as use cases are better understood.
}

// FileProvider defines configuration for the File provider.
type FileProvider struct {
	// TODO: Add config as use cases are better understood.
}

Note: Provider-specific configuration is defined in the {$PROVIDER_NAME}Provider API.

Gateway

Gateway defines desired configuration of Gateway API controllers that reconcile and translate Gateway API resources into the Intermediate Representation (IR). Refer to the Envoy Gateway design doc for additional details.

Provider

Provider defines the desired configuration of an Envoy Gateway provider. A provider is an infrastructure component that Envoy Gateway calls to establish its runtime configuration. Provider is a union type. Therefore, Envoy Gateway can be configured with only one provider based on the type discriminator field. Refer to the Envoy Gateway design doc for additional details.

Control Plane Configuration

The configuration file is defined by the EnvoyGateway API type. At startup, Envoy Gateway searches for the configuration at “/etc/envoy-gateway/config.yaml”.

Start Envoy Gateway:

$ ./envoy-gateway

Since the configuration file does not exist, Envoy Gateway will start with default configuration parameters.

The Kubernetes provider can be configured explicitly using provider.kubernetes:

$ cat << EOF > /etc/envoy-gateway/config.yaml
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
provider:
  type: Kubernetes
  kubernetes: {}
EOF

This configuration will cause Envoy Gateway to use the Kubernetes provider with default configuration parameters.

The Kubernetes provider can be configured using the provider field. For example, the foo field can be set to “bar”:

$ cat << EOF > /etc/envoy-gateway/config.yaml
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
provider:
  type: Kubernetes
  kubernetes:
    foo: bar
EOF

Note: The Provider API from the Kubernetes package is currently undefined and foo: bar is provided for illustration purposes only.

The same API structure is followed for each supported provider. The following example causes Envoy Gateway to use the File provider:

$ cat << EOF > /etc/envoy-gateway/config.yaml
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
provider:
  type: File
  file:
    foo: bar
EOF

Note: The Provider API from the File package is currently undefined and foo: bar is provided for illustration purposes only.

Gateway API-related configuration is expressed through the gateway field. If unspecified, Envoy Gateway will use default configuration parameters for gateway. The following example causes the GatewayClass controller to manage GatewayClasses with controllerName foo instead of the default gateway.envoyproxy.io/gatewayclass-controller:

$ cat << EOF > /etc/envoy-gateway/config.yaml
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
gateway:
  controllerName: foo

With any of the above configuration examples, Envoy Gateway can be started without any additional arguments:

$ ./envoy-gateway

Data Plane API

The data plane is configured dynamically through Kubernetes resources, primarily Gateway API objects. Optionally, the data plane infrastructure can be configured by referencing a custom resource (CR) through spec.parametersRef of the managed GatewayClass. The EnvoyProxy API defines the data plane infrastructure configuration and is represented as the CR referenced by the managed GatewayClass. Key points of this API are:

If unreferenced by gatewayclass.spec.parametersRef, default parameters will be used to configure the data plane infrastructure, e.g. expose Envoy network endpoints using a LoadBalancer service.
Envoy Gateway will follow Gateway API recommendations regarding updates to the EnvoyProxy CR:
It is recommended that this resource be used as a template for Gateways. This means that a Gateway is based on the state of the GatewayClass at the time it was created and changes to the GatewayClass or associated parameters are not propagated down to existing Gateways.

The initial EnvoyProxy API:

// gateway/api/config/v1alpha1/envoyproxy.go

package v1alpha1

import (
	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
)

// EnvoyProxy is the Schema for the envoyproxies API.
type EnvoyProxy struct {
	metav1.TypeMeta   `json:",inline"`
	metav1.ObjectMeta `json:"metadata,omitempty"`

	Spec   EnvoyProxySpec   `json:"spec,omitempty"`
	Status EnvoyProxyStatus `json:"status,omitempty"`
}

// EnvoyProxySpec defines the desired state of Envoy Proxy infrastructure
// configuration.
type EnvoyProxySpec struct {
	// Undefined by this design spec.
}

// EnvoyProxyStatus defines the observed state of EnvoyProxy.
type EnvoyProxyStatus struct {
	// Undefined by this design spec.
}

The EnvoyProxySpec and EnvoyProxyStatus fields will be defined in the future as proxy infrastructure configuration use cases are better understood.

Data Plane Configuration

GatewayClass and Gateway resources define the data plane infrastructure. Note that all examples assume Envoy Gateway is running with the Kubernetes provider.

apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: example-class
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: example-gateway
spec:
  gatewayClassName: example-class
  listeners:
  - name: http
    protocol: HTTP
    port: 80

Since the GatewayClass does not define spec.parametersRef, the data plane is provisioned using default configuration parameters. The Envoy proxies will be configured with a http listener and a Kubernetes LoadBalancer service listening on port 80.

The following example will configure the data plane to use a ClusterIP service instead of the default LoadBalancer service:

apiVersion: gateway.networking.k8s.io/v1beta1
kind: GatewayClass
metadata:
  name: example-class
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    name: example-config
    group: config.gateway.envoyproxy.io
    kind: EnvoyProxy
---
apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: example-gateway
spec:
  gatewayClassName: example-class
  listeners:
  - name: http
    protocol: HTTP
    port: 80
---
apiVersion: config.gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: example-config
spec:
  networkPublishing:
    type: ClusterIPService

Note: The NetworkPublishing API is currently undefined and is provided here for illustration purposes only.