This is the multi-page printable view of this section. Click here to print.
Operations
1 - Customize EnvoyProxy
Envoy Gateway provides an EnvoyProxy CRD that can be linked to the ParametersRef in a Gateway and GatewayClass, allowing cluster admins to customize the managed EnvoyProxy Deployment and Service. To learn more about GatewayClass and ParametersRef, please refer to Gateway API documentation.
Prerequisites
Follow the steps from the Quickstart task to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Verify the Gateway status:
kubectl get gateway/eg -o yaml
egctl x status gateway -v
Before you start, you need to add Infrastructure.ParametersRef in Gateway, and refer to EnvoyProxy Config:
Note: MergeGateways cannot be set to true in your EnvoyProxy config if attaching to the Gateway.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
spec:
gatewayClassName: eg
infrastructure:
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: custom-proxy-config
listeners:
- name: http
protocol: HTTP
port: 80
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
spec:
gatewayClassName: eg
infrastructure:
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: custom-proxy-config
listeners:
- name: http
protocol: HTTP
port: 80
You can also attach the EnvoyProxy resource to the GatewayClass using the parametersRef field.
This configuration is discouraged if you plan on creating multiple Gateways linking to the same
GatewayClass and would like different infrastructure configurations for each of them.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: custom-proxy-config
namespace: default
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: custom-proxy-config
namespace: default
Customize EnvoyProxy Deployment Replicas
You can customize the EnvoyProxy Deployment Replicas via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
replicas: 2
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
replicas: 2
After you apply the config, you should see the replicas of envoyproxy changes to 2. And also you can dynamically change the value.
kubectl get deployment -l gateway.envoyproxy.io/owning-gateway-name=eg -n envoy-gateway-system
Customize EnvoyProxy Image
You can customize the EnvoyProxy Image via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
image: envoyproxy/envoy:v1.25-latest
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
image: envoyproxy/envoy:v1.25-latest
After applying the config, you can get the deployment image, and see it has changed.
Customize EnvoyProxy Pod Annotations
You can customize the EnvoyProxy Pod Annotations via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
pod:
annotations:
custom1: deploy-annotation1
custom2: deploy-annotation2
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
pod:
annotations:
custom1: deploy-annotation1
custom2: deploy-annotation2
After applying the config, you can get the envoyproxy pods, and see new annotations has been added.
Customize EnvoyProxy Deployment Resources
You can customize the EnvoyProxy Deployment Resources via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
resources:
requests:
cpu: 150m
memory: 640Mi
limits:
cpu: 500m
memory: 1Gi
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
resources:
requests:
cpu: 150m
memory: 640Mi
limits:
cpu: 500m
memory: 1Gi
Customize EnvoyProxy Deployment Env
You can customize the EnvoyProxy Deployment Env via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
env:
- name: env_a
value: env_a_value
- name: env_b
value: env_b_value
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
env:
- name: env_a
value: env_a_value
- name: env_b
value: env_b_value
Envoy Gateway has provided two initial
envENVOY_GATEWAY_NAMESPACEandENVOY_POD_NAMEfor envoyproxy container.
After applying the config, you can get the envoyproxy deployment, and see resources has been changed.
Customize EnvoyProxy Deployment Volumes or VolumeMounts
You can customize the EnvoyProxy Deployment Volumes or VolumeMounts via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
volumeMounts:
- mountPath: /certs
name: certs
readOnly: true
pod:
volumes:
- name: certs
secret:
secretName: envoy-cert
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
container:
volumeMounts:
- mountPath: /certs
name: certs
readOnly: true
pod:
volumes:
- name: certs
secret:
secretName: envoy-cert
After applying the config, you can get the envoyproxy deployment, and see resources has been changed.
Customize EnvoyProxy Service Annotations
You can customize the EnvoyProxy Service Annotations via EnvoyProxy Config like:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyService:
annotations:
custom1: svc-annotation1
custom2: svc-annotation2
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyService:
annotations:
custom1: svc-annotation1
custom2: svc-annotation2
After applying the config, you can get the envoyproxy service, and see annotations has been added.
Customize EnvoyProxy Bootstrap Config
You can customize the EnvoyProxy bootstrap config via EnvoyProxy Config. There are two ways to customize it:
- Replace: the whole bootstrap config will be replaced by the config you provided.
- Merge: the config you provided will be merged into the default bootstrap config.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
bootstrap:
type: Replace
value: |
admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 20000
dynamic_resources:
ads_config:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
ads: {}
resource_api_version: V3
cds_config:
ads: {}
resource_api_version: V3
static_resources:
clusters:
- connect_timeout: 10s
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
typed_extension_protocol_options:
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
"explicit_http_config":
"http2_protocol_options": {}
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: "/sds/xds-certificate.json"
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: "/sds/xds-trusted-ca.json"
resource_api_version: V3
layered_runtime:
layers:
- name: runtime-0
rtds_layer:
rtds_config:
ads: {}
resource_api_version: V3
name: runtime-0
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
bootstrap:
type: Replace
value: |
admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 20000
dynamic_resources:
ads_config:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
ads: {}
resource_api_version: V3
cds_config:
ads: {}
resource_api_version: V3
static_resources:
clusters:
- connect_timeout: 10s
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
typed_extension_protocol_options:
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
"explicit_http_config":
"http2_protocol_options": {}
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: "/sds/xds-certificate.json"
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: "/sds/xds-trusted-ca.json"
resource_api_version: V3
layered_runtime:
layers:
- name: runtime-0
rtds_layer:
rtds_config:
ads: {}
resource_api_version: V3
name: runtime-0
You can use egctl translate to get the default xDS Bootstrap configuration used by Envoy Gateway.
After applying the config, the bootstrap config will be overridden by the new config you provided.
Any errors in the configuration will be surfaced as status within the GatewayClass resource.
You can also validate this configuration using egctl translate.
Customize EnvoyProxy Horizontal Pod Autoscaler
You can enable Horizontal Pod Autoscaler for EnvoyProxy Deployment. However, before enabling the HPA for EnvoyProxy, please ensure that the metrics-server component is installed in the cluster.
Once confirmed, you can apply it via EnvoyProxy Config as shown below:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyHpa:
minReplicas: 2
maxReplicas: 10
metrics:
- resource:
name: cpu
target:
averageUtilization: 60
type: Utilization
type: Resource
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyHpa:
minReplicas: 2
maxReplicas: 10
metrics:
- resource:
name: cpu
target:
averageUtilization: 60
type: Utilization
type: Resource
After applying the config, the EnvoyProxy HPA (Horizontal Pod Autoscaler) is generated. However, upon activating the EnvoyProxy’s HPA, the Envoy Gateway will no longer reference the replicas field specified in the envoyDeployment, as outlined here.
Customize EnvoyProxy Command line options
You can customize the EnvoyProxy Command line options via spec.extraArgs in EnvoyProxy Config.
For example, the following configuration will add --disable-extensions arg in order to disable envoy.access_loggers/envoy.access_loggers.wasm extension:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
extraArgs:
- --disable-extensions envoy.access_loggers/envoy.access_loggers.wasm
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
extraArgs:
- --disable-extensions envoy.access_loggers/envoy.access_loggers.wasm
Customize EnvoyProxy with Patches
You can customize the EnvoyProxy using patches.
Patching Deployment for EnvoyProxy
For example, the following configuration will add resource limits to the envoy and the shutdown-manager containers in the envoyproxy deployment:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: eg
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
patch:
type: StrategicMerge
value:
spec:
template:
spec:
containers:
- name: envoy
resources:
limits:
cpu: 500m
memory: 1024Mi
- name: shutdown-manager
resources:
limits:
cpu: 200m
memory: 1024Mi
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: eg
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyDeployment:
patch:
type: StrategicMerge
value:
spec:
template:
spec:
containers:
- name: envoy
resources:
limits:
cpu: 500m
memory: 1024Mi
- name: shutdown-manager
resources:
limits:
cpu: 200m
memory: 1024Mi
After applying the configuration, you will see the change in both containers in the envoyproxy deployment.
Patching Service for EnvoyProxy
For example, the following configuration will add an annotation for the envoyproxy service:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: eg
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyService:
patch:
type: StrategicMerge
value:
metadata:
annotations:
custom-annotation: foobar
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: eg
namespace: default
spec:
provider:
type: Kubernetes
kubernetes:
envoyService:
patch:
type: StrategicMerge
value:
metadata:
annotations:
custom-annotation: foobar
After applying the configuration, you will see the custom-annotation: foobar has been added to the envoyproxy service.
Customize Filter Order
Under the hood, Envoy Gateway uses a series of Envoy HTTP filters to process HTTP requests and responses, and to apply various policies.
By default, Envoy Gateway applies the following filters in the order shown:
- envoy.filters.http.fault
- envoy.filters.http.cors
- envoy.filters.http.ext_authz
- envoy.filters.http.basic_authn
- envoy.filters.http.oauth2
- envoy.filters.http.jwt_authn
- envoy.filters.http.ext_proc
- envoy.filters.http.wasm
- envoy.filters.http.rbac
- envoy.filters.http.local_ratelimit
- envoy.filters.http.ratelimit
- envoy.filters.http.router
The default order in which these filters are applied is opinionated and may not suit all use cases.
To address this, Envoy Gateway allows you to adjust the execution order of these filters with the filterOrder field in the EnvoyProxy resource.
filterOrder is a list of customized filter order configurations. Each configuration can specify a filter
name and a filter to place it before or after. These configurations are applied in the order they are listed.
If a filter occurs in multiple configurations, the final order is the result of applying all these configurations in order.
To avoid conflicts, it is recommended to only specify one configuration per filter.
For example, the following configuration moves the envoy.filters.http.wasm filter before the envoy.filters.http.jwt_authn
filter and the envoy.filters.http.cors filter after the envoy.filters.http.basic_authn filter:
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
filterOrder:
- name: envoy.filters.http.wasm
before: envoy.filters.http.jwt_authn
- name: envoy.filters.http.cors
after: envoy.filters.http.basic_authn
EOF
Save and apply the following resource to your cluster:
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: default
spec:
filterOrder:
- name: envoy.filters.http.wasm
before: envoy.filters.http.jwt_authn
- name: envoy.filters.http.cors
after: envoy.filters.http.basic_authn
2 - Deployment Mode
Deployment modes
One GatewayClass per Envoy Gateway Controller
- An Envoy Gateway is associated with a single GatewayClass resource under one controller. This is the simplest deployment mode and is suitable for scenarios where each Gateway needs to have its own dedicated set of resources and configurations.
Multiple GatewayClasses per Envoy Gateway Controller
- An Envoy Gateway is associated with multiple GatewayClass resources under one controller.
- Support for accepting multiple GatewayClasses was added here.
Separate Envoy Gateway Controllers
If you’ve instantiated multiple GatewayClasses, you can also run separate Envoy Gateway controllers in different namespaces, linking a GatewayClass to each of them for multi-tenancy. Please follow the example Multi-tenancy.
Merged Gateways onto a single EnvoyProxy fleet
By default, each Gateway has its own dedicated set of Envoy Proxy and its configurations. However, for some deployments, it may be more convenient to merge listeners across multiple Gateways and deploy a single Envoy Proxy fleet.
This can help to efficiently utilize the infra resources in the cluster and manage them in a centralized manner, or have a single IP address for all of the listeners.
Setting the mergeGateways field in the EnvoyProxy resource linked to GatewayClass will result in merging all Gateway listeners under one GatewayClass resource.
- The tuple of port, protocol, and hostname must be unique across all Listeners.
Please follow the example Merged gateways deployment.
Supported Modes
Kubernetes
- The default deployment model is - Envoy Gateway watches for resources such a
Service&HTTPRoutein all namespaces and creates managed data plane resources such as EnvoyProxyDeploymentin the namespace where Envoy Gateway is running. - Envoy Gateway also supports Namespaced deployment mode, you can watch resources in the specific namespaces by assigning
EnvoyGateway.provider.kubernetes.watch.namespacesorEnvoyGateway.provider.kubernetes.watch.namespaceSelectorand creates managed data plane resources in the namespace where Envoy Gateway is running. - Support for alternate deployment modes is being tracked here.
Multi-tenancy
Kubernetes
A
tenantis a group within an organization (e.g. a team or department) who shares organizational resources. We recommend eachtenantdeploy their own Envoy Gateway controller in their respectivenamespace. Below is an example of deploying Envoy Gateway by themarketingandproductteams in separate namespaces.Lets deploy Envoy Gateway in the
marketingnamespace and also watch resources only in this namespace. We are also setting the controller name to a unique string heregateway.envoyproxy.io/marketing-gatewayclass-controller.
helm install \
--set config.envoyGateway.gateway.controllerName=gateway.envoyproxy.io/marketing-gatewayclass-controller \
--set config.envoyGateway.provider.kubernetes.watch.type=Namespaces \
--set config.envoyGateway.provider.kubernetes.watch.namespaces={marketing} \
eg-marketing oci://docker.io/envoyproxy/gateway-helm \
--version v1.2.1 -n marketing --create-namespace
Lets create a GatewayClass linked to the marketing team’s Envoy Gateway controller, and as well other resources linked to it, so the backend application operated by this team can be exposed to external clients.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg-marketing
spec:
controllerName: gateway.envoyproxy.io/marketing-gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: marketing
spec:
gatewayClassName: eg-marketing
listeners:
- name: http
protocol: HTTP
port: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend
namespace: marketing
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: marketing
labels:
app: backend
service: backend
spec:
ports:
- name: http
port: 3000
targetPort: 3000
selector:
app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
namespace: marketing
spec:
replicas: 1
selector:
matchLabels:
app: backend
version: v1
template:
metadata:
labels:
app: backend
version: v1
spec:
serviceAccountName: backend
containers:
- image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
imagePullPolicy: IfNotPresent
name: backend
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: marketing
spec:
parentRefs:
- name: eg
hostnames:
- "www.marketing.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
Save and apply the following resources to your cluster:
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg-marketing
spec:
controllerName: gateway.envoyproxy.io/marketing-gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: marketing
spec:
gatewayClassName: eg-marketing
listeners:
- name: http
protocol: HTTP
port: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend
namespace: marketing
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: marketing
labels:
app: backend
service: backend
spec:
ports:
- name: http
port: 3000
targetPort: 3000
selector:
app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
namespace: marketing
spec:
replicas: 1
selector:
matchLabels:
app: backend
version: v1
template:
metadata:
labels:
app: backend
version: v1
spec:
serviceAccountName: backend
containers:
- image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
imagePullPolicy: IfNotPresent
name: backend
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: marketing
spec:
parentRefs:
- name: eg
hostnames:
- "www.marketing.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
Lets port forward to the generated envoy proxy service in the marketing namespace and send a request to it.
export ENVOY_SERVICE=$(kubectl get svc -n marketing --selector=gateway.envoyproxy.io/owning-gateway-namespace=marketing,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
kubectl -n marketing port-forward service/${ENVOY_SERVICE} 8888:8080 &
curl --verbose --header "Host: www.marketing.example.com" http://localhost:8888/get
* Trying 127.0.0.1:8888...
* Connected to localhost (127.0.0.1) port 8888 (#0)
> GET /get HTTP/1.1
> Host: www.marketing.example.com
> User-Agent: curl/7.86.0
> Accept: */*
>
Handling connection for 8888
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Thu, 20 Apr 2023 19:19:42 GMT
< content-length: 521
< x-envoy-upstream-service-time: 0
< server: envoy
<
{
"path": "/get",
"host": "www.marketing.example.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.86.0"
],
"X-Envoy-Expected-Rq-Timeout-Ms": [
"15000"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"10.1.0.157"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"c637977c-458a-48ae-92b3-f8c429849322"
]
},
"namespace": "marketing",
"ingress": "",
"service": "",
"pod": "backend-74888f465f-bcs8f"
* Connection #0 to host localhost left intact
- Lets deploy Envoy Gateway in the
productnamespace and also watch resources only in this namespace.
helm install \
--set config.envoyGateway.gateway.controllerName=gateway.envoyproxy.io/product-gatewayclass-controller \
--set config.envoyGateway.provider.kubernetes.watch.type=Namespaces \
--set config.envoyGateway.provider.kubernetes.watch.namespaces={product} \
eg-product oci://docker.io/envoyproxy/gateway-helm \
--version v1.2.1 -n product --create-namespace
Lets create a GatewayClass linked to the product team’s Envoy Gateway controller, and as well other resources linked to it, so the backend application operated by this team can be exposed to external clients.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg-product
spec:
controllerName: gateway.envoyproxy.io/product-gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: product
spec:
gatewayClassName: eg-product
listeners:
- name: http
protocol: HTTP
port: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend
namespace: product
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: product
labels:
app: backend
service: backend
spec:
ports:
- name: http
port: 3000
targetPort: 3000
selector:
app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
namespace: product
spec:
replicas: 1
selector:
matchLabels:
app: backend
version: v1
template:
metadata:
labels:
app: backend
version: v1
spec:
serviceAccountName: backend
containers:
- image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
imagePullPolicy: IfNotPresent
name: backend
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: product
spec:
parentRefs:
- name: eg
hostnames:
- "www.product.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
Save and apply the following resources to your cluster:
---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg-product
spec:
controllerName: gateway.envoyproxy.io/product-gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: product
spec:
gatewayClassName: eg-product
listeners:
- name: http
protocol: HTTP
port: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: backend
namespace: product
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: product
labels:
app: backend
service: backend
spec:
ports:
- name: http
port: 3000
targetPort: 3000
selector:
app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: backend
namespace: product
spec:
replicas: 1
selector:
matchLabels:
app: backend
version: v1
template:
metadata:
labels:
app: backend
version: v1
spec:
serviceAccountName: backend
containers:
- image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
imagePullPolicy: IfNotPresent
name: backend
ports:
- containerPort: 3000
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: product
spec:
parentRefs:
- name: eg
hostnames:
- "www.product.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
Lets port forward to the generated envoy proxy service in the product namespace and send a request to it.
export ENVOY_SERVICE=$(kubectl get svc -n product --selector=gateway.envoyproxy.io/owning-gateway-namespace=product,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
kubectl -n product port-forward service/${ENVOY_SERVICE} 8889:8080 &
curl --verbose --header "Host: www.product.example.com" http://localhost:8889/get
* Trying 127.0.0.1:8889...
* Connected to localhost (127.0.0.1) port 8889 (#0)
> GET /get HTTP/1.1
> Host: www.product.example.com
> User-Agent: curl/7.86.0
> Accept: */*
>
Handling connection for 8889
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Thu, 20 Apr 2023 19:20:17 GMT
< content-length: 517
< x-envoy-upstream-service-time: 0
< server: envoy
<
{
"path": "/get",
"host": "www.product.example.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/7.86.0"
],
"X-Envoy-Expected-Rq-Timeout-Ms": [
"15000"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"10.1.0.156"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"39196453-2250-4331-b756-54003b2853c2"
]
},
"namespace": "product",
"ingress": "",
"service": "",
"pod": "backend-74888f465f-64fjs"
* Connection #0 to host localhost left intact
With the below command you can ensure that you are no able to access the marketing team’s backend exposed using the www.marketing.example.com hostname
and the product team’s data plane.
curl --verbose --header "Host: www.marketing.example.com" http://localhost:8889/get
* Trying 127.0.0.1:8889...
* Connected to localhost (127.0.0.1) port 8889 (#0)
> GET /get HTTP/1.1
> Host: www.marketing.example.com
> User-Agent: curl/7.86.0
> Accept: */*
>
Handling connection for 8889
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< date: Thu, 20 Apr 2023 19:22:13 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host localhost left intact
Merged gateways deployment
In this example, we will deploy GatewayClass
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: merged-eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: custom-proxy-config
namespace: envoy-gateway-system
with a referenced EnvoyProxy resource configured to enable merged Gateways deployment mode.
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: custom-proxy-config
namespace: envoy-gateway-system
spec:
mergeGateways: true
Deploy merged-gateways example
Deploy resources on your cluster from the example.
kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/merged-gateways.yaml
Verify that Gateways are deployed and programmed
kubectl get gateways -n default
NAMESPACE NAME CLASS ADDRESS PROGRAMMED AGE
default merged-eg-1 merged-eg 172.18.255.202 True 2m4s
default merged-eg-2 merged-eg 172.18.255.202 True 2m4s
default merged-eg-3 merged-eg 172.18.255.202 True 2m4s
Verify that HTTPRoutes are deployed
kubectl get httproute -n default
NAMESPACE NAME HOSTNAMES AGE
default hostname1-route ["www.merged1.com"] 2m4s
default hostname2-route ["www.merged2.com"] 2m4s
default hostname3-route ["www.merged3.com"] 2m4s
If you take a look at the deployed Envoy Proxy service you would notice that all of the Gateway listeners ports are added to that service.
kubectl get service -n envoy-gateway-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy-gateway ClusterIP 10.96.141.4 <none> 18000/TCP,18001/TCP 6m43s
envoy-gateway-metrics-service ClusterIP 10.96.113.191 <none> 19001/TCP 6m43s
envoy-merged-eg-668ac7ae LoadBalancer 10.96.48.255 172.18.255.202 8081:30467/TCP,8082:31793/TCP,8080:31153/TCP 3m17s
There should be also one deployment (envoy-merged-eg-668ac7ae-775f9865d-55zhs) for every Gateway and its name should reference the name of the GatewayClass.
kubectl get pods -n envoy-gateway-system
NAME READY STATUS RESTARTS AGE
envoy-gateway-5d998778f6-wr6m9 1/1 Running 0 6m43s
envoy-merged-eg-668ac7ae-775f9865d-55zhs 2/2 Running 0 3m17s
Testing the Configuration
Get the name of the merged gateways Envoy service:
export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gatewayclass=merged-eg -o jsonpath='{.items[0].metadata.name}')
Fetch external IP of the service:
export GATEWAY_HOST=$(kubectl get svc/${ENVOY_SERVICE} -n envoy-gateway-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
In certain environments, the load balancer may be exposed using a hostname, instead of an IP address. If so, replace
ip in the above command with hostname.
Curl the route hostname-route2 through Envoy proxy:
curl --header "Host: www.merged2.com" http://$GATEWAY_HOST:8081/example2
{
"path": "/example2",
"host": "www.merged2.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/8.4.0"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"172.18.0.2"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"deed2767-a483-4291-9429-0e256ab3a65f"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "merged-backend-64ddb65fd7-ttv5z"
}
Curl the route hostname-route1 through Envoy proxy:
curl --header "Host: www.merged1.com" http://$GATEWAY_HOST:8080/example
{
"path": "/example",
"host": "www.merged1.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/8.4.0"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"172.18.0.2"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"20a53440-6327-4c3c-bc8b-8e79e7311043"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "merged-backend-64ddb65fd7-ttv5z"
}
Verify deployment of multiple GatewayClass
Install the GatewayClass, Gateway, HTTPRoute and example app from Quickstart example:
kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/v1.2.1/quickstart.yaml -n default
Lets create also and additional Gateway linked to the GatewayClass and backend application from Quickstart example.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg-2
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 8080
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: eg-2
namespace: default
spec:
parentRefs:
- name: eg-2
hostnames:
- "www.quickstart.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
Save and apply the following resources to your cluster:
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg-2
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 8080
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: eg-2
namespace: default
spec:
parentRefs:
- name: eg-2
hostnames:
- "www.quickstart.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
Verify that Gateways are deployed and programmed
kubectl get gateways -n default
NAME CLASS ADDRESS PROGRAMMED AGE
eg eg 172.18.255.203 True 114s
eg-2 eg 172.18.255.204 True 89s
merged-eg-1 merged-eg 172.18.255.202 True 8m33s
merged-eg-2 merged-eg 172.18.255.202 True 8m33s
merged-eg-3 merged-eg 172.18.255.202 True 8m33s
Verify that HTTPRoutes are deployed
kubectl get httproute -n default
NAMESPACE NAME HOSTNAMES AGE
default backend ["www.example.com"] 2m29s
default eg-2 ["www.quickstart.example.com"] 87s
default hostname1-route ["www.merged1.com"] 10m4s
default hostname2-route ["www.merged2.com"] 10m4s
default hostname3-route ["www.merged3.com"] 10m4s
Verify that services are now deployed separately.
kubectl get service -n envoy-gateway-system
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
envoy-default-eg-2-7e515b2f LoadBalancer 10.96.121.46 172.18.255.204 8080:32705/TCP 3m27s
envoy-default-eg-e41e7b31 LoadBalancer 10.96.11.244 172.18.255.203 80:31930/TCP 2m26s
envoy-gateway ClusterIP 10.96.141.4 <none> 18000/TCP,18001/TCP 14m25s
envoy-gateway-metrics-service ClusterIP 10.96.113.191 <none> 19001/TCP 14m25s
envoy-merged-eg-668ac7ae LoadBalancer 10.96.243.32 172.18.255.202 8082:31622/TCP,8080:32262/TCP,8081:32305/TCP 10m59s
There should be two deployments for each of newly deployed Gateway and its name should reference the name of the namespace and the Gateway.
kubectl get pods -n envoy-gateway-system
NAME READY STATUS RESTARTS AGE
envoy-default-eg-2-7e515b2f-8c98fdf88-p6jhg 2/2 Running 0 3m27s
envoy-default-eg-e41e7b31-6f998d85d7-jpvmj 2/2 Running 0 2m26s
envoy-gateway-5d998778f6-wr6m9 1/1 Running 0 14m25s
envoy-merged-eg-668ac7ae-5958f7b7f6-9h9v2 2/2 Running 0 10m59s
Testing the Configuration
Get the name of the merged gateways Envoy service:
export DEFAULT_ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
Fetch external IP of the service:
export DEFAULT_GATEWAY_HOST=$(kubectl get svc/${DEFAULT_ENVOY_SERVICE} -n envoy-gateway-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
Curl the route Quickstart backend route through Envoy proxy:
curl --header "Host: www.example.com" http://$DEFAULT_GATEWAY_HOST
{
"path": "/",
"host": "www.example.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/8.4.0"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"172.18.0.2"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"70a40595-67a1-4776-955b-2dee361baed7"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "backend-96f75bbf-6w67z"
}
Curl the route hostname-route3 through Envoy proxy:
curl --header "Host: www.merged3.com" http://$GATEWAY_HOST:8082/example3
{
"path": "/example3",
"host": "www.merged3.com",
"method": "GET",
"proto": "HTTP/1.1",
"headers": {
"Accept": [
"*/*"
],
"User-Agent": [
"curl/8.4.0"
],
"X-Envoy-Internal": [
"true"
],
"X-Forwarded-For": [
"172.18.0.2"
],
"X-Forwarded-Proto": [
"http"
],
"X-Request-Id": [
"47aeaef3-abb5-481a-ab92-c2ae3d0862d6"
]
},
"namespace": "default",
"ingress": "",
"service": "",
"pod": "merged-backend-64ddb65fd7-k84gv"
}
3 - Standalone Deployment Mode
Notice
Standalone mode is an experimental feature, please DO NOT use it in production.Envoy Gateway also supports running in standalone mode. In this mode, Envoy Gateway does not need to rely on Kubernetes and can be deployed directly on bare metal or virtual machines.
Currently, Envoy Gateway only support the file provider and the host infrastructure provider combinations.
- The file provider will configure the Envoy Gateway to get all gateway-api resources from file system.
- The host infrastructure provider will configure the Envoy Gateway to deploy one Envoy Proxy as a host process.
Quick Start
In this quick-start, we will run Envoy Gateway in standalone mode with the file provider and the host infrastructure provider.
Prerequisites
Create a local directory just for testing:
mkdir -p /tmp/envoy-gateway-test
Download the Envoy Gateway binary from v1.2.x release.
Create Certificates
All runners in Envoy Gateway are using TLS connection, so create these TLS certificates locally to ensure the Envoy Gateway works properly.
envoy-gateway certgen --local
Start Envoy Gateway
Start Envoy Gateway by the following command:
envoy-gateway server --config-path standalone.yaml
with standalone.yaml configuration:
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyGateway
gateway:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
provider:
type: Custom
custom:
resource:
type: File
file:
paths: ["/tmp/envoy-gateway-test"]
infrastructure:
type: Host
host: {}
logging:
level:
default: info
extensionApis:
enableBackend: true
As you can see, we have enabled the Backend API, this API will be used to represent our local endpoints.
Trigger an Update
Any changes under watched paths will be considered as an update by the file provider.
For instance, copying example file into /tmp/envoy-gateway-test/ will trigger an update of gateway-api resources:
cp examples/standalone/quickstart.yaml /tmp/envoy-gateway-test/quickstart.yaml
From the Envoy Gateway log, you should be able to observe that the Envoy Proxy has been started, and its admin address has been returned.
Test Connection
Starts a simple local server as an endpoint:
python3 -m http.server 3000
Curl the example server through Envoy Proxy:
curl --verbose --header "Host: www.example.com" http://0.0.0.0:8888/
* Trying 0.0.0.0:8888...
* Connected to 0.0.0.0 (127.0.0.1) port 8888 (#0)
> GET / HTTP/1.1
> Host: www.example.com
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< server: SimpleHTTP/0.6 Python/3.10.12
< date: Sat, 26 Oct 2024 13:20:34 GMT
< content-type: text/html; charset=utf-8
< content-length: 1870
<
...
* Connection #0 to host 0.0.0.0 left intact
4 - Use egctl
egctl is a command line tool to provide additional functionality for Envoy Gateway users.
egctl experimental translate
This subcommand allows users to translate from an input configuration type to an output configuration type.
The translate subcommand can translate Kubernetes resources to:
Gateway API resources This is useful in order to see how validation would occur if these resources were applied to Kubernetes.
Use the
--to gateway-apiparameter to translate to Gateway API resources.Envoy Gateway intermediate representation (IR) This represents Envoy Gateway’s translation of the Gateway API resources.
Use the
--to irparameter to translate to Envoy Gateway intermediate representation.Envoy Proxy xDS This is the xDS configuration provided to Envoy Proxy.
Use the
--to xdsparameter to translate to Envoy Proxy xDS.
In the below example, we will translate the Kubernetes resources (including the Gateway API resources) into xDS resources.
cat <<EOF | egctl x translate --from gateway-api --to xds -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: v1
kind: Namespace
metadata:
name: default
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: default
labels:
app: backend
service: backend
spec:
clusterIP: "1.1.1.1"
type: ClusterIP
ports:
- name: http
port: 3000
targetPort: 3000
protocol: TCP
selector:
app: backend
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: default
spec:
parentRefs:
- name: eg
hostnames:
- "www.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
configKey: default-eg
configs:
- '@type': type.googleapis.com/envoy.admin.v3.BootstrapConfigDump
bootstrap:
admin:
accessLog:
- name: envoy.access_loggers.file
typedConfig:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socketAddress:
address: 127.0.0.1
portValue: 19000
dynamicResources:
cdsConfig:
apiConfigSource:
apiType: DELTA_GRPC
grpcServices:
- envoyGrpc:
clusterName: xds_cluster
setNodeOnFirstMessageOnly: true
transportApiVersion: V3
resourceApiVersion: V3
ldsConfig:
apiConfigSource:
apiType: DELTA_GRPC
grpcServices:
- envoyGrpc:
clusterName: xds_cluster
setNodeOnFirstMessageOnly: true
transportApiVersion: V3
resourceApiVersion: V3
layeredRuntime:
layers:
- name: runtime-0
rtdsLayer:
name: runtime-0
rtdsConfig:
apiConfigSource:
apiType: DELTA_GRPC
grpcServices:
- envoyGrpc:
clusterName: xds_cluster
transportApiVersion: V3
resourceApiVersion: V3
staticResources:
clusters:
- connectTimeout: 10s
loadAssignment:
clusterName: xds_cluster
endpoints:
- lbEndpoints:
- endpoint:
address:
socketAddress:
address: envoy-gateway
portValue: 18000
name: xds_cluster
transportSocket:
name: envoy.transport_sockets.tls
typedConfig:
'@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
commonTlsContext:
tlsCertificateSdsSecretConfigs:
- name: xds_certificate
sdsConfig:
pathConfigSource:
path: /sds/xds-certificate.json
resourceApiVersion: V3
tlsParams:
tlsMaximumProtocolVersion: TLSv1_3
validationContextSdsSecretConfig:
name: xds_trusted_ca
sdsConfig:
pathConfigSource:
path: /sds/xds-trusted-ca.json
resourceApiVersion: V3
type: STRICT_DNS
typedExtensionProtocolOptions:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
'@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicitHttpConfig:
http2ProtocolOptions: {}
- '@type': type.googleapis.com/envoy.admin.v3.ClustersConfigDump
dynamicActiveClusters:
- cluster:
'@type': type.googleapis.com/envoy.config.cluster.v3.Cluster
commonLbConfig:
localityWeightedLbConfig: {}
connectTimeout: 10s
dnsLookupFamily: V4_ONLY
loadAssignment:
clusterName: default-backend-rule-0-match-0-www.example.com
endpoints:
- lbEndpoints:
- endpoint:
address:
socketAddress:
address: 1.1.1.1
portValue: 3000
loadBalancingWeight: 1
loadBalancingWeight: 1
locality: {}
name: default-backend-rule-0-match-0-www.example.com
outlierDetection: {}
type: STATIC
- '@type': type.googleapis.com/envoy.admin.v3.ListenersConfigDump
dynamicListeners:
- activeState:
listener:
'@type': type.googleapis.com/envoy.config.listener.v3.Listener
accessLog:
- filter:
responseFlagFilter:
flags:
- NR
name: envoy.access_loggers.file
typedConfig:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
address:
socketAddress:
address: 0.0.0.0
portValue: 10080
defaultFilterChain:
filters:
- name: envoy.filters.network.http_connection_manager
typedConfig:
'@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
accessLog:
- name: envoy.access_loggers.file
typedConfig:
'@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/stdout
httpFilters:
- name: envoy.filters.http.router
typedConfig:
'@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
rds:
configSource:
apiConfigSource:
apiType: DELTA_GRPC
grpcServices:
- envoyGrpc:
clusterName: xds_cluster
setNodeOnFirstMessageOnly: true
transportApiVersion: V3
resourceApiVersion: V3
routeConfigName: default-eg-http
statPrefix: http
upgradeConfigs:
- upgradeType: websocket
useRemoteAddress: true
name: default-eg-http
- '@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
dynamicRouteConfigs:
- routeConfig:
'@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
name: default-eg-http
virtualHosts:
- domains:
- www.example.com
name: default-eg-http-www.example.com
routes:
- match:
prefix: /
route:
cluster: default-backend-rule-0-match-0-www.example.com
resourceType: all
You can also use the --type/-t flag to retrieve specific output types. In the below example, we will translate the Kubernetes resources (including the Gateway API resources) into xDS route resources.
cat <<EOF | egctl x translate --from gateway-api --to xds -t route -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: v1
kind: Namespace
metadata:
name: default
---
apiVersion: v1
kind: Service
metadata:
name: backend
namespace: default
labels:
app: backend
service: backend
spec:
clusterIP: "1.1.1.1"
type: ClusterIP
ports:
- name: http
port: 3000
targetPort: 3000
protocol: TCP
selector:
app: backend
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: default
spec:
parentRefs:
- name: eg
hostnames:
- "www.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
'@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
configKey: default-eg
dynamicRouteConfigs:
- routeConfig:
'@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
name: default-eg-http
virtualHosts:
- domains:
- www.example.com
name: default-eg-http
routes:
- match:
prefix: /
route:
cluster: default-backend-rule-0-match-0-www.example.com
resourceType: route
Add Missing Resources
You can pass the --add-missing-resources flag to use dummy non Gateway API resources instead of specifying them explicitly.
For example, this will provide the similar result as the above:
cat <<EOF | egctl x translate --add-missing-resources --from gateway-api --to gateway-api -t route -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
namespace: default
spec:
parentRefs:
- name: eg
hostnames:
- "www.example.com"
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
You can see the output contains a EnvoyProxy resource that can be used as a starting point to modify the xDS bootstrap resource for the managed Envoy Proxy fleet.
envoyProxy:
metadata:
creationTimestamp: null
name: default-envoy-proxy
namespace: envoy-gateway-system
spec:
bootstrap: |
admin:
access_log:
- name: envoy.access_loggers.file
typed_config:
"@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
path: /dev/null
address:
socket_address:
address: 127.0.0.1
port_value: 19000
dynamic_resources:
ads_config:
api_type: DELTA_GRPC
transport_api_version: V3
grpc_services:
- envoy_grpc:
cluster_name: xds_cluster
set_node_on_first_message_only: true
lds_config:
ads: {}
resource_api_version: V3
cds_config:
ads: {}
resource_api_version: V3
static_resources:
clusters:
- connect_timeout: 10s
load_assignment:
cluster_name: xds_cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: envoy-gateway
port_value: 18000
typed_extension_protocol_options:
"envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
"@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
"explicit_http_config":
"http2_protocol_options": {}
name: xds_cluster
type: STRICT_DNS
transport_socket:
name: envoy.transport_sockets.tls
typed_config:
"@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
common_tls_context:
tls_params:
tls_maximum_protocol_version: TLSv1_3
tls_certificate_sds_secret_configs:
- name: xds_certificate
sds_config:
path_config_source:
path: "/sds/xds-certificate.json"
resource_api_version: V3
validation_context_sds_secret_config:
name: xds_trusted_ca
sds_config:
path_config_source:
path: "/sds/xds-trusted-ca.json"
resource_api_version: V3
layered_runtime:
layers:
- name: runtime-0
rtds_layer:
rtds_config:
ads: {}
resource_api_version: V3
name: runtime-0
logging: {}
status: {}
gatewayClass:
metadata:
creationTimestamp: null
name: eg
namespace: envoy-gateway-system
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: default-envoy-proxy
namespace: envoy-gateway-system
status:
conditions:
- lastTransitionTime: "2023-04-19T20:30:46Z"
message: Valid GatewayClass
reason: Accepted
status: "True"
type: Accepted
gateways:
- metadata:
creationTimestamp: null
name: eg
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
port: 80
protocol: HTTP
status:
listeners:
- attachedRoutes: 1
conditions:
- lastTransitionTime: "2023-04-19T20:30:46Z"
message: Sending translated listener configuration to the data plane
reason: Programmed
status: "True"
type: Programmed
- lastTransitionTime: "2023-04-19T20:30:46Z"
message: Listener has been successfully translated
reason: Accepted
status: "True"
type: Accepted
name: http
supportedKinds:
- group: gateway.networking.k8s.io
kind: HTTPRoute
- group: gateway.networking.k8s.io
kind: GRPCRoute
httpRoutes:
- metadata:
creationTimestamp: null
name: backend
namespace: default
spec:
hostnames:
- www.example.com
parentRefs:
- name: eg
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
status:
parents:
- conditions:
- lastTransitionTime: "2023-04-19T20:30:46Z"
message: Route is accepted
reason: Accepted
status: "True"
type: Accepted
- lastTransitionTime: "2023-04-19T20:30:46Z"
message: Resolved all the Object references for the Route
reason: ResolvedRefs
status: "True"
type: ResolvedRefs
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
name: eg
Sometimes you might find that egctl doesn’t provide an expected result. For example, the following example provides an empty route resource:
cat <<EOF | egctl x translate --from gateway-api --type route --to xds -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
spec:
gatewayClassName: eg
listeners:
- name: tls
protocol: TLS
port: 8443
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
spec:
parentRefs:
- name: eg
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
xds:
envoy-gateway-system-eg:
'@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
Validating Gateway API Configuration
You can add an additional target gateway-api to show the processed Gateway API resources. For example, translating the above resources with the new argument shows that the HTTPRoute is rejected because there is no ready listener for it:
cat <<EOF | egctl x translate --from gateway-api --type route --to gateway-api,xds -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
spec:
gatewayClassName: eg
listeners:
- name: tls
protocol: TLS
port: 8443
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: backend
spec:
parentRefs:
- name: eg
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
EOF
gatewayClass:
metadata:
creationTimestamp: null
name: eg
namespace: envoy-gateway-system
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
status:
conditions:
- lastTransitionTime: "2023-04-19T20:54:52Z"
message: Valid GatewayClass
reason: Accepted
status: "True"
type: Accepted
gateways:
- metadata:
creationTimestamp: null
name: eg
namespace: envoy-gateway-system
spec:
gatewayClassName: eg
listeners:
- name: tls
port: 8443
protocol: TLS
status:
listeners:
- attachedRoutes: 0
conditions:
- lastTransitionTime: "2023-04-19T20:54:52Z"
message: Listener must have TLS set when protocol is TLS.
reason: Invalid
status: "False"
type: Programmed
name: tls
supportedKinds:
- group: gateway.networking.k8s.io
kind: TLSRoute
httpRoutes:
- metadata:
creationTimestamp: null
name: backend
namespace: envoy-gateway-system
spec:
parentRefs:
- name: eg
rules:
- backendRefs:
- group: ""
kind: Service
name: backend
port: 3000
weight: 1
matches:
- path:
type: PathPrefix
value: /
status:
parents:
- conditions:
- lastTransitionTime: "2023-04-19T20:54:52Z"
message: There are no ready listeners for this parent ref
reason: NoReadyListeners
status: "False"
type: Accepted
- lastTransitionTime: "2023-04-19T20:54:52Z"
message: Service envoy-gateway-system/backend not found
reason: BackendNotFound
status: "False"
type: ResolvedRefs
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parentRef:
name: eg
xds:
envoy-gateway-system-eg:
'@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
egctl experimental status
This subcommand allows users to show the summary of the status of specific or all resource types, in order to quickly find out the status of any resources.
By default, egctl x status display all the conditions for one resource type. You can either add --quiet to only
display the latest condition, or add --verbose to display more details about current status.
Note
The resource types that this subcommand currently supports:
xRoute,HTTPRoute,GRPCRoute,TLSRoute,TCPRoute,UDPRoutexPolicy,BackendTLSPolicy,BackendTrafficPolicy,ClientTrafficPolicy,EnvoyPatchPolicy,SecurityPolicyall,GatewayClass,Gateway
Some examples of this command after installing Multi-tenancy example manifest:
- Show the summary of GatewayClass.
~ egctl x status gatewayclass
NAME TYPE STATUS REASON
eg-marketing Accepted True Accepted
eg-product Accepted True Accepted
- Show the summary of all resource types under all namespaces, the resource type with empty status will be ignored.
~ egctl x status all -A
NAME TYPE STATUS REASON
gatewayclass/eg-marketing Accepted True Accepted
gatewayclass/eg-product Accepted True Accepted
NAMESPACE NAME TYPE STATUS REASON
marketing gateway/eg Programmed True Programmed
Accepted True Accepted
product gateway/eg Programmed True Programmed
Accepted True Accepted
NAMESPACE NAME PARENT TYPE STATUS REASON
marketing httproute/backend gateway/eg ResolvedRefs True ResolvedRefs
Accepted True Accepted
product httproute/backend gateway/eg ResolvedRefs True ResolvedRefs
Accepted True Accepted
- Show the summary of all the Gateways with details under all namespaces.
~ egctl x status gateway --verbose --all-namespaces
NAMESPACE NAME TYPE STATUS REASON MESSAGE OBSERVED GENERATION LAST TRANSITION TIME
marketing eg Programmed True Programmed Address assigned to the Gateway, 1/1 envoy Deployment replicas available 1 2024-02-02 18:17:14 +0800 CST
Accepted True Accepted The Gateway has been scheduled by Envoy Gateway 1 2024-02-01 17:50:39 +0800 CST
product eg Programmed True Programmed Address assigned to the Gateway, 1/1 envoy Deployment replicas available 1 2024-02-02 18:17:14 +0800 CST
Accepted True Accepted The Gateway has been scheduled by Envoy Gateway 1 2024-02-01 17:52:42 +0800 CST
- Show the summary of the latest Gateways condition under
productnamespace.
~ egctl x status gateway --quiet -n product
NAME TYPE STATUS REASON
eg Programmed True Programmed
- Show the summary of latest HTTPRoutes condition under all namespaces.
~ egctl x status httproute --quiet --all-namespaces
NAMESPACE NAME PARENT TYPE STATUS REASON
marketing backend gateway/eg ResolvedRefs True ResolvedRefs
product backend gateway/eg ResolvedRefs True ResolvedRefs
egctl experimental dashboard
This subcommand streamlines the process for users to access the Envoy admin dashboard. By executing the following command:
egctl x dashboard envoy-proxy -n envoy-gateway-system envoy-engw-eg-a9c23fbb-558f94486c-82wh4
You will see the following output:
egctl x dashboard envoy-proxy -n envoy-gateway-system envoy-engw-eg-a9c23fbb-558f94486c-82wh4
http://localhost:19000
the Envoy admin dashboard will automatically open in your default web browser. This eliminates the need to manually locate and expose the admin port.
egctl experimental install
This subcommand can be used to install envoy-gateway.
egctl x install
By default, this will install both the envoy-gateway workload resource and the required gateway-api and envoy-gatewayCRDs.
We can specify to install only workload resources via --skip-crds
egctl x install --skip-crds
We can specify to install only CRDs resources via --only-crds
egctl x install --only-crds
We can specify --name and --namespace to install envoy-gateway in different places to support multi-tenant mode.
Note: If CRDs are already installed, then we need to specify
--skip-crdsto avoid repeated installation of CRDs resources.
egctl x install --name shop-backend --namespace shop
egctl experimental uninstall
This subcommand can be used to uninstall envoy-gateway.
egctl x uninstall
By default, this will only uninstall the envoy-gateway workload resource, if we want to also uninstall CRDs, we need to specify --with-crds
egctl x uninstall --with-crds