This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Operations

This section includes Operations tasks.

1 - Customize EnvoyProxy

Envoy Gateway provides an EnvoyProxy CRD that can be linked to the ParametersRef in a Gateway and GatewayClass, allowing cluster admins to customize the managed EnvoyProxy Deployment and Service. To learn more about GatewayClass and ParametersRef, please refer to Gateway API documentation.

Prerequisites

Follow the steps from the Quickstart task to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.

Verify the Gateway status:

kubectl get gateway/eg -o yaml

egctl x status gateway -v

Before you start, you need to add Infrastructure.ParametersRef in Gateway, and refer to EnvoyProxy Config: Note: MergeGateways cannot be set to true in your EnvoyProxy config if attaching to the Gateway.

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: custom-proxy-config
  listeners:
    - name: http
      protocol: HTTP
      port: 80
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  infrastructure:
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: custom-proxy-config
  listeners:
    - name: http
      protocol: HTTP
      port: 80

You can also attach the EnvoyProxy resource to the GatewayClass using the parametersRef field. This configuration is discouraged if you plan on creating multiple Gateways linking to the same GatewayClass and would like different infrastructure configurations for each of them.

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: custom-proxy-config
    namespace: default
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: custom-proxy-config
    namespace: default

Customize EnvoyProxy Deployment Replicas

You can customize the EnvoyProxy Deployment Replicas via EnvoyProxy Config like:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 2
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        replicas: 2

After you apply the config, you should see the replicas of envoyproxy changes to 2. And also you can dynamically change the value.

kubectl get deployment -l gateway.envoyproxy.io/owning-gateway-name=eg -n envoy-gateway-system

Customize EnvoyProxy Image

You can customize the EnvoyProxy Image via EnvoyProxy Config like:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        container:
          image: envoyproxy/envoy:v1.25-latest
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        container:
          image: envoyproxy/envoy:v1.25-latest

After applying the config, you can get the deployment image, and see it has changed.

Customize EnvoyProxy Pod Annotations

You can customize the EnvoyProxy Pod Annotations via EnvoyProxy Config like:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        pod:
          annotations:
            custom1: deploy-annotation1
            custom2: deploy-annotation2
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        pod:
          annotations:
            custom1: deploy-annotation1
            custom2: deploy-annotation2

After applying the config, you can get the envoyproxy pods, and see new annotations has been added.

Customize EnvoyProxy Deployment Resources

You can customize the EnvoyProxy Deployment Resources via EnvoyProxy Config like:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        container:
          resources:
            requests:
              cpu: 150m
              memory: 640Mi
            limits:
              cpu: 500m
              memory: 1Gi
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        container:
          resources:
            requests:
              cpu: 150m
              memory: 640Mi
            limits:
              cpu: 500m
              memory: 1Gi

Customize EnvoyProxy Deployment Env

You can customize the EnvoyProxy Deployment Env via EnvoyProxy Config like:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        container:
          env:
          - name: env_a
            value: env_a_value
          - name: env_b
            value: env_b_value
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        container:
          env:
          - name: env_a
            value: env_a_value
          - name: env_b
            value: env_b_value

Envoy Gateway has provided two initial env ENVOY_GATEWAY_NAMESPACE and ENVOY_POD_NAME for envoyproxy container.

After applying the config, you can get the envoyproxy deployment, and see resources has been changed.

Customize EnvoyProxy Deployment Volumes or VolumeMounts

You can customize the EnvoyProxy Deployment Volumes or VolumeMounts via EnvoyProxy Config like:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        container:
          volumeMounts:
          - mountPath: /certs
            name: certs
            readOnly: true
        pod:
          volumes:
          - name: certs
            secret:
              secretName: envoy-cert
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        container:
          volumeMounts:
          - mountPath: /certs
            name: certs
            readOnly: true
        pod:
          volumes:
          - name: certs
            secret:
              secretName: envoy-cert

After applying the config, you can get the envoyproxy deployment, and see resources has been changed.

Customize EnvoyProxy Service Annotations

You can customize the EnvoyProxy Service Annotations via EnvoyProxy Config like:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        annotations:
          custom1: svc-annotation1
          custom2: svc-annotation2
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        annotations:
          custom1: svc-annotation1
          custom2: svc-annotation2

After applying the config, you can get the envoyproxy service, and see annotations has been added.

Customize EnvoyProxy Bootstrap Config

You can customize the EnvoyProxy bootstrap config via EnvoyProxy Config. There are two ways to customize it:

  • Replace: the whole bootstrap config will be replaced by the config you provided.
  • Merge: the config you provided will be merged into the default bootstrap config.
cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default 
spec:
  bootstrap:
    type: Replace
    value: |
      admin:
        access_log:
        - name: envoy.access_loggers.file
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/null
        address:
          socket_address:
            address: 127.0.0.1
            port_value: 20000
      dynamic_resources:
        ads_config:
          api_type: DELTA_GRPC
          transport_api_version: V3
          grpc_services:
          - envoy_grpc:
              cluster_name: xds_cluster
          set_node_on_first_message_only: true
        lds_config:
          ads: {}
          resource_api_version: V3
        cds_config:
          ads: {}
          resource_api_version: V3
      static_resources:
        clusters:
        - connect_timeout: 10s
          load_assignment:
            cluster_name: xds_cluster
            endpoints:
            - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: envoy-gateway
                      port_value: 18000
          typed_extension_protocol_options:
            "envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
               "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
               "explicit_http_config":
                 "http2_protocol_options": {}
          name: xds_cluster
          type: STRICT_DNS
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
              common_tls_context:
                tls_params:
                  tls_maximum_protocol_version: TLSv1_3
                tls_certificate_sds_secret_configs:
                - name: xds_certificate
                  sds_config:
                    path_config_source:
                      path: "/sds/xds-certificate.json"
                    resource_api_version: V3
                validation_context_sds_secret_config:
                  name: xds_trusted_ca
                  sds_config:
                    path_config_source:
                      path: "/sds/xds-trusted-ca.json"
                    resource_api_version: V3
      layered_runtime:
        layers:
        - name: runtime-0
          rtds_layer:
            rtds_config:
              ads: {}
              resource_api_version: V3
            name: runtime-0
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  bootstrap:
    type: Replace
    value: |
      admin:
        access_log:
        - name: envoy.access_loggers.file
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/null
        address:
          socket_address:
            address: 127.0.0.1
            port_value: 20000
      dynamic_resources:
        ads_config:
          api_type: DELTA_GRPC
          transport_api_version: V3
          grpc_services:
          - envoy_grpc:
              cluster_name: xds_cluster
          set_node_on_first_message_only: true
        lds_config:
          ads: {}
          resource_api_version: V3
        cds_config:
          ads: {}
          resource_api_version: V3
      static_resources:
        clusters:
        - connect_timeout: 10s
          load_assignment:
            cluster_name: xds_cluster
            endpoints:
            - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: envoy-gateway
                      port_value: 18000
          typed_extension_protocol_options:
            "envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
               "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
               "explicit_http_config":
                 "http2_protocol_options": {}
          name: xds_cluster
          type: STRICT_DNS
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
              common_tls_context:
                tls_params:
                  tls_maximum_protocol_version: TLSv1_3
                tls_certificate_sds_secret_configs:
                - name: xds_certificate
                  sds_config:
                    path_config_source:
                      path: "/sds/xds-certificate.json"
                    resource_api_version: V3
                validation_context_sds_secret_config:
                  name: xds_trusted_ca
                  sds_config:
                    path_config_source:
                      path: "/sds/xds-trusted-ca.json"
                    resource_api_version: V3
      layered_runtime:
        layers:
        - name: runtime-0
          rtds_layer:
            rtds_config:
              ads: {}
              resource_api_version: V3
            name: runtime-0

You can use egctl translate to get the default xDS Bootstrap configuration used by Envoy Gateway.

After applying the config, the bootstrap config will be overridden by the new config you provided. Any errors in the configuration will be surfaced as status within the GatewayClass resource. You can also validate this configuration using egctl translate.

Customize EnvoyProxy Horizontal Pod Autoscaler

You can enable Horizontal Pod Autoscaler for EnvoyProxy Deployment. However, before enabling the HPA for EnvoyProxy, please ensure that the metrics-server component is installed in the cluster.

Once confirmed, you can apply it via EnvoyProxy Config as shown below:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default 
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyHpa:
        minReplicas: 2
        maxReplicas: 10
        metrics:
          - resource:
              name: cpu
              target:
                averageUtilization: 60
                type: Utilization
            type: Resource
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default 
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyHpa:
        minReplicas: 2
        maxReplicas: 10
        metrics:
          - resource:
              name: cpu
              target:
                averageUtilization: 60
                type: Utilization
            type: Resource

After applying the config, the EnvoyProxy HPA (Horizontal Pod Autoscaler) is generated. However, upon activating the EnvoyProxy’s HPA, the Envoy Gateway will no longer reference the replicas field specified in the envoyDeployment, as outlined here.

Customize EnvoyProxy Command line options

You can customize the EnvoyProxy Command line options via spec.extraArgs in EnvoyProxy Config. For example, the following configuration will add --disable-extensions arg in order to disable envoy.access_loggers/envoy.access_loggers.wasm extension:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default 
spec:
  extraArgs:
    - --disable-extensions envoy.access_loggers/envoy.access_loggers.wasm 
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default 
spec:
  extraArgs:
    - --disable-extensions envoy.access_loggers/envoy.access_loggers.wasm

Customize EnvoyProxy with Patches

You can customize the EnvoyProxy using patches.

Patching Deployment for EnvoyProxy

For example, the following configuration will add resource limits to the envoy and the shutdown-manager containers in the envoyproxy deployment:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: eg
  namespace: default 
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        patch:
          type: StrategicMerge
          value:
            spec:
              template:
                spec:
                  containers:
                  - name: envoy
                    resources:
                      limits:
                        cpu: 500m
                        memory: 1024Mi
                  - name: shutdown-manager
                    resources:
                      limits:
                        cpu: 200m
                        memory: 1024Mi
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: eg
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        patch:
          type: StrategicMerge
          value:
            spec:
              template:
                spec:
                  containers:
                  - name: envoy
                    resources:
                      limits:
                        cpu: 500m
                        memory: 1024Mi
                  - name: shutdown-manager
                    resources:
                      limits:
                        cpu: 200m
                        memory: 1024Mi

After applying the configuration, you will see the change in both containers in the envoyproxy deployment.

Patching Service for EnvoyProxy

For example, the following configuration will add an annotation for the envoyproxy service:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: eg
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        patch:
          type: StrategicMerge
          value:
            metadata:
              annotations:
                custom-annotation: foobar
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: eg
  namespace: default
spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyService:
        patch:
          type: StrategicMerge
          value:
            metadata:
              annotations:
                custom-annotation: foobar

After applying the configuration, you will see the custom-annotation: foobar has been added to the envoyproxy service.

Customize Filter Order

Under the hood, Envoy Gateway uses a series of Envoy HTTP filters to process HTTP requests and responses, and to apply various policies.

By default, Envoy Gateway applies the following filters in the order shown:

  • envoy.filters.http.fault
  • envoy.filters.http.cors
  • envoy.filters.http.ext_authz
  • envoy.filters.http.basic_authn
  • envoy.filters.http.oauth2
  • envoy.filters.http.jwt_authn
  • envoy.filters.http.ext_proc
  • envoy.filters.http.wasm
  • envoy.filters.http.rbac
  • envoy.filters.http.local_ratelimit
  • envoy.filters.http.ratelimit
  • envoy.filters.http.router

The default order in which these filters are applied is opinionated and may not suit all use cases. To address this, Envoy Gateway allows you to adjust the execution order of these filters with the filterOrder field in the EnvoyProxy resource.

filterOrder is a list of customized filter order configurations. Each configuration can specify a filter name and a filter to place it before or after. These configurations are applied in the order they are listed. If a filter occurs in multiple configurations, the final order is the result of applying all these configurations in order. To avoid conflicts, it is recommended to only specify one configuration per filter.

For example, the following configuration moves the envoy.filters.http.wasm filter before the envoy.filters.http.jwt_authn filter and the envoy.filters.http.cors filter after the envoy.filters.http.basic_authn filter:

cat <<EOF | kubectl apply -f -
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  filterOrder:
    - name: envoy.filters.http.wasm
      before: envoy.filters.http.jwt_authn
    - name: envoy.filters.http.cors
      after: envoy.filters.http.basic_authn
EOF

Save and apply the following resource to your cluster:

---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: default
spec:
  filterOrder:
    - name: envoy.filters.http.wasm
      before: envoy.filters.http.jwt_authn
    - name: envoy.filters.http.cors
      after: envoy.filters.http.basic_authn

2 - Deployment Mode

Deployment modes

One GatewayClass per Envoy Gateway Controller

  • An Envoy Gateway is associated with a single GatewayClass resource under one controller. This is the simplest deployment mode and is suitable for scenarios where each Gateway needs to have its own dedicated set of resources and configurations.

Multiple GatewayClasses per Envoy Gateway Controller

  • An Envoy Gateway is associated with multiple GatewayClass resources under one controller.
  • Support for accepting multiple GatewayClasses was added here.

Separate Envoy Gateway Controllers

If you’ve instantiated multiple GatewayClasses, you can also run separate Envoy Gateway controllers in different namespaces, linking a GatewayClass to each of them for multi-tenancy. Please follow the example Multi-tenancy.

Merged Gateways onto a single EnvoyProxy fleet

By default, each Gateway has its own dedicated set of Envoy Proxy and its configurations. However, for some deployments, it may be more convenient to merge listeners across multiple Gateways and deploy a single Envoy Proxy fleet.

This can help to efficiently utilize the infra resources in the cluster and manage them in a centralized manner, or have a single IP address for all of the listeners. Setting the mergeGateways field in the EnvoyProxy resource linked to GatewayClass will result in merging all Gateway listeners under one GatewayClass resource.

  • The tuple of port, protocol, and hostname must be unique across all Listeners.

Please follow the example Merged gateways deployment.

Supported Modes

Kubernetes

  • The default deployment model is - Envoy Gateway watches for resources such a Service & HTTPRoute in all namespaces and creates managed data plane resources such as EnvoyProxy Deployment in the namespace where Envoy Gateway is running.
  • Envoy Gateway also supports Namespaced deployment mode, you can watch resources in the specific namespaces by assigning EnvoyGateway.provider.kubernetes.watch.namespaces or EnvoyGateway.provider.kubernetes.watch.namespaceSelector and creates managed data plane resources in the namespace where Envoy Gateway is running.
  • Support for alternate deployment modes is being tracked here.

Multi-tenancy

Kubernetes

  • A tenant is a group within an organization (e.g. a team or department) who shares organizational resources. We recommend each tenant deploy their own Envoy Gateway controller in their respective namespace. Below is an example of deploying Envoy Gateway by the marketing and product teams in separate namespaces.

  • Lets deploy Envoy Gateway in the marketing namespace and also watch resources only in this namespace. We are also setting the controller name to a unique string here gateway.envoyproxy.io/marketing-gatewayclass-controller.

helm install \
--set config.envoyGateway.gateway.controllerName=gateway.envoyproxy.io/marketing-gatewayclass-controller \
--set config.envoyGateway.provider.kubernetes.watch.type=Namespaces \
--set config.envoyGateway.provider.kubernetes.watch.namespaces={marketing} \
eg-marketing oci://docker.io/envoyproxy/gateway-helm \
--version v1.2.1 -n marketing --create-namespace

Lets create a GatewayClass linked to the marketing team’s Envoy Gateway controller, and as well other resources linked to it, so the backend application operated by this team can be exposed to external clients.

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg-marketing
spec:
  controllerName: gateway.envoyproxy.io/marketing-gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: marketing
spec:
  gatewayClassName: eg-marketing
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: backend
  namespace: marketing
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: marketing
  labels:
    app: backend
    service: backend
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: marketing
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
      version: v1
  template:
    metadata:
      labels:
        app: backend
        version: v1
    spec:
      serviceAccountName: backend
      containers:
        - image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
          imagePullPolicy: IfNotPresent
          name: backend
          ports:
            - containerPort: 3000
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
  namespace: marketing
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.marketing.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
EOF

Save and apply the following resources to your cluster:

---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg-marketing
spec:
  controllerName: gateway.envoyproxy.io/marketing-gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: marketing
spec:
  gatewayClassName: eg-marketing
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: backend
  namespace: marketing
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: marketing
  labels:
    app: backend
    service: backend
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: marketing
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
      version: v1
  template:
    metadata:
      labels:
        app: backend
        version: v1
    spec:
      serviceAccountName: backend
      containers:
        - image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
          imagePullPolicy: IfNotPresent
          name: backend
          ports:
            - containerPort: 3000
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
  namespace: marketing
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.marketing.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

Lets port forward to the generated envoy proxy service in the marketing namespace and send a request to it.

export ENVOY_SERVICE=$(kubectl get svc -n marketing --selector=gateway.envoyproxy.io/owning-gateway-namespace=marketing,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
kubectl -n marketing port-forward service/${ENVOY_SERVICE} 8888:8080 &

curl --verbose --header "Host: www.marketing.example.com" http://localhost:8888/get

*   Trying 127.0.0.1:8888...
* Connected to localhost (127.0.0.1) port 8888 (#0)
> GET /get HTTP/1.1
> Host: www.marketing.example.com
> User-Agent: curl/7.86.0
> Accept: */*
>
Handling connection for 8888
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Thu, 20 Apr 2023 19:19:42 GMT
< content-length: 521
< x-envoy-upstream-service-time: 0
< server: envoy
<
{
 "path": "/get",
 "host": "www.marketing.example.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/7.86.0"
  ],
  "X-Envoy-Expected-Rq-Timeout-Ms": [
   "15000"
  ],
  "X-Envoy-Internal": [
   "true"
  ],
  "X-Forwarded-For": [
   "10.1.0.157"
  ],
  "X-Forwarded-Proto": [
   "http"
  ],
  "X-Request-Id": [
   "c637977c-458a-48ae-92b3-f8c429849322"
  ]
 },
 "namespace": "marketing",
 "ingress": "",
 "service": "",
 "pod": "backend-74888f465f-bcs8f"
* Connection #0 to host localhost left intact
  • Lets deploy Envoy Gateway in the product namespace and also watch resources only in this namespace.
helm install \
--set config.envoyGateway.gateway.controllerName=gateway.envoyproxy.io/product-gatewayclass-controller \
--set config.envoyGateway.provider.kubernetes.watch.type=Namespaces \
--set config.envoyGateway.provider.kubernetes.watch.namespaces={product} \
eg-product oci://docker.io/envoyproxy/gateway-helm \
--version v1.2.1 -n product --create-namespace

Lets create a GatewayClass linked to the product team’s Envoy Gateway controller, and as well other resources linked to it, so the backend application operated by this team can be exposed to external clients.

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg-product
spec:
  controllerName: gateway.envoyproxy.io/product-gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: product
spec:
  gatewayClassName: eg-product
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: backend
  namespace: product
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: product
  labels:
    app: backend
    service: backend
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: product
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
      version: v1
  template:
    metadata:
      labels:
        app: backend
        version: v1
    spec:
      serviceAccountName: backend
      containers:
        - image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
          imagePullPolicy: IfNotPresent
          name: backend
          ports:
            - containerPort: 3000
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
  namespace: product
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.product.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
EOF

Save and apply the following resources to your cluster:

---
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg-product
spec:
  controllerName: gateway.envoyproxy.io/product-gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: product
spec:
  gatewayClassName: eg-product
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: backend
  namespace: product
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: product
  labels:
    app: backend
    service: backend
spec:
  ports:
    - name: http
      port: 3000
      targetPort: 3000
  selector:
    app: backend
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: backend
  namespace: product
spec:
  replicas: 1
  selector:
    matchLabels:
      app: backend
      version: v1
  template:
    metadata:
      labels:
        app: backend
        version: v1
    spec:
      serviceAccountName: backend
      containers:
        - image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
          imagePullPolicy: IfNotPresent
          name: backend
          ports:
            - containerPort: 3000
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
  namespace: product
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.product.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

Lets port forward to the generated envoy proxy service in the product namespace and send a request to it.

export ENVOY_SERVICE=$(kubectl get svc -n product --selector=gateway.envoyproxy.io/owning-gateway-namespace=product,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')
kubectl -n product port-forward service/${ENVOY_SERVICE} 8889:8080 &

curl --verbose --header "Host: www.product.example.com" http://localhost:8889/get

*   Trying 127.0.0.1:8889...
* Connected to localhost (127.0.0.1) port 8889 (#0)
> GET /get HTTP/1.1
> Host: www.product.example.com
> User-Agent: curl/7.86.0
> Accept: */*
>
Handling connection for 8889
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< content-type: application/json
< x-content-type-options: nosniff
< date: Thu, 20 Apr 2023 19:20:17 GMT
< content-length: 517
< x-envoy-upstream-service-time: 0
< server: envoy
<
{
 "path": "/get",
 "host": "www.product.example.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/7.86.0"
  ],
  "X-Envoy-Expected-Rq-Timeout-Ms": [
   "15000"
  ],
  "X-Envoy-Internal": [
   "true"
  ],
  "X-Forwarded-For": [
   "10.1.0.156"
  ],
  "X-Forwarded-Proto": [
   "http"
  ],
  "X-Request-Id": [
   "39196453-2250-4331-b756-54003b2853c2"
  ]
 },
 "namespace": "product",
 "ingress": "",
 "service": "",
 "pod": "backend-74888f465f-64fjs"
* Connection #0 to host localhost left intact

With the below command you can ensure that you are no able to access the marketing team’s backend exposed using the www.marketing.example.com hostname and the product team’s data plane.

curl --verbose --header "Host: www.marketing.example.com" http://localhost:8889/get

*   Trying 127.0.0.1:8889...
* Connected to localhost (127.0.0.1) port 8889 (#0)
> GET /get HTTP/1.1
> Host: www.marketing.example.com
> User-Agent: curl/7.86.0
> Accept: */*
>
Handling connection for 8889
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< date: Thu, 20 Apr 2023 19:22:13 GMT
< server: envoy
< content-length: 0
<
* Connection #0 to host localhost left intact

Merged gateways deployment

In this example, we will deploy GatewayClass

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: merged-eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
  parametersRef:
    group: gateway.envoyproxy.io
    kind: EnvoyProxy
    name: custom-proxy-config
    namespace: envoy-gateway-system

with a referenced EnvoyProxy resource configured to enable merged Gateways deployment mode.

apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
  name: custom-proxy-config
  namespace: envoy-gateway-system
spec:
  mergeGateways: true

Deploy merged-gateways example

Deploy resources on your cluster from the example.

kubectl apply -f https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/merged-gateways.yaml

Verify that Gateways are deployed and programmed

kubectl get gateways -n default

NAMESPACE   NAME          CLASS       ADDRESS          PROGRAMMED   AGE
default     merged-eg-1   merged-eg   172.18.255.202   True         2m4s
default     merged-eg-2   merged-eg   172.18.255.202   True         2m4s
default     merged-eg-3   merged-eg   172.18.255.202   True         2m4s

Verify that HTTPRoutes are deployed

kubectl get httproute -n default
NAMESPACE   NAME              HOSTNAMES             AGE
default     hostname1-route   ["www.merged1.com"]   2m4s
default     hostname2-route   ["www.merged2.com"]   2m4s
default     hostname3-route   ["www.merged3.com"]   2m4s

If you take a look at the deployed Envoy Proxy service you would notice that all of the Gateway listeners ports are added to that service.

kubectl get service -n envoy-gateway-system
NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                        AGE
envoy-gateway                   ClusterIP      10.96.141.4     <none>           18000/TCP,18001/TCP                            6m43s
envoy-gateway-metrics-service   ClusterIP      10.96.113.191   <none>           19001/TCP                                      6m43s
envoy-merged-eg-668ac7ae        LoadBalancer   10.96.48.255    172.18.255.202   8081:30467/TCP,8082:31793/TCP,8080:31153/TCP   3m17s

There should be also one deployment (envoy-merged-eg-668ac7ae-775f9865d-55zhs) for every Gateway and its name should reference the name of the GatewayClass.

kubectl get pods -n envoy-gateway-system
NAME                                       READY   STATUS    RESTARTS       AGE
envoy-gateway-5d998778f6-wr6m9             1/1     Running   0              6m43s
envoy-merged-eg-668ac7ae-775f9865d-55zhs   2/2     Running   0              3m17s

Testing the Configuration

Get the name of the merged gateways Envoy service:

export ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gatewayclass=merged-eg -o jsonpath='{.items[0].metadata.name}')

Fetch external IP of the service:

export GATEWAY_HOST=$(kubectl get svc/${ENVOY_SERVICE} -n envoy-gateway-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

In certain environments, the load balancer may be exposed using a hostname, instead of an IP address. If so, replace ip in the above command with hostname.

Curl the route hostname-route2 through Envoy proxy:

curl --header "Host: www.merged2.com" http://$GATEWAY_HOST:8081/example2

{
 "path": "/example2",
 "host": "www.merged2.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/8.4.0"
  ],
  "X-Envoy-Internal": [
   "true"
  ],
  "X-Forwarded-For": [
   "172.18.0.2"
  ],
  "X-Forwarded-Proto": [
   "http"
  ],
  "X-Request-Id": [
   "deed2767-a483-4291-9429-0e256ab3a65f"
  ]
 },
 "namespace": "default",
 "ingress": "",
 "service": "",
 "pod": "merged-backend-64ddb65fd7-ttv5z"
}

Curl the route hostname-route1 through Envoy proxy:

curl --header "Host: www.merged1.com" http://$GATEWAY_HOST:8080/example

{
 "path": "/example",
 "host": "www.merged1.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/8.4.0"
  ],
  "X-Envoy-Internal": [
   "true"
  ],
  "X-Forwarded-For": [
   "172.18.0.2"
  ],
  "X-Forwarded-Proto": [
   "http"
  ],
  "X-Request-Id": [
   "20a53440-6327-4c3c-bc8b-8e79e7311043"
  ]
 },
 "namespace": "default",
 "ingress": "",
 "service": "",
 "pod": "merged-backend-64ddb65fd7-ttv5z"
}

Verify deployment of multiple GatewayClass

Install the GatewayClass, Gateway, HTTPRoute and example app from Quickstart example:

kubectl apply -f https://github.com/envoyproxy/gateway/releases/download/v1.2.1/quickstart.yaml -n default

Lets create also and additional Gateway linked to the GatewayClass and backend application from Quickstart example.

cat <<EOF | kubectl apply -f -
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-2
  namespace: default
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: eg-2
  namespace: default
spec:
  parentRefs:
    - name: eg-2
  hostnames:
    - "www.quickstart.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
EOF

Save and apply the following resources to your cluster:

---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg-2
  namespace: default
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 8080
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: eg-2
  namespace: default
spec:
  parentRefs:
    - name: eg-2
  hostnames:
    - "www.quickstart.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /

Verify that Gateways are deployed and programmed

kubectl get gateways -n default

NAME          CLASS       ADDRESS          PROGRAMMED   AGE
eg            eg          172.18.255.203   True         114s
eg-2          eg          172.18.255.204   True         89s
merged-eg-1   merged-eg   172.18.255.202   True         8m33s
merged-eg-2   merged-eg   172.18.255.202   True         8m33s
merged-eg-3   merged-eg   172.18.255.202   True         8m33s

Verify that HTTPRoutes are deployed

kubectl get httproute -n default

NAMESPACE   NAME              HOSTNAMES                        AGE
default     backend           ["www.example.com"]              2m29s
default     eg-2              ["www.quickstart.example.com"]   87s
default     hostname1-route   ["www.merged1.com"]              10m4s
default     hostname2-route   ["www.merged2.com"]              10m4s
default     hostname3-route   ["www.merged3.com"]              10m4s

Verify that services are now deployed separately.

kubectl get service -n envoy-gateway-system

NAME                            TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                        AGE
envoy-default-eg-2-7e515b2f     LoadBalancer   10.96.121.46    172.18.255.204   8080:32705/TCP                                 3m27s
envoy-default-eg-e41e7b31       LoadBalancer   10.96.11.244    172.18.255.203   80:31930/TCP                                   2m26s
envoy-gateway                   ClusterIP      10.96.141.4     <none>           18000/TCP,18001/TCP                            14m25s
envoy-gateway-metrics-service   ClusterIP      10.96.113.191   <none>           19001/TCP                                      14m25s
envoy-merged-eg-668ac7ae        LoadBalancer   10.96.243.32    172.18.255.202   8082:31622/TCP,8080:32262/TCP,8081:32305/TCP   10m59s

There should be two deployments for each of newly deployed Gateway and its name should reference the name of the namespace and the Gateway.

kubectl get pods -n envoy-gateway-system

NAME                                          READY   STATUS    RESTARTS   AGE
envoy-default-eg-2-7e515b2f-8c98fdf88-p6jhg   2/2     Running   0          3m27s
envoy-default-eg-e41e7b31-6f998d85d7-jpvmj    2/2     Running   0          2m26s
envoy-gateway-5d998778f6-wr6m9                1/1     Running   0          14m25s
envoy-merged-eg-668ac7ae-5958f7b7f6-9h9v2     2/2     Running   0          10m59s

Testing the Configuration

Get the name of the merged gateways Envoy service:

export DEFAULT_ENVOY_SERVICE=$(kubectl get svc -n envoy-gateway-system --selector=gateway.envoyproxy.io/owning-gateway-namespace=default,gateway.envoyproxy.io/owning-gateway-name=eg -o jsonpath='{.items[0].metadata.name}')

Fetch external IP of the service:

export DEFAULT_GATEWAY_HOST=$(kubectl get svc/${DEFAULT_ENVOY_SERVICE} -n envoy-gateway-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

Curl the route Quickstart backend route through Envoy proxy:

curl --header "Host: www.example.com" http://$DEFAULT_GATEWAY_HOST

{
 "path": "/",
 "host": "www.example.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/8.4.0"
  ],
  "X-Envoy-Internal": [
   "true"
  ],
  "X-Forwarded-For": [
   "172.18.0.2"
  ],
  "X-Forwarded-Proto": [
   "http"
  ],
  "X-Request-Id": [
   "70a40595-67a1-4776-955b-2dee361baed7"
  ]
 },
 "namespace": "default",
 "ingress": "",
 "service": "",
 "pod": "backend-96f75bbf-6w67z"
}

Curl the route hostname-route3 through Envoy proxy:

curl --header "Host: www.merged3.com" http://$GATEWAY_HOST:8082/example3

{
 "path": "/example3",
 "host": "www.merged3.com",
 "method": "GET",
 "proto": "HTTP/1.1",
 "headers": {
  "Accept": [
   "*/*"
  ],
  "User-Agent": [
   "curl/8.4.0"
  ],
  "X-Envoy-Internal": [
   "true"
  ],
  "X-Forwarded-For": [
   "172.18.0.2"
  ],
  "X-Forwarded-Proto": [
   "http"
  ],
  "X-Request-Id": [
   "47aeaef3-abb5-481a-ab92-c2ae3d0862d6"
  ]
 },
 "namespace": "default",
 "ingress": "",
 "service": "",
 "pod": "merged-backend-64ddb65fd7-k84gv"
}

3 - Use egctl

egctl is a command line tool to provide additional functionality for Envoy Gateway users.

egctl experimental translate

This subcommand allows users to translate from an input configuration type to an output configuration type.

The translate subcommand can translate Kubernetes resources to:

  • Gateway API resources This is useful in order to see how validation would occur if these resources were applied to Kubernetes.

    Use the --to gateway-api parameter to translate to Gateway API resources.

  • Envoy Gateway intermediate representation (IR) This represents Envoy Gateway’s translation of the Gateway API resources.

    Use the --to ir parameter to translate to Envoy Gateway intermediate representation.

  • Envoy Proxy xDS This is the xDS configuration provided to Envoy Proxy.

    Use the --to xds parameter to translate to Envoy Proxy xDS.

In the below example, we will translate the Kubernetes resources (including the Gateway API resources) into xDS resources.

cat <<EOF | egctl x translate --from gateway-api --to xds -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: default
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
---
apiVersion: v1
kind: Namespace
metadata:
  name: default 
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: default
  labels:
    app: backend
    service: backend
spec:
  clusterIP: "1.1.1.1"
  type: ClusterIP
  ports:
    - name: http
      port: 3000
      targetPort: 3000
      protocol: TCP
  selector:
    app: backend
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
  namespace: default
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
EOF

configKey: default-eg
configs:
- '@type': type.googleapis.com/envoy.admin.v3.BootstrapConfigDump
  bootstrap:
    admin:
      accessLog:
      - name: envoy.access_loggers.file
        typedConfig:
          '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
          path: /dev/null
      address:
        socketAddress:
          address: 127.0.0.1
          portValue: 19000
    dynamicResources:
      cdsConfig:
        apiConfigSource:
          apiType: DELTA_GRPC
          grpcServices:
          - envoyGrpc:
              clusterName: xds_cluster
          setNodeOnFirstMessageOnly: true
          transportApiVersion: V3
        resourceApiVersion: V3
      ldsConfig:
        apiConfigSource:
          apiType: DELTA_GRPC
          grpcServices:
          - envoyGrpc:
              clusterName: xds_cluster
          setNodeOnFirstMessageOnly: true
          transportApiVersion: V3
        resourceApiVersion: V3
    layeredRuntime:
      layers:
      - name: runtime-0
        rtdsLayer:
          name: runtime-0
          rtdsConfig:
            apiConfigSource:
              apiType: DELTA_GRPC
              grpcServices:
              - envoyGrpc:
                  clusterName: xds_cluster
              transportApiVersion: V3
            resourceApiVersion: V3
    staticResources:
      clusters:
      - connectTimeout: 10s
        loadAssignment:
          clusterName: xds_cluster
          endpoints:
          - lbEndpoints:
            - endpoint:
                address:
                  socketAddress:
                    address: envoy-gateway
                    portValue: 18000
        name: xds_cluster
        transportSocket:
          name: envoy.transport_sockets.tls
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
            commonTlsContext:
              tlsCertificateSdsSecretConfigs:
              - name: xds_certificate
                sdsConfig:
                  pathConfigSource:
                    path: /sds/xds-certificate.json
                  resourceApiVersion: V3
              tlsParams:
                tlsMaximumProtocolVersion: TLSv1_3
              validationContextSdsSecretConfig:
                name: xds_trusted_ca
                sdsConfig:
                  pathConfigSource:
                    path: /sds/xds-trusted-ca.json
                  resourceApiVersion: V3
        type: STRICT_DNS
        typedExtensionProtocolOptions:
          envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
            '@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
            explicitHttpConfig:
              http2ProtocolOptions: {}
- '@type': type.googleapis.com/envoy.admin.v3.ClustersConfigDump
  dynamicActiveClusters:
  - cluster:
      '@type': type.googleapis.com/envoy.config.cluster.v3.Cluster
      commonLbConfig:
        localityWeightedLbConfig: {}
      connectTimeout: 10s
      dnsLookupFamily: V4_ONLY
      loadAssignment:
        clusterName: default-backend-rule-0-match-0-www.example.com
        endpoints:
        - lbEndpoints:
          - endpoint:
              address:
                socketAddress:
                  address: 1.1.1.1
                  portValue: 3000
            loadBalancingWeight: 1
          loadBalancingWeight: 1
          locality: {}
      name: default-backend-rule-0-match-0-www.example.com
      outlierDetection: {}
      type: STATIC
- '@type': type.googleapis.com/envoy.admin.v3.ListenersConfigDump
  dynamicListeners:
  - activeState:
      listener:
        '@type': type.googleapis.com/envoy.config.listener.v3.Listener
        accessLog:
        - filter:
            responseFlagFilter:
              flags:
              - NR
          name: envoy.access_loggers.file
          typedConfig:
            '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/stdout
        address:
          socketAddress:
            address: 0.0.0.0
            portValue: 10080
        defaultFilterChain:
          filters:
          - name: envoy.filters.network.http_connection_manager
            typedConfig:
              '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
              accessLog:
              - name: envoy.access_loggers.file
                typedConfig:
                  '@type': type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
                  path: /dev/stdout
              httpFilters:
              - name: envoy.filters.http.router
                typedConfig:
                  '@type': type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
              rds:
                configSource:
                  apiConfigSource:
                    apiType: DELTA_GRPC
                    grpcServices:
                    - envoyGrpc:
                        clusterName: xds_cluster
                    setNodeOnFirstMessageOnly: true
                    transportApiVersion: V3
                  resourceApiVersion: V3
                routeConfigName: default-eg-http
              statPrefix: http
              upgradeConfigs:
              - upgradeType: websocket
              useRemoteAddress: true
        name: default-eg-http
- '@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
  dynamicRouteConfigs:
  - routeConfig:
      '@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
      name: default-eg-http
      virtualHosts:
      - domains:
        - www.example.com
        name: default-eg-http-www.example.com
        routes:
        - match:
            prefix: /
          route:
            cluster: default-backend-rule-0-match-0-www.example.com
resourceType: all

You can also use the --type/-t flag to retrieve specific output types. In the below example, we will translate the Kubernetes resources (including the Gateway API resources) into xDS route resources.

cat <<EOF | egctl x translate --from gateway-api --to xds -t route -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: default
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
---
apiVersion: v1
kind: Namespace
metadata:
  name: default 
---
apiVersion: v1
kind: Service
metadata:
  name: backend
  namespace: default
  labels:
    app: backend
    service: backend
spec:
  clusterIP: "1.1.1.1"
  type: ClusterIP
  ports:
    - name: http
      port: 3000
      targetPort: 3000
      protocol: TCP
  selector:
    app: backend
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
  namespace: default
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
EOF

'@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump
configKey: default-eg
dynamicRouteConfigs:
- routeConfig:
    '@type': type.googleapis.com/envoy.config.route.v3.RouteConfiguration
    name: default-eg-http
    virtualHosts:
    - domains:
      - www.example.com
      name: default-eg-http
      routes:
      - match:
          prefix: /
        route:
          cluster: default-backend-rule-0-match-0-www.example.com
resourceType: route

Add Missing Resources

You can pass the --add-missing-resources flag to use dummy non Gateway API resources instead of specifying them explicitly.

For example, this will provide the similar result as the above:

cat <<EOF | egctl x translate --add-missing-resources --from gateway-api --to gateway-api -t route -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: default
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
  namespace: default
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
EOF

You can see the output contains a EnvoyProxy resource that can be used as a starting point to modify the xDS bootstrap resource for the managed Envoy Proxy fleet.

envoyProxy:
  metadata:
    creationTimestamp: null
    name: default-envoy-proxy
    namespace: envoy-gateway-system
  spec:
    bootstrap: |
      admin:
        access_log:
        - name: envoy.access_loggers.file
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
            path: /dev/null
        address:
          socket_address:
            address: 127.0.0.1
            port_value: 19000
      dynamic_resources:
        ads_config:
          api_type: DELTA_GRPC
          transport_api_version: V3
          grpc_services:
          - envoy_grpc:
              cluster_name: xds_cluster
          set_node_on_first_message_only: true
        lds_config:
          ads: {}
          resource_api_version: V3
        cds_config:
          ads: {}
          resource_api_version: V3
      static_resources:
        clusters:
        - connect_timeout: 10s
          load_assignment:
            cluster_name: xds_cluster
            endpoints:
            - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: envoy-gateway
                      port_value: 18000
          typed_extension_protocol_options:
            "envoy.extensions.upstreams.http.v3.HttpProtocolOptions":
               "@type": "type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions"
               "explicit_http_config":
                 "http2_protocol_options": {}
          name: xds_cluster
          type: STRICT_DNS
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext
              common_tls_context:
                tls_params:
                  tls_maximum_protocol_version: TLSv1_3
                tls_certificate_sds_secret_configs:
                - name: xds_certificate
                  sds_config:
                    path_config_source:
                      path: "/sds/xds-certificate.json"
                    resource_api_version: V3
                validation_context_sds_secret_config:
                  name: xds_trusted_ca
                  sds_config:
                    path_config_source:
                      path: "/sds/xds-trusted-ca.json"
                    resource_api_version: V3
      layered_runtime:
        layers:
        - name: runtime-0
          rtds_layer:
            rtds_config:
              ads: {}
              resource_api_version: V3
            name: runtime-0      
    logging: {}
  status: {}
gatewayClass:
  metadata:
    creationTimestamp: null
    name: eg
    namespace: envoy-gateway-system
  spec:
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
    parametersRef:
      group: gateway.envoyproxy.io
      kind: EnvoyProxy
      name: default-envoy-proxy
      namespace: envoy-gateway-system
  status:
    conditions:
    - lastTransitionTime: "2023-04-19T20:30:46Z"
      message: Valid GatewayClass
      reason: Accepted
      status: "True"
      type: Accepted
gateways:
- metadata:
    creationTimestamp: null
    name: eg
    namespace: default
  spec:
    gatewayClassName: eg
    listeners:
    - name: http
      port: 80
      protocol: HTTP
  status:
    listeners:
    - attachedRoutes: 1
      conditions:
      - lastTransitionTime: "2023-04-19T20:30:46Z"
        message: Sending translated listener configuration to the data plane
        reason: Programmed
        status: "True"
        type: Programmed
      - lastTransitionTime: "2023-04-19T20:30:46Z"
        message: Listener has been successfully translated
        reason: Accepted
        status: "True"
        type: Accepted
      name: http
      supportedKinds:
      - group: gateway.networking.k8s.io
        kind: HTTPRoute
      - group: gateway.networking.k8s.io
        kind: GRPCRoute
httpRoutes:
- metadata:
    creationTimestamp: null
    name: backend
    namespace: default
  spec:
    hostnames:
    - www.example.com
    parentRefs:
    - name: eg
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: backend
        port: 3000
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /
  status:
    parents:
    - conditions:
      - lastTransitionTime: "2023-04-19T20:30:46Z"
        message: Route is accepted
        reason: Accepted
        status: "True"
        type: Accepted
      - lastTransitionTime: "2023-04-19T20:30:46Z"
        message: Resolved all the Object references for the Route
        reason: ResolvedRefs
        status: "True"
        type: ResolvedRefs
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
      parentRef:
        name: eg

Sometimes you might find that egctl doesn’t provide an expected result. For example, the following example provides an empty route resource:

cat <<EOF | egctl x translate --from gateway-api --type route --to xds -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
    - name: tls
      protocol: TLS
      port: 8443
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
EOF

xds:
  envoy-gateway-system-eg:
    '@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump

Validating Gateway API Configuration

You can add an additional target gateway-api to show the processed Gateway API resources. For example, translating the above resources with the new argument shows that the HTTPRoute is rejected because there is no ready listener for it:

cat <<EOF | egctl x translate --from gateway-api --type route --to gateway-api,xds -f -
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
spec:
  gatewayClassName: eg
  listeners:
    - name: tls
      protocol: TLS
      port: 8443
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
spec:
  parentRefs:
    - name: eg
  rules:
    - backendRefs:
        - group: ""
          kind: Service
          name: backend
          port: 3000
          weight: 1
      matches:
        - path:
            type: PathPrefix
            value: /
EOF

gatewayClass:
  metadata:
    creationTimestamp: null
    name: eg
    namespace: envoy-gateway-system
  spec:
    controllerName: gateway.envoyproxy.io/gatewayclass-controller
  status:
    conditions:
    - lastTransitionTime: "2023-04-19T20:54:52Z"
      message: Valid GatewayClass
      reason: Accepted
      status: "True"
      type: Accepted
gateways:
- metadata:
    creationTimestamp: null
    name: eg
    namespace: envoy-gateway-system
  spec:
    gatewayClassName: eg
    listeners:
    - name: tls
      port: 8443
      protocol: TLS
  status:
    listeners:
    - attachedRoutes: 0
      conditions:
      - lastTransitionTime: "2023-04-19T20:54:52Z"
        message: Listener must have TLS set when protocol is TLS.
        reason: Invalid
        status: "False"
        type: Programmed
      name: tls
      supportedKinds:
      - group: gateway.networking.k8s.io
        kind: TLSRoute
httpRoutes:
- metadata:
    creationTimestamp: null
    name: backend
    namespace: envoy-gateway-system
  spec:
    parentRefs:
    - name: eg
    rules:
    - backendRefs:
      - group: ""
        kind: Service
        name: backend
        port: 3000
        weight: 1
      matches:
      - path:
          type: PathPrefix
          value: /
  status:
    parents:
    - conditions:
      - lastTransitionTime: "2023-04-19T20:54:52Z"
        message: There are no ready listeners for this parent ref
        reason: NoReadyListeners
        status: "False"
        type: Accepted
      - lastTransitionTime: "2023-04-19T20:54:52Z"
        message: Service envoy-gateway-system/backend not found
        reason: BackendNotFound
        status: "False"
        type: ResolvedRefs
      controllerName: gateway.envoyproxy.io/gatewayclass-controller
      parentRef:
        name: eg
xds:
  envoy-gateway-system-eg:
    '@type': type.googleapis.com/envoy.admin.v3.RoutesConfigDump

egctl experimental status

This subcommand allows users to show the summary of the status of specific or all resource types, in order to quickly find out the status of any resources.

By default, egctl x status display all the conditions for one resource type. You can either add --quiet to only display the latest condition, or add --verbose to display more details about current status.

Some examples of this command after installing Multi-tenancy example manifest:

  • Show the summary of GatewayClass.
~ egctl x status gatewayclass

NAME           TYPE       STATUS    REASON
eg-marketing   Accepted   True      Accepted
eg-product     Accepted   True      Accepted
  • Show the summary of all resource types under all namespaces, the resource type with empty status will be ignored.
~ egctl x status all -A

NAME                        TYPE       STATUS    REASON
gatewayclass/eg-marketing   Accepted   True      Accepted
gatewayclass/eg-product     Accepted   True      Accepted

NAMESPACE   NAME         TYPE         STATUS    REASON
marketing   gateway/eg   Programmed   True      Programmed
                         Accepted     True      Accepted
product     gateway/eg   Programmed   True      Programmed
                         Accepted     True      Accepted

NAMESPACE   NAME                PARENT       TYPE           STATUS    REASON
marketing   httproute/backend   gateway/eg   ResolvedRefs   True      ResolvedRefs
                                             Accepted       True      Accepted
product     httproute/backend   gateway/eg   ResolvedRefs   True      ResolvedRefs
                                             Accepted       True      Accepted
  • Show the summary of all the Gateways with details under all namespaces.
~ egctl x status gateway --verbose --all-namespaces

NAMESPACE   NAME      TYPE         STATUS    REASON       MESSAGE                                                                    OBSERVED GENERATION   LAST TRANSITION TIME
marketing   eg        Programmed   True      Programmed   Address assigned to the Gateway, 1/1 envoy Deployment replicas available   1                     2024-02-02 18:17:14 +0800 CST
                      Accepted     True      Accepted     The Gateway has been scheduled by Envoy Gateway                            1                     2024-02-01 17:50:39 +0800 CST
product     eg        Programmed   True      Programmed   Address assigned to the Gateway, 1/1 envoy Deployment replicas available   1                     2024-02-02 18:17:14 +0800 CST
                      Accepted     True      Accepted     The Gateway has been scheduled by Envoy Gateway                            1                     2024-02-01 17:52:42 +0800 CST
  • Show the summary of the latest Gateways condition under product namespace.
~ egctl x status gateway --quiet -n product

NAME      TYPE         STATUS    REASON
eg        Programmed   True      Programmed
  • Show the summary of latest HTTPRoutes condition under all namespaces.
~ egctl x status httproute --quiet --all-namespaces

NAMESPACE   NAME      PARENT       TYPE           STATUS    REASON
marketing   backend   gateway/eg   ResolvedRefs   True      ResolvedRefs
product     backend   gateway/eg   ResolvedRefs   True      ResolvedRefs

egctl experimental dashboard

This subcommand streamlines the process for users to access the Envoy admin dashboard. By executing the following command:

egctl x dashboard envoy-proxy -n envoy-gateway-system envoy-engw-eg-a9c23fbb-558f94486c-82wh4

You will see the following output:

egctl x dashboard envoy-proxy -n envoy-gateway-system envoy-engw-eg-a9c23fbb-558f94486c-82wh4
http://localhost:19000

the Envoy admin dashboard will automatically open in your default web browser. This eliminates the need to manually locate and expose the admin port.

egctl experimental install

This subcommand can be used to install envoy-gateway.

egctl x install

By default, this will install both the envoy-gateway workload resource and the required gateway-api and envoy-gatewayCRDs.

We can specify to install only workload resources via --skip-crds

egctl x install --skip-crds

We can specify to install only CRDs resources via --only-crds

egctl x install --only-crds

We can specify --name and --namespace to install envoy-gateway in different places to support multi-tenant mode.

Note: If CRDs are already installed, then we need to specify --skip-crds to avoid repeated installation of CRDs resources.

egctl x install --name shop-backend --namespace shop

egctl experimental uninstall

This subcommand can be used to uninstall envoy-gateway.

egctl x uninstall

By default, this will only uninstall the envoy-gateway workload resource, if we want to also uninstall CRDs, we need to specify --with-crds

egctl x uninstall --with-crds