Overload Manager

Envoy Gateway configures the Envoy overload manager in the bootstrap of every managed Envoy proxy. The overload manager protects the proxy from running out of memory or file descriptors by shedding load before the process fails.

This page describes the defaults Envoy Gateway applies today. They are implementation defaults and may change in future releases. There is no dedicated EnvoyProxy field to configure them yet, see #3604.

These defaults are part of the default bootstrap. If you customize the bootstrap with spec.bootstrap in the EnvoyProxy resource, a Replace or JSONPatch bootstrap can change or remove the overload manager configuration. In that case, your bootstrap defines which limits apply.

Downstream connection limit

With the default bootstrap, Envoy Gateway configures the envoy.resource_monitors.global_downstream_max_connections resource monitor with a limit of 50000 active downstream connections, shared by all listeners of an Envoy proxy. Once the limit is reached, Envoy rejects new connections until existing ones close.

Heap memory limits

When the Envoy container has a memory limit, Envoy Gateway also configures the envoy.resource_monitors.fixed_heap resource monitor. Its maximum heap size is set to 80% of the container memory limit, which leaves headroom for memory that is not allocated on the heap.

The monitor reports heap pressure as the heap memory used by Envoy divided by that maximum heap size. Two overload actions are triggered by it:

For example, with a memory limit of 1Gi, the maximum heap size is about 819Mi, and Envoy stops accepting requests when its heap usage reaches about 803Mi.

The heap usage measured by Envoy is lower than the memory usage of the container reported by Kubernetes, which also counts the Envoy binary, thread stacks and kernel memory charged to the container. Container memory metrics can therefore be above 78.4% of the limit before Envoy rejects requests.

If the Envoy container has no memory limit, which is the default, no heap monitor and no heap overload actions are configured. Only the downstream connection limit applies.

You can set the memory limit as described in Customize EnvoyProxy Deployment Resources. The same applies to an Envoy proxy deployed as a DaemonSet.

Readiness and metrics

The listener that serves the readiness probe bypasses the overload manager, so an overloaded Envoy proxy keeps reporting ready while it rejects traffic. When Prometheus metrics are enabled, the listener that serves them bypasses the overload manager as well.

Autoscaling

If you autoscale the Envoy proxy on memory usage, as described in Customize EnvoyProxy Horizontal Pod Autoscaler, scale out before Envoy stops accepting requests. Use a ContainerResource metric for the envoy container. A Resource metric sums the memory of all containers in the pod, including the shutdown-manager container, so its threshold does not match the Envoy memory limit.

The HPA computes memory utilization against the memory request, and measures container memory, which is higher than the heap usage Envoy measures. The HPA uses the average utilization of all pods, so one pod can reach the limit before the average does. By default, the HPA scales out only when utilization is more than 10% above the target. Keep 1.1 × target utilization × memory request well below 78.4% of the memory limit, and leave room for the time new pods take to become ready.

For example, with a memory request and limit of 1Gi on the envoy container, a target utilization of 60% scales out when the average memory of the envoy containers is above about 676Mi, before Envoy stops accepting requests at about 803Mi of heap usage.

spec:
  provider:
    type: Kubernetes
    kubernetes:
      envoyDeployment:
        container:
          resources:
            requests:
              memory: 1Gi
            limits:
              memory: 1Gi
      envoyHpa:
        minReplicas: 2
        maxReplicas: 10
        metrics:
          - type: ContainerResource
            containerResource:
              name: memory
              container: envoy
              target:
                type: Utilization
                averageUtilization: 60

Monitoring

The overload manager emits the following statistics. The heap statistics only exist when the Envoy container has a memory limit.

Alert on stop_accepting_requests.active to detect an Envoy proxy that is shedding load, and warn on a heap pressure above 90 to act before it does. If you restrict the exported statistics with telemetry.metrics.matches in the EnvoyProxy resource, include these statistics in the match list.