Proxy Tracing
4 minute read
Envoy Gateway provides observability for the ControlPlane and the underlying EnvoyProxy instances. This task show you how to config proxy tracing.
Prerequisites
Follow the steps from the Quickstart to install Envoy Gateway and the example manifest. Before proceeding, you should be able to query the example backend using HTTP.
Envoy Gateway provides an add-ons Helm Chart, which includes all the needing components for observability. By default, the OpenTelemetry Collector is disabled.
Install the add-ons Helm Chart:
helm install eg-addons oci://docker.io/envoyproxy/gateway-addons-helm --version v1.2.4 --set opentelemetry-collector.enabled=true -n monitoring --create-namespace
Expose Tempo endpoints:
TEMPO_IP=$(kubectl get svc tempo -n monitoring -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
Traces
By default, Envoy Gateway doesn’t send traces to any sink.
You can enable traces by setting the telemetry.tracing in the EnvoyProxy CRD.
Currently, Envoy Gateway support OpenTelemetry, Zipkin and Datadog tracer.
Tracing Provider
The following configurations show how to apply proxy with different providers:
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: otel
namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: otel
namespace: envoy-gateway-system
spec:
telemetry:
tracing:
# sample 100% of requests
samplingRate: 100
provider:
backendRefs:
- name: otel-collector
namespace: monitoring
port: 4317
type: OpenTelemetry
customTags:
# This is an example of using a literal as a tag value
provider:
type: Literal
literal:
value: "otel"
"k8s.pod.name":
type: Environment
environment:
name: ENVOY_POD_NAME
defaultValue: "-"
"k8s.namespace.name":
type: Environment
environment:
name: ENVOY_GATEWAY_NAMESPACE
defaultValue: "envoy-gateway-system"
# This is an example of using a header value as a tag value
header1:
type: RequestHeader
requestHeader:
name: X-Header-1
defaultValue: "-"
EOF
Verify OpenTelemetry traces from tempo:
curl -s "http://$TEMPO_IP:3100/api/search?tags=component%3Dproxy+provider%3Dotel" | jq .traces
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: zipkin
namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: zipkin
namespace: envoy-gateway-system
spec:
telemetry:
tracing:
# sample 100% of requests
samplingRate: 100
provider:
backendRefs:
- name: otel-collector
namespace: monitoring
port: 9411
type: Zipkin
zipkin:
enable128BitTraceId: true
customTags:
# This is an example of using a literal as a tag value
provider:
type: Literal
literal:
value: "zipkin"
"k8s.pod.name":
type: Environment
environment:
name: ENVOY_POD_NAME
defaultValue: "-"
"k8s.namespace.name":
type: Environment
environment:
name: ENVOY_GATEWAY_NAMESPACE
defaultValue: "envoy-gateway-system"
# This is an example of using a header value as a tag value
header1:
type: RequestHeader
requestHeader:
name: X-Header-1
defaultValue: "-"
EOF
Verify zipkin traces from tempo:
curl -s "http://$TEMPO_IP:3100/api/search?tags=component%3Dproxy+provider%3Dzipkin" | jq .traces
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: datadog
namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: datadog
namespace: envoy-gateway-system
spec:
telemetry:
tracing:
# sample 100% of requests
samplingRate: 100
provider:
backendRefs:
- name: datadog-agent
namespace: monitoring
port: 8126
type: Datadog
customTags:
# This is an example of using a literal as a tag value
provider:
type: Literal
literal:
value: "datadog"
"k8s.pod.name":
type: Environment
environment:
name: ENVOY_POD_NAME
defaultValue: "-"
"k8s.namespace.name":
type: Environment
environment:
name: ENVOY_GATEWAY_NAMESPACE
defaultValue: "envoy-gateway-system"
# This is an example of using a header value as a tag value
header1:
type: RequestHeader
requestHeader:
name: X-Header-1
defaultValue: "-"
EOF
Verify Datadog traces in Datadog APM
Query trace by trace id:
curl -s "http://$TEMPO_IP:3100/api/traces/<trace_id>" | jq
Sampling Rate
Envoy Gateway use 100% sample rate, which means all requests will be traced.
This may cause performance issues when traffic is very high, you can adjust
the sample rate by setting the telemetry.tracing.samplingRate in the EnvoyProxy CRD.
The following configurations show how to apply proxy with 1% sample rates:
kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
parametersRef:
group: gateway.envoyproxy.io
kind: EnvoyProxy
name: otel
namespace: envoy-gateway-system
---
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: EnvoyProxy
metadata:
name: otel
namespace: envoy-gateway-system
spec:
telemetry:
tracing:
# sample 1% of requests
samplingRate: 1
provider:
backendRefs:
- name: otel-collector
namespace: monitoring
port: 4317
type: OpenTelemetry
customTags:
# This is an example of using a literal as a tag value
provider:
type: Literal
literal:
value: "otel"
"k8s.pod.name":
type: Environment
environment:
name: ENVOY_POD_NAME
defaultValue: "-"
"k8s.namespace.name":
type: Environment
environment:
name: ENVOY_GATEWAY_NAMESPACE
defaultValue: "envoy-gateway-system"
# This is an example of using a header value as a tag value
header1:
type: RequestHeader
requestHeader:
name: X-Header-1
defaultValue: "-"
EOF
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.