API Gateway with Envoy: Deployment Patterns and Policy Ownership

Envoy can be the data plane of an API gateway, run as a standalone edge proxy, or participate in a service mesh. Those are different operating models. Envoy provides listeners, filters, clusters, routing, observability, and static or dynamic configuration; a gateway product or platform must still define who owns desired state, API consumers, policy, rollout, tenancy, and support.

Start by naming the product boundary. “We use Envoy” is not enough to tell an on-call engineer whether to edit a bootstrap file, a Kubernetes HTTPRoute, a vendor control plane, or a custom xDS service.

The raw Envoy behavior in this guide is reviewed against stable Envoy 1.39.1. Pin the deployed patch release; Envoy's floating latest documentation follows active development and is not used as evidence here.

Key Takeaways

• Separate Envoy Proxy, Envoy Gateway, a service mesh, and third-party Envoy-based gateway products; they share technology but not one configuration or feature contract.
• Static configuration is valid for simple deployments. xDS adds dynamic configuration but also creates a control-plane API and consistency problem the platform must own.
• At an edge listener, use_remote_address and trusted XFF hops define client-IP semantics; the default path is not an end-user identity guarantee.
• In ext_authz, the authorization service returns a decision while Envoy enforces it. failure_mode_allow changes Envoy's behavior on service errors.
• Assign retries, timeouts, transformations, and authorization to one authoritative layer, then test filter order and failure timing.

Distinguish the Components

The Envoy 1.39.1 overview describes a general L3/L4 and HTTP proxy with optional dynamic configuration. Envoy Gateway is a separate project whose control plane translates Kubernetes Gateway API resources into xDS for Envoy Proxy. Evaluate the exact project and version instead of transferring capabilities between them.

Choose a Deployment Pattern

1. Standalone Envoy at the edge

flowchart LR
    C[Client] --> E[Edge Envoy]
    E --> A[API service]

This is reasonable when a small team needs explicit proxy behavior and is prepared to own complete static configuration, delivery, certificates, filters, and rollback. Envoy 1.39.1's xDS overview confirms that fully static configuration is supported; Envoy does not require a dynamic control plane.

The trade-off is platform ownership. Static files do not create consumer onboarding, policy approval, distributed quota state, or safe multi-team tenancy. Build only the control surface the organization is willing to maintain.

2. Envoy data plane with a gateway control plane

flowchart LR
    O[Platform operator] --> P[Gateway control plane]
    P -->|xDS resources| E[Envoy data plane]
    C[Client] --> E
    E --> A[API service]

This pattern centralizes desired state and can update listeners, routes, clusters, endpoints, and secrets dynamically. The platform must define resource validation, ordering, rejection, rollout, rollback, and behavior during control-plane disconnection. The xDS APIs are a transport and resource model, not a complete change-management policy.

Record the last accepted configuration and expose rejected updates. Test a partial resource update, an invalid listener, a missing cluster, certificate rotation, and loss of the management server. Do not call a control-plane update successful only because its API accepted the request.

3. Public API gateway before Envoy mesh proxies

flowchart LR
    C[External client] --> G[API gateway]
    G --> W1[Service workload with Envoy]
    W1 --> W2[Downstream workload with Envoy]

Use this when the gateway owns the external API contract and Envoy-based mesh components own service traffic. Authenticate the gateway-to-workload connection, remove caller-supplied identity headers at the edge, and decide which verified context crosses the boundary. Avoid configuring the edge and mesh to retry or transform the same operation independently.

Make Client-IP Handling Explicit

Envoy 1.39.1's HTTP header documentation explains that X-Forwarded-For (XFF) can be forged by clients and that the immediately connected peer is the first reliable network fact. The use_remote_address Boolean and xff_num_trusted_hops, whose default is 0, must be evaluated together:

• use_remote_address: false (default) and xff_num_trusted_hops: 0: when XFF is present, Envoy selects its rightmost address; when XFF is absent, Envoy uses the immediate downstream connection address.
• use_remote_address: false and xff_num_trusted_hops: N where N > 0: Envoy selects the (N + 1)th address from the right of XFF; if the list is too short, it falls back to the immediate downstream connection address.
• use_remote_address: true and xff_num_trusted_hops: 0: Envoy uses the immediate downstream connection address rather than XFF to select the trusted client address.
• use_remote_address: true and xff_num_trusted_hops: N where N > 0: Envoy selects the Nth address from the right of XFF; if the list is too short, it falls back to the immediate downstream connection address.

The documentation generally recommends use_remote_address: true for a front proxy and may require false for an internal mesh proxy. Neither value is universally secure without topology. If the number of proxies changes, a hop-count rule can select the wrong address. Prefer a stable authenticated proxy path, restrict direct access, and test zero, expected, and extra-hop requests.

Envoy generates or mutates x-request-id according to edge and preservation settings. Use that value for correlation, not for authentication. If an external request ID is preserved, it remains caller-influenced unless an edge boundary replaces it.

Keep Authorization Verdict and Enforcement Separate

Envoy 1.39.1's ext_authz filter sends selected request context to an external authorization service. The service makes an allow or deny decision; Envoy applies the result in the request path. Filter order determines whether authentication metadata and transformations exist before the check and which later filters run after it.

This excerpt is intentionally incomplete and uses the Envoy 1.39.1 v3 API. It shows a fail-closed error path and must be integrated into a full listener and cluster configuration for that release:

http_filters:
  - name: envoy.filters.http.ext_authz
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
      grpc_service:
        envoy_grpc:
          cluster_name: authorization_service
        timeout: 0.5s
      failure_mode_allow: false
  - name: envoy.filters.http.router
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

failure_mode_allow is a behavior-changing Boolean whose protobuf default is false:

• false (default): an authorization-service communication error or HTTP 5xx causes Envoy to reject the request.
• true: Envoy allows the request to continue on those service errors; metrics still record the event.

failure_mode_allow_header_add is a second behavior-changing Boolean and also defaults to false. When it is false, Envoy adds no fail-open marker. When it is true, Envoy adds x-envoy-auth-failure-mode-allowed: true only when failure_mode_allow is also true and the authorization-service communication fails or returns HTTP 5xx. Strip caller-supplied copies at the trusted edge, and do not treat this header alone as authoritative on a path that can bypass that edge.

These settings govern infrastructure errors, not an explicit deny decision. Choose failure_mode_allow for each ext_authz filter instance and its configuration scope, not as a per-route override: the versioned ExtAuthzPerRoute API can disable the filter or provide check settings, but it does not override failure_mode_allow. Routes that require different error policies need separately scoped filter configurations. Make fail-open traffic observable, never label the authorization service's verdict as an Envoy operating mode, limit the body and headers sent to the service, protect credentials, and state whether a truncated or absent body can change the decision.

Design xDS Delivery as a Safety System

Dynamic configuration adds several states: desired, delivered, accepted, rejected, and serving. Track them separately.

• Validate cross-resource references before rollout.
• Use staged cohorts rather than updating an entire fleet at once.
• Observe ACK/NACK and the exact resource version on each proxy.
• Keep certificates and secrets out of ordinary logs and diffs.
• Define whether rollback restores one resource or a coherent snapshot.
• Bound how long a proxy may serve stale state during control-plane loss.
• Rehearse recovery when the control plane returns with a newer but invalid configuration.

Aggregated xDS can improve ordering across resource types, but the platform still owns dependency correctness and rollout. A healthy control-plane API does not prove that every proxy accepted the same usable route graph.

Assign Filters and Resilience Once

Envoy filter order is part of policy. Authentication must run before a policy that consumes authenticated claims. Header mutation before authorization can change what the policy sees; mutation after authorization can change what the upstream receives. Logging before a final transformation may record a different request from the one delivered.

For retries and timeouts, record:

• end-to-end and per-try deadlines;
• retryable status, reset, and connection conditions;
• method and operation safety;
• maximum attempts across client, gateway, mesh, and application;
• request-body buffering and streaming behavior;
• circuit-breaker and outlier-ejection interaction;
• cancellation propagation when the client leaves.

One Envoy layer may be the right owner, but multiple Envoy hops do not share an automatic global retry budget. Verify actual upstream attempts under reset, timeout, overload, and partial-response conditions.

Preserve the Application Boundary

The gateway may authenticate a consumer and enforce route-level permission. It should not claim to complete object-level authorization unless it has the authoritative domain state. Forward the minimum verified context over an authenticated connection and let the service decide whether that principal may act on the requested account, order, or document.

Likewise, a successful external authorization decision describes the input and policy evaluated at that moment. It does not prove the upstream will interpret the path, method, or normalized headers identically. Test normalization and keep critical domain checks in the service.

Verification Checklist

• The exact Envoy, Envoy Gateway, mesh, or gateway-product version and boundary are recorded.
• Only intended listeners and admin interfaces are reachable.
• Static or xDS configuration has a validated rollback artifact.
• XFF tests cover direct, expected-proxy, forged, and extra-hop paths.
• Missing, invalid, denied, auth-service-error, and allowed requests produce distinct observations.
• failure_mode_allow behavior matches the filter configuration scope's documented threat model.
• Filter order is tested using the actual transformed headers and identity metadata.
• Retry totals and deadlines remain bounded across every Envoy and non-Envoy hop.
• gRPC, streaming, WebSocket, large-body, and cancellation paths are exercised.
• Desired, delivered, accepted, and serving resource versions can be distinguished.

Summary

Envoy is a capable proxy foundation, not one universal API gateway product. Choose whether configuration is static, owned by Envoy Gateway, delivered by a custom xDS control plane, or managed by another product. Then make the client-IP path, authorization verdict, fail-open or fail-closed behavior, filter order, retries, and application boundary explicit. The architecture is complete only when operators know which system to change and can prove which configuration each proxy is serving.

FAQ

Does Envoy require xDS?

No. Envoy supports static configuration. xDS is useful for dynamically managing a fleet, but its control plane and rollout behavior become platform responsibilities.

Is Envoy Gateway the same as Envoy Proxy?

No. Envoy Proxy is the data-plane proxy. Envoy Gateway is a Kubernetes Gateway API control plane that configures Envoy Proxy.

Should failure_mode_allow be enabled?

Only when the affected routes' availability and threat models justify allowing requests during authorization-service errors. The default is false; whichever path is chosen must be observable and tested. Routes that need different behavior require separately scoped filter configurations.

Next Steps

Compare Envoy and other gateway runtime families, design external authorization boundaries, and apply a reproducible gateway benchmark method.