API Gateway Timeouts and Retries: Safe Policies and APISIX Configuration

Safe API gateway retry policy begins with an end-to-end deadline, assigns bounded time to connection, request transmission, and response waiting, and retries only operations that can be repeated without an unintended second effect. Keep the attempt count small, stop within the caller's remaining time, and combine retries with health checks, capacity protection, and observability.

Retries can hide a brief node failure. They can also duplicate a payment, amplify an outage, and make a request finish after its caller has already given up. “Retry every 5xx three times” is therefore not a reliability strategy.

Key Takeaways

• Start with the caller's end-to-end deadline, then allocate smaller downstream time budgets.
• Configure connect, send, and read timeouts explicitly; they protect different phases.
• Do not automatically retry a non-idempotent operation unless the application can prove that replay is safe.
• Limit both attempts and total retry time; account for retries performed by clients, meshes, libraries, and upstream services.
• Use health checks to avoid repeatedly selecting known unhealthy nodes, not as permission for unlimited retry.
• Test ambiguous failures, slow responses, refused connections, and mutations before rollout.

Understand the Three Upstream Timeouts

API gateways usually expose phase-specific timeouts between the gateway and an upstream:

These are not automatically the same as one total request deadline. A retry can consume another connect and read interval, while queueing, TLS, plugins, and downstream transmission consume additional time. Derive the phase values from measured latency and the caller's budget rather than setting every field to a large common value.

For example, if a caller abandons a request after three seconds, the gateway cannot safely spend three seconds on an initial read and another three on a retry. Reserve time for gateway processing, network variance, and the response path. If the remaining budget cannot fit another useful attempt, fail instead of retrying.

Decide Retry Eligibility from Semantics

RFC 9110 section 9.2.2 permits automatic retry of idempotent requests after a communication failure. It also states that a proxy must not automatically retry non-idempotent requests. This matters most when the gateway cannot tell whether the upstream performed the operation before the connection failed.

Use a policy matrix rather than a blanket method list:

HTTP method alone is not proof. A poorly designed GET can have side effects, and a POST can be made replay-safe with a correctly implemented idempotency key. The service owner must define the invariant and the gateway policy must preserve it.

Bound Amplification Across Layers

Retries multiply. If a client makes three attempts, a gateway makes three attempts per client attempt, and an upstream library does the same, one user action can create 27 downstream calls.

Inventory every retrying layer and assign one owner for each failure boundary. A practical gateway policy usually includes:

• zero or one retry, not an open-ended count;
• a total retry-time bound shorter than the remaining request budget;
• selection of a different healthy node where the implementation supports it;
• no retry after the downstream response has begun;
• concurrency and rate controls that cap amplification during an outage;
• jittered backoff at clients that initiate a later end-to-end attempt.

An immediate in-gateway failover attempt and a later client retry serve different purposes. The former may avoid one failed node with little delay; the latter should back off so the system can recover. Do not add sleep inside a request merely to make a gateway retry resemble client backoff.

Configure APISIX for a Bounded Read Retry

Apache APISIX 3.18 exposes retries, retry_timeout, and timeout.connect, timeout.send, and timeout.read on an Upstream. The current Admin API documentation says that retries: 0 disables retries; otherwise APISIX uses the underlying NGINX mechanism. If retries is omitted, its default is based on the available backend nodes, so set it explicitly when request semantics matter.

The following illustrative Upstream is intended only for replay-safe read operations. The values must be replaced with measurements from the actual service.

curl "http://127.0.0.1:9180/apisix/admin/upstreams/orders-read" \
  -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "type": "roundrobin",
    "nodes": {
      "orders-1.internal:8080": 1,
      "orders-2.internal:8080": 1
    },
    "retries": 1,
    "retry_timeout": 2,
    "timeout": {
      "connect": 0.5,
      "send": 1,
      "read": 2
    },
    "checks": {
      "active": {
        "type": "http",
        "http_path": "/health",
        "timeout": 1,
        "healthy": {
          "interval": 5,
          "successes": 2
        },
        "unhealthy": {
          "interval": 2,
          "http_failures": 3,
          "tcp_failures": 2,
          "timeouts": 2
        }
      }
    }
  }'

Attach only replay-safe routes to this Upstream:

curl "http://127.0.0.1:9180/apisix/admin/routes/orders-read" \
  -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "uri": "/orders/*",
    "methods": ["GET", "HEAD"],
    "upstream_id": "orders-read"
  }'

The active health check helps APISIX stop selecting nodes that repeatedly fail its configured probes while healthy alternatives remain. The documented behavior is fail-open at the pool boundary: if no healthy node can be selected, APISIX continues to access the Upstream. Health checks therefore do not make every application request replay-safe, and a health endpoint can be healthy while a dependency required by one operation is failing.

APISIX also supports passive health checks based on proxied traffic. Its health-check documentation notes that passive checks alone cannot mark an unhealthy node healthy again because that node no longer receives requests; active checks are normally needed for recovery detection.

Disable Automatic Retry for Mutations

Do not place unsafe mutations behind an Upstream whose inherited retry count is unknown. Use a separate Upstream or a route-level design that makes the policy explicit. This example disables retry for order creation:

curl "http://127.0.0.1:9180/apisix/admin/upstreams/orders-write" \
  -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "type": "roundrobin",
    "nodes": {
      "orders-1.internal:8080": 1,
      "orders-2.internal:8080": 1
    },
    "retries": 0,
    "timeout": {
      "connect": 0.5,
      "send": 1,
      "read": 2
    }
  }'

curl "http://127.0.0.1:9180/apisix/admin/routes/orders-write" \
  -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "uri": "/orders",
    "methods": ["POST"],
    "upstream_id": "orders-write"
  }'

If the application implements idempotency keys, test concurrent duplicate requests, key expiration, payload mismatches under one key, and atomic persistence of the outcome. Merely forwarding an Idempotency-Key header does not guarantee replay safety.

Choose Retry Conditions Conservatively

Not every failure benefits from another attempt:

• Connection refusal or reset before request application: another healthy node may succeed.
• Read timeout after the request was sent: outcome may be ambiguous; retry only if semantics tolerate replay.
• HTTP 500: may be deterministic for this input and repeat on every node.
• HTTP 429: indicates admission pressure; an immediate retry usually adds load.
• HTTP 503: can be transient, but can also mean the whole pool is saturated.
• Invalid request or authorization failure: retrying without changing the request cannot help.

Confirm the exact APISIX and NGINX retry conditions for the protocol and release you operate. The retries count is only one part of behavior; protocol, failure phase, response state, and underlying proxy configuration also matter.

Validate with Fault Injection

Use a staging environment with at least two distinguishable upstream nodes. Record which node handled each attempt, then test:

1. Refuse the connection on one node and verify a replay-safe request uses at most the allowed extra attempt.
2. Delay connection establishment, request reading, and response generation separately to identify which timeout expires.
3. Return selected 5xx responses and verify only intended conditions are retried.
4. Close the connection after applying a mutation but before returning the response; confirm the gateway does not replay it.
5. Make every node unhealthy and verify APISIX's documented fail-open behavior, the actual node selection and attempt count, and a bounded total latency.
6. Combine client retry with gateway retry under load and confirm upstream traffic stays within capacity.

Do not assert one fixed client status for every experiment. Depending on the failure phase and configuration, the gateway can return different 5xx statuses. Assert the invariants that matter: maximum attempts, total latency, node selection, and absence of duplicate effects.

Monitor the Retry Budget

Use APISIX's prometheus plugin with upstream and application telemetry to track:

• request and upstream latency distributions;
• gateway and upstream status codes;
• upstream health state;
• request volume at the client, gateway, and service to reveal amplification;
• timeout and connection failure counts;
• duplicate-effect or idempotency-conflict signals from the application.

Alert on a change in ratios, not just raw errors. A small client-error increase paired with a much larger upstream-request increase is a classic retry-amplification signal.

Timeout and Retry Checklist

• What is the caller's end-to-end deadline?
• How much time is available for connect, send, read, gateway work, and the response path?
• Is the operation provably replay-safe, including ambiguous outcomes?
• Which layers retry, and what is the maximum combined amplification?
• Are retry count and retry time explicitly bounded?
• Can the retry select another healthy node without crossing a consistency boundary?
• Do rate and concurrency limits prevent an outage from becoming a retry storm?
• Have slow, refused, partial, 5xx, and mutation failures been tested?
• Do metrics show attempts and effects, not only final client responses?

FAQ

Should an API gateway retry all GET requests?

No. GET is defined as idempotent, but a retry still consumes time and capacity, and some implementations misuse GET for side effects. Retry only when the operation is actually replay-safe, another attempt can fit within the deadline, and the failure is likely transient.

How many retries should an API gateway use?

There is no universal number. Zero or one is a safer starting point than several attempts. Choose the bound from the end-to-end latency budget, number of healthy alternatives, measured transient-failure rate, and retry behavior in other layers.

Are health checks a replacement for retries?

No. Health checks reduce selection of known unhealthy nodes. Retries handle some failures that occur during a request. Both need bounded behavior, and neither makes a non-idempotent operation safe to repeat.

Next Steps

Review how API gateways, reverse proxies, and load balancers compose before assigning retry ownership across layers. For centrally governed Apache APISIX deployments, explore API7 Enterprise.