API Gateway Reliability Engineering: SLOs, Failure Isolation, and Incident Response

API gateway reliability is not the percentage of time a gateway process stays up. It is the gateway's contribution to a user-visible service objective: admitting valid requests, applying required policy, selecting an eligible upstream, and returning a correct response within the promised time.

The gateway can isolate some failures and expose useful evidence. It cannot guarantee application correctness, make unsafe retries safe, or recover a dependency whose capacity and data are gone. Reliability engineering begins by defining the user journey and failure ownership, then selecting controls that preserve that contract under stress.

Key Takeaways

• Define SLIs from user outcomes before selecting the easiest gateway metrics.
• Separate gateway availability, policy dependency, upstream service, and end-to-end correctness failures.
• Treat health checks, retries, circuit breakers, load shedding, and traffic shifting as different controls.
• Document fail-open and fail-closed behavior for every external policy and state dependency.
• Use error budgets to govern rollout and recovery, not as permission to consume all remaining failure at once.
• During an incident, preserve configuration revision, response source, attempt count, and degraded-mode evidence.

Start with a User-Centered Reliability Contract

Google's SRE guidance on SLOs recommends working backward from what users care about and stating how and when the objective is measured. For a gateway-fronted order lookup, a contract might define:

• Good event: an authenticated, authorized request within the supported size limit returns an application-valid response in less than 500 ms.
• Valid population: production GET /orders/{id} requests, excluding approved synthetic tests and documented client cancellations.
• Objective: 99.9% good events over a rolling 28-day window.
• Measurement points: gateway client-facing outcome plus an application correctness signal.

The gateway can measure status and latency, but HTTP 200 alone cannot prove that the returned order is correct. Conversely, a gateway 429 may be deliberate load shedding rather than an upstream failure, yet it still affects the user unless the product contract excludes or separately budgets that admission class.

Avoid one SLO for unrelated workloads. Interactive reads, asynchronous submissions, streaming responses, and administrative APIs have different latency and correctness expectations.

Map Failure Ownership Before Adding Controls

flowchart LR
    C[Client] --> E[Edge or load balancer]
    E --> G[Gateway data plane]
    G --> P[Policy and quota dependencies]
    G --> U[Upstream pool]
    U --> D[Application dependencies]
    CP[Gateway control plane] -. configuration stream .-> G
    G -. metrics logs traces .-> O[Observability systems]

For each arrow, record what happens when it is slow, unavailable, stale, or contradictory. The data plane may continue with its last valid configuration during a control-plane outage, but that trades change freshness for request continuity. A quota-store outage may fail closed to preserve an entitlement boundary or fail open to preserve availability. Neither choice is universally correct.

A useful failure table is explicit:

“Keep serving” is incomplete unless the team knows which policy, configuration, or evidence may be stale or missing.

Match the Control to the Failure

Stacking all controls can amplify rather than reduce failure. A client retry, gateway retry, service-mesh retry, and application retry multiply work. A circuit breaker that returns quickly may make an upstream look less loaded while users still fail. A stale cache can preserve reads but violate a freshness or revocation requirement.

Use APISIX Health Checks with the Pool Boundary in Mind

Apache APISIX 3.19.0 supports active and passive upstream health checks. Active checks use HTTP, HTTPS, or TCP probes. Passive checks classify proxied outcomes.

Passive checks alone cannot mark an unhealthy node healthy again because the node no longer receives requests; combine them with active checks when automatic recovery detection is required. APISIX also documents a pool-level fail-open boundary: if no healthy node can be selected, it continues to access the upstream. That behavior avoids treating health classification as an absolute traffic stop, but it can send requests to a failing pool.

Decide whether that behavior matches the service contract. If contacting an unhealthy node is worse than a controlled gateway response, add an explicit admission or breaker policy and test the exhausted-pool path. Do not claim that enabling health checks guarantees that traffic reaches only healthy nodes.

Probe design matters. A shallow TCP probe can show that a process accepts connections while its database is unavailable. A deep probe can create load or take the service out because of one non-critical dependency. Use separate liveness, readiness, and diagnostic signals, and keep the routing probe bounded.

Use a Circuit Breaker as a Route Policy, Not a Diagnosis

APISIX 3.19.0's api-breaker plugin considers configured upstream response statuses and failure counts, then returns a configured response while the circuit is open. It probes recovery on an exponential schedule up to max_breaker_sec and closes after the configured healthy successes.

This excerpt deliberately chooses values rather than relying on defaults:

{
  "api-breaker": {
    "break_response_code": 503,
    "break_response_body": "Service temporarily unavailable",
    "max_breaker_sec": 30,
    "unhealthy": {
      "http_statuses": [500, 502, 503, 504],
      "failures": 5
    },
    "healthy": {
      "http_statuses": [200, 204],
      "successes": 2
    }
  }
}

This is a plugin fragment, not a complete Route. It uses HTTP status as evidence; it does not inspect application correctness, distinguish every failure cause, or replace per-node health checks. Choose statuses from application semantics. A 500 caused by one invalid input should not necessarily remove a whole service, while a successful probe may not prove that the original operation recovered.

The synthetic 503 is generated by the gateway when the circuit is open. Preserve response-source telemetry so on-call engineers do not attribute it to the upstream.

Shed Load Before Collapse

When demand exceeds safe capacity, an early bounded rejection is often better than letting queues grow until every request times out. Define:

• the protected resource: connections, gateway workers, upstream concurrency, database capacity, or model queue;
• the admission key and its trusted source;
• priority classes and starvation safeguards;
• rejection status and retry guidance;
• maximum queue and deadline;
• the signal that ends degraded mode.

Rate limiting protects a rate window; concurrency limiting protects simultaneous work. Neither knows business priority unless policy supplies it. A caller-supplied priority header is not authoritative until authenticated identity and server-side entitlement map it to a class.

Govern Changes with Error Budgets

An error budget is the failure allowance implied by an SLO, not spare outage time to consume casually. Use burn rate and remaining budget to decide whether to:

• continue or pause a rollout;
• reduce canary weight;
• disable a costly optional plugin;
• stop feature work and repair reliability;
• invoke a documented degraded mode.

APISIX 3.19.0's traffic-split plugin can direct traffic to weighted upstreams. Its documentation warns that ratios may be less accurate because weighted round robin is used, especially after state resets. Treat a weight as routing intent, not a guarantee that every short interval receives that exact percentage.

Compare the canary and baseline on the same valid request population. Gate promotion on user-visible error, latency, saturation, and semantic checks; a healthy gateway process is not enough.

Build Incident Evidence into the Gateway Path

During an incident, answer these questions quickly:

1. Which user journey and SLO are burning?
2. Which gateway clusters, Routes, Services, and revisions are affected?
3. Did APISIX, NGINX, or the upstream generate the response?
4. How much latency occurred in the gateway versus upstream waiting?
5. Which health, breaker, retry, cache, or load-shed state was active?
6. How many upstream attempts came from one client request?
7. Did a configuration, deployment, certificate, policy, or dependency change precede the event?
8. Is the mitigation preserving security, consistency, and evidence obligations?

Keep identifiers useful for correlation without treating them as identity. A request ID can join metrics, logs, and traces; if the caller can supply or preserve it, it is still correlation context rather than proof of who acted.

Test the Reliability Contract

Run controlled exercises for:

• one gateway instance failing while others remain healthy;
• control-plane unavailability and stale configuration;
• slow, refused, reset, and application-error upstreams;
• every upstream node classified unhealthy;
• policy and quota dependencies failing open or closed as designed;
• a mutation timing out after it may have been applied;
• metrics, logging, and tracing loss during a user-visible incident;
• canary regression and rollback under live load;
• recovery without a traffic surge or retry storm.

Assert end-to-end invariants: maximum attempts, maximum latency, allowed degraded behavior, configuration age, and absence of duplicate effects. Do not assert only one status code if the failure phase can legitimately produce several.

Reliability Engineering Checklist

• Does each workload have a user-centered SLI, objective, window, and valid population?
• Are application correctness and gateway status measured separately?
• Is ownership defined for every dependency and degraded mode?
• Are health checks, retries, breakers, and load shedding used for distinct purposes?
• Is APISIX's no-healthy-node fail-open behavior tested and acceptable?
• Are policy dependencies explicitly fail-open or fail-closed with a time bound?
• Are retries limited to replay-safe work within the remaining deadline?
• Do rollout gates use observed canary outcomes rather than configured weight alone?
• Can telemetry distinguish gateway-, NGINX-, and upstream-generated responses?
• Does the incident runbook preserve required security and audit controls?

FAQ

Does a highly available gateway guarantee API reliability?

No. It removes one infrastructure failure mode. End-to-end reliability also depends on policy services, upstreams, data stores, networks, clients, and semantic correctness.

Should APISIX fail closed when a policy dependency fails?

It depends on the policy. Authorization and entitlement checks often require fail-closed behavior; optional enrichment may safely fail open. Record the decision, scope, duration, and alert for each dependency.

Are health checks and circuit breakers interchangeable?

No. Health checks classify node reachability or responses. A breaker stops calls according to a failure policy at its configured scope. They observe different evidence and have different recovery behavior.

Next Steps

Instrument the contract with Prometheus and Grafana gateway monitoring, bound attempts through safe timeout and retry policies, and connect capacity to elastic gateway scaling.