---
title: "API Gateway Reliability Engineering: SLOs, Failure Isolation, and Incident Response - API7.ai"
description: "Turn gateway telemetry into user-centered SLOs, isolate upstream and policy failures, govern degraded modes, and run evidence-based incidents."
url: "https://api7.ai/learning-center/api-gateway-guide/api-gateway-reliability-engineering"
---

[Skip to article](#column-content)

Column contentsAPI Gateway Guide

[API Gateway Guide](https://api7.ai/learning-center/api-gateway-guide)

Search this column

01 Gateway Fundamentals

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=fundamentals)[1. What Is an API Gateway? API Gateway Software and Use Cases](https://api7.ai/learning-center/api-gateway-guide/what-is-an-api-gateway)[2. Why Do Modern Architectures Need an API Gateway?](https://api7.ai/learning-center/api-gateway-guide/why-do-modern-architectures-need-api-gateway)[3. Reverse Proxy vs Load Balancer vs API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-vs-reverse-proxy-vs-load-balancer)[4. API Gateway and Microservices: Decoupling and Traffic Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-and-microservices)[5. Understanding Core API Gateway Features](https://api7.ai/learning-center/api-gateway-guide/core-api-gateway-features)[6. Open Source vs. Commercial API Gateways: How to Choose the Right One?](https://api7.ai/learning-center/api-gateway-guide/open-source-vs-commercial-api-gateway)[7. API Gateway vs Service Mesh: How to Choose?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-vs-service-mesh)[8. Envoy vs NGINX, Java, and Go for API Gateways](https://api7.ai/learning-center/api-gateway-guide/nginx-vs-envoy-vs-java-vs-go)[9. Core Concepts of API Gateway: Service, Route, Upstream, Plugin, and Consumer](https://api7.ai/learning-center/api-gateway-guide/core-concepts-of-api-gateway)[10. API Gateway Comparison: Apache APISIX vs. Kong vs. Traefik vs. KrakenD vs. Tyk](https://api7.ai/learning-center/api-gateway-guide/api-gateway-comparison-apisix-kong-traefik-krakend-tyk)

02 Gateway Architecture and Design

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=architecture)[11. Deep Dive into API Gateway Architecture: Core Components and Design Principles](https://api7.ai/learning-center/api-gateway-guide/api-gateway-architecture)[12. Choosing the Right API Gateway for Your Architecture](https://api7.ai/learning-center/api-gateway-guide/api-gateway-monolithic-vs-microservices)[13. How to Architect an API Gateway for High Availability (HA)?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-high-availability)[14. API Gateway Plugin Architecture: How to Extend and Customize Your API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-plugin-system)[15. How Event-Driven Architecture (EDA) Works with API Gateway?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-event-driven-architecture)[16. API Gateway Request Lifecycle Explained](https://api7.ai/learning-center/api-gateway-guide/api-gateway-request-lifecycle-explained)[17. Cloud-Native API Gateway on Kubernetes](https://api7.ai/learning-center/api-gateway-guide/cloud-native-api-gateway-kubernetes)[18. API Gateway Control Plane vs. Data Plane](https://api7.ai/learning-center/api-gateway-guide/api-gateway-control-plane-vs-data-plane)[19. How API Gateways Work with GitOps: Automating API Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-gitops-integration)[20. How to Perform A/B Testing and Canary Releases with an API Gateway](https://api7.ai/learning-center/api-gateway-guide/ab-testing-canary-release-api-gateway)

03 Gateway Setup and Traffic Handling

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=traffic-protocols)[21. How to Deploy an API Gateway on Kubernetes: Step-by-Step Guide for Engineers](https://api7.ai/learning-center/api-gateway-guide/deploy-api-gateway-on-kubernetes)[22. Hands-on Setup with Apache APISIX API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-getting-started-guide)[23. API Gateway Auth: OAuth 2.0, JWT & OIDC with APISIX](https://api7.ai/learning-center/api-gateway-guide/api-gateway-authentication-apache-apisix-oauth-jwt-oidc)[24. Mastering Traffic Control: Efficiently Managing API Requests with Gateways](https://api7.ai/learning-center/api-gateway-guide/api-gateway-traffic-control-strategies)[25. Building Reliable API Gateways with Logging and Monitoring](https://api7.ai/learning-center/api-gateway-guide/api-gateway-logging-monitoring-best-practices)[26. Dynamic Routing in API Gateways: Best Practices for Flexible Traffic Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-dynamic-routing-management)[27. How API Gateways Support WebSocket](https://api7.ai/learning-center/api-gateway-guide/how-api-gateways-support-websocket)[28. API Gateway for GraphQL: Security, Cost, and Caching](https://api7.ai/learning-center/api-gateway-guide/api-gateway-handle-graphql)[29. How API Gateways Handle HTTP/3 QUIC Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-http3-quic)[30. How API Gateways Handle MQTT Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-mqtt)

04 Gateway Integrations, AI, and MCP

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=ai-mcp)[31. How API Gateways Handle gRPC Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-grpc-support)[32. How API Gateways Implement Custom Plugins](https://api7.ai/learning-center/api-gateway-guide/api-gateway-custom-plugins)[33. How API Gateways Integrate with Kafka](https://api7.ai/learning-center/api-gateway-guide/api-gateway-kafka-integration)[34. How to Implement API Aggregation in API Gateways](https://api7.ai/learning-center/api-gateway-guide/api-gateway-api-aggregation)[35. How API Gateways Enable Serverless Architecture](https://api7.ai/learning-center/api-gateway-guide/api-gateway-serverless-architecture)[36. How API Gateways Proxy LLM Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-proxy-llm-requests)[37. How API Gateways Enhance MCP Servers](https://api7.ai/learning-center/api-gateway-guide/api-gateway-enhance-mcp-server)[38. MCP Gateway vs MCP Proxy vs AI Gateway vs API Gateway](https://api7.ai/learning-center/api-gateway-guide/ai-gateway-vs-mcp-gateway-vs-api-gateway)[39. What Is an AI Gateway? Architecture and Benefits](https://api7.ai/learning-center/api-gateway-guide/what-is-an-ai-gateway)[40. What Is an MCP Gateway? Architecture, Use Cases & How It Works (2026 Guide)](https://api7.ai/learning-center/api-gateway-guide/what-is-mcp-gateway)

05 Gateway Performance

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=performance)[41. How API Gateway Reduces Latency: Performance Optimization Strategies](https://api7.ai/learning-center/api-gateway-guide/how-api-gateway-reduces-latency)[42. API Gateway Caching Strategy: Comprehensive Analysis and Implementation Guide](https://api7.ai/learning-center/api-gateway-guide/api-gateway-caching-strategy-analysis)[43. How to Optimize API Gateway Load Balancing: Advanced Strategies and Best Practices](https://api7.ai/learning-center/api-gateway-guide/how-to-optimize-api-gateway-load-balancing)[44. How API Gateway Efficiently Handles Large-Scale Traffic: Architecture and Strategies](https://api7.ai/learning-center/api-gateway-guide/how-api-gateway-handles-large-scale-traffic)[45. How to Improve API Gateway QoS: Traffic Classes, Capacity Protection, and SLOs](https://api7.ai/learning-center/api-gateway-guide/api-gateway-quality-of-service)[46. API Gateway Log Storage: Pipelines, Retention, Security, and Cost](https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-storage)[47. API Gateway Timeouts and Retries: Safe Policies and APISIX Configuration](https://api7.ai/learning-center/api-gateway-guide/api-gateway-timeouts-retries)[48. API Gateway Concurrency Control: Capacity Budgets, Backpressure, and Load Shedding](https://api7.ai/learning-center/api-gateway-guide/api-gateway-concurrency-control)[49. Asynchronous API Gateway Patterns: Queues, Callbacks, and Performance Boundaries](https://api7.ai/learning-center/api-gateway-guide/asynchronous-api-gateway-patterns)[50. API Gateway TLS Performance: Handshakes, Session Reuse, and Measurement](https://api7.ai/learning-center/api-gateway-guide/api-gateway-tls-performance)

06 Gateway Security

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=security)[51. DDoS Defense at the API Gateway: Layered Controls and Failure Planning](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ddos-defense)[52. Distributed API Gateway Rate Limiting: Local vs Redis Counters and Accuracy Trade-offs](https://api7.ai/learning-center/api-gateway-guide/distributed-api-gateway-rate-limiting)[53. API Gateway IP Allowlist and Denylist Management: Trust, Proxies, and Operations](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ip-allowlist-denylist-management)[54. API Gateway Security Scanning: What to Test and How to Operationalize Findings](https://api7.ai/learning-center/api-gateway-guide/api-gateway-security-scanning)[55. SQL Injection and XSS at the API Gateway: Detection Limits and Defense in Depth](https://api7.ai/learning-center/api-gateway-guide/sql-injection-xss-api-gateway)[56. API Access Log Auditing: Evidence, Integrity, Retention, and Review](https://api7.ai/learning-center/api-gateway-guide/api-access-log-auditing)[57. Preventing API Scraping and Abuse: Identity, Behavioral Signals, and Rate Controls](https://api7.ai/learning-center/api-gateway-guide/prevent-api-scraping-abuse)[58. API Gateway mTLS: Client Identity, Certificate Rotation, and APISIX Configuration](https://api7.ai/learning-center/api-gateway-guide/api-gateway-mtls)[59. Fine-Grained API Gateway Authorization: RBAC, ABAC, and External Policy Decisions](https://api7.ai/learning-center/api-gateway-guide/fine-grained-api-gateway-authorization)[60. API Gateway and WAF Reference Architecture: Ownership, Header Trust, and Failure Modes](https://api7.ai/learning-center/api-gateway-guide/api-gateway-waf-reference-architecture)

07 Gateway Ecosystem and Comparisons

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=ecosystem)[61. Apache APISIX vs Kong: Migration, Operations, and Team Fit](https://api7.ai/learning-center/api-gateway-guide/apache-apisix-vs-kong)[62. Traefik vs Kong vs APISIX: A Reproducible API Gateway Benchmark Method](https://api7.ai/learning-center/api-gateway-guide/traefik-kong-apisix-benchmark-method)[63. Amazon API Gateway Explained: Deployment Models, Trade-offs, and Use Cases](https://api7.ai/learning-center/api-gateway-guide/amazon-api-gateway-explained)[64. API Gateway and Istio: Clear North-South and East-West Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-istio)[65. API Gateway with NGINX: Layering, Ownership, and Migration Patterns](https://api7.ai/learning-center/api-gateway-guide/api-gateway-nginx)[66. API Gateway with Envoy: Deployment Patterns and Policy Ownership](https://api7.ai/learning-center/api-gateway-guide/api-gateway-envoy)[67. API Gateway with Cloudflare API Shield: Edge and Origin Security Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-cloudflare-api-shield)[68. Apigee vs MuleSoft vs Amazon API Gateway: A Decision Framework](https://api7.ai/learning-center/api-gateway-guide/apigee-mulesoft-amazon-api-gateway)[69. Should an API Gateway Connect Directly to a Database? Patterns and Risks](https://api7.ai/learning-center/api-gateway-guide/api-gateway-database-access-patterns)[70. API Gateway Architecture for Large-Scale SaaS Platforms](https://api7.ai/learning-center/api-gateway-guide/api-gateway-large-scale-saas-architecture)

08 Industry Applications and Integrations

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=applications)[71. API Gateway CI/CD: Validation, Promotion, and Safe Rollback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ci-cd)[72. API Gateways for AI and ML Inference: Traffic, Security, and Observability](https://api7.ai/learning-center/api-gateway-guide/api-gateways-ai-ml-inference)[73. API Gateway Architecture for Financial Services: Controls and Failure Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-financial-services)[74. API Gateway Architecture for E-Commerce: Checkout Reliability and Traffic Peaks](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ecommerce)[75. API Gateways for Mobile Applications: Authentication, Payloads, and Versioning](https://api7.ai/learning-center/api-gateway-guide/api-gateways-mobile-applications)[76. API Gateways for Web3 Applications: RPC Traffic, Identity, and Abuse Controls](https://api7.ai/learning-center/api-gateway-guide/api-gateways-web3-applications)[77. OpenAPI-Driven API Gateway Operations: Import, Validation, and Drift Control](https://api7.ai/learning-center/api-gateway-guide/openapi-driven-api-gateway-operations)[78. gRPC-Web and gRPC Transcoding Through an API Gateway](https://api7.ai/learning-center/api-gateway-guide/grpc-web-grpc-transcoding-api-gateway)[79. API Gateway Pricing and Cost Optimization: A Workload-Based Model](https://api7.ai/learning-center/api-gateway-guide/api-gateway-pricing-cost-optimization)[80. Evaluating API Gateway Directions: Evidence and Architectural Impact](https://api7.ai/learning-center/api-gateway-guide/api-gateway-trends)

09 Advanced Gateway Applications

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=operations)[81. API Gateways in Zero Trust Architecture: Identity, Policy, and Limits](https://api7.ai/learning-center/api-gateway-guide/api-gateways-zero-trust-architecture)[82. API Gateway Log Analytics: Correlation, Queries, and Operational Signals](https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-analytics)[83. From API Sprawl to Governed Delivery: The Gateway's Role in Modernization](https://api7.ai/learning-center/api-gateway-guide/api-gateway-modernization)[84. API Monetization at the Gateway: Plans, Metering, Entitlements, and Billing Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-monetization-gateway)[85. Safe AI-Assisted API Gateway Optimization: Recommendations, Guardrails, and Approval](https://api7.ai/learning-center/api-gateway-guide/ai-assisted-api-gateway-optimization)[86. API Gateways for Low-Code and No-Code Integration: Governance and Safety](https://api7.ai/learning-center/api-gateway-guide/api-gateways-low-code-no-code-integration)[87. API Gateway and CIAM Integration: Token Validation, Consent, and Identity Context](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ciam-integration)[88. Elastic API Gateway Scaling: Signals, Capacity, and Safe Scale-In](https://api7.ai/learning-center/api-gateway-guide/elastic-api-gateway-scaling)[89. Multi-Tenant SaaS API Gateways: Isolation, Quotas, and Tenant-Aware Policy](https://api7.ai/learning-center/api-gateway-guide/multi-tenant-saas-api-gateway-policy)[90. API Gateway DevSecOps: Policy as Code, Testing, and Runtime Feedback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-devsecops)

10 Platform Engineering and Planning

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=platform-governance)[91. API Discovery and Documentation at the Gateway: Inventory, Ownership, and Drift](https://api7.ai/learning-center/api-gateway-guide/api-discovery-documentation-gateway)[92. WebAssembly Plugins for API Gateways: Portability, Isolation, and APISIX Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-webassembly-plugins)[93. Monitoring API Gateways with Prometheus and Grafana: Metrics, Dashboards, and Alerts](https://api7.ai/learning-center/api-gateway-guide/monitor-api-gateways-prometheus-grafana)[94. API Gateway Reliability Engineering: SLOs, Failure Isolation, and Incident Response](https://api7.ai/learning-center/api-gateway-guide/api-gateway-reliability-engineering)[95. API Gateways for Edge AI Inference: Routing, Constraints, and Observability](https://api7.ai/learning-center/api-gateway-guide/api-gateways-edge-ai-inference)[96. Reducing Cloud Cost with API Gateway Controls: Measurement Before Optimization](https://api7.ai/learning-center/api-gateway-guide/reducing-cloud-cost-api-gateway-controls)[97. Improving API Developer Experience with Gateway Standards and Self-Service](https://api7.ai/learning-center/api-gateway-guide/improving-api-developer-experience-gateway-self-service)[98. API Gateway Change Management: Compatibility, Progressive Delivery, and Rollback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-change-management-rollback)[99. Operating Serverless APIs: Cold Starts, Idempotency, and Asynchronous Handoffs](https://api7.ai/learning-center/api-gateway-guide/operating-serverless-apis-gateway)[100. Planning an API Gateway Roadmap for the Next Five Years](https://api7.ai/learning-center/api-gateway-guide/planning-api-gateway-roadmap-next-five-years)

[API Gateway Guide](https://api7.ai/learning-center/api-gateway-guide)[Platform Engineering and Planning](https://api7.ai/learning-center/api-gateway-guide?topic=platform-governance)

On this page

On this page

- [Key Takeaways](#key-takeaways)

- [Start with a User-Centered Reliability Contract](#start-with-a-user-centered-reliability-contract)

- [Map Failure Ownership Before Adding Controls](#map-failure-ownership-before-adding-controls)

- [Match the Control to the Failure](#match-the-control-to-the-failure)

- [Use APISIX Health Checks with the Pool Boundary in Mind](#use-apisix-health-checks-with-the-pool-boundary-in-mind)

- [Use a Circuit Breaker as a Route Policy, Not a Diagnosis](#use-a-circuit-breaker-as-a-route-policy-not-a-diagnosis)

- [Shed Load Before Collapse](#shed-load-before-collapse)

- [Govern Changes with Error Budgets](#govern-changes-with-error-budgets)

- [Build Incident Evidence into the Gateway Path](#build-incident-evidence-into-the-gateway-path)

- [Test the Reliability Contract](#test-the-reliability-contract)

- [Reliability Engineering Checklist](#reliability-engineering-checklist)

- [FAQ](#faq)

  - [Does a highly available gateway guarantee API reliability?](#does-a-highly-available-gateway-guarantee-api-reliability)
  - [Should APISIX fail closed when a policy dependency fails?](#should-apisix-fail-closed-when-a-policy-dependency-fails)
  - [Are health checks and circuit breakers interchangeable?](#are-health-checks-and-circuit-breakers-interchangeable)

- [Next Steps](#next-steps)

Article 94

# API Gateway Reliability Engineering: SLOs, Failure Isolation, and Incident Response

API7.ai

October 29, 2026

API gateway reliability is not the percentage of time a gateway process stays up. It is the gateway's contribution to a user-visible service objective: admitting valid requests, applying required policy, selecting an eligible upstream, and returning a correct response within the promised time.

The gateway can isolate some failures and expose useful evidence. It cannot guarantee application correctness, make unsafe retries safe, or recover a dependency whose capacity and data are gone. Reliability engineering begins by defining the user journey and failure ownership, then selecting controls that preserve that contract under stress.

## Key Takeaways

- Define SLIs from user outcomes before selecting the easiest gateway metrics.
- Separate gateway availability, policy dependency, upstream service, and end-to-end correctness failures.
- Treat health checks, retries, circuit breakers, load shedding, and traffic shifting as different controls.
- Document fail-open and fail-closed behavior for every external policy and state dependency.
- Use error budgets to govern rollout and recovery, not as permission to consume all remaining failure at once.
- During an incident, preserve configuration revision, response source, attempt count, and degraded-mode evidence.

## Start with a User-Centered Reliability Contract

[Google's SRE guidance on SLOs](https://sre.google/sre-book/service-level-objectives/) recommends working backward from what users care about and stating how and when the objective is measured. For a gateway-fronted order lookup, a contract might define:

- **Good event:** an authenticated, authorized request within the supported size limit returns an application-valid response in less than 500 ms.
- **Valid population:** production `GET /orders/{id}` requests, excluding approved synthetic tests and documented client cancellations.
- **Objective:** 99.9% good events over a rolling 28-day window.
- **Measurement points:** gateway client-facing outcome plus an application correctness signal.

The gateway can measure status and latency, but HTTP `200` alone cannot prove that the returned order is correct. Conversely, a gateway `429` may be deliberate load shedding rather than an upstream failure, yet it still affects the user unless the product contract excludes or separately budgets that admission class.

Avoid one SLO for unrelated workloads. Interactive reads, asynchronous submissions, streaming responses, and administrative APIs have different latency and correctness expectations.

## Map Failure Ownership Before Adding Controls

```
flowchart LR
    C[Client] --> E[Edge or load balancer]
    E --> G[Gateway data plane]
    G --> P[Policy and quota dependencies]
    G --> U[Upstream pool]
    U --> D[Application dependencies]
    CP[Gateway control plane] -. configuration stream .-> G
    G -. metrics logs traces .-> O[Observability systems]
```

For each arrow, record what happens when it is slow, unavailable, stale, or contradictory. The data plane may continue with its last valid configuration during a control-plane outage, but that trades change freshness for request continuity. A quota-store outage may fail closed to preserve an entitlement boundary or fail open to preserve availability. Neither choice is universally correct.

A useful failure table is explicit:

| Dependency          | Primary owner     | User-visible failure        | Intended degraded mode                      | Maximum duration       |
| ------------------- | ----------------- | --------------------------- | ------------------------------------------- | ---------------------- |
| Gateway instance    | Platform          | Connection or 5xx           | Remove instance from edge pool              | Probe and drain budget |
| Policy service      | Security/platform | Reject, bypass, or latency  | Fail closed or scoped cached decision       | Policy-specific        |
| Upstream node       | Service owner     | Connect error, timeout, 5xx | Select another eligible node if replay-safe | Request deadline       |
| Configuration store | Platform          | Stale or unavailable change | Last valid configuration                    | Staleness budget       |
| Telemetry backend   | Observability     | Lost or delayed evidence    | Bounded local buffer or drop policy         | Retention budget       |

“Keep serving” is incomplete unless the team knows which policy, configuration, or evidence may be stale or missing.

## Match the Control to the Failure

| Control                 | Main purpose                                                     | It does not prove                                  |
| ----------------------- | ---------------------------------------------------------------- | -------------------------------------------------- |
| Active health check     | Detect a node through synthetic probes and allow recovery probes | That every operation and dependency is healthy     |
| Passive health check    | Learn from proxied request outcomes                              | Recovery of a node that no longer receives traffic |
| Retry                   | Make another bounded attempt for replay-safe work                | That the first attempt had no effect               |
| Circuit breaker         | Stop or reduce calls after configured failure evidence           | Root cause or semantic correctness                 |
| Rate/concurrency limit  | Bound admitted work                                              | Fairness or downstream success                     |
| Traffic split           | Shift a controlled share between releases or pools               | Exact ratios in every short window                 |
| Cache or stale response | Preserve selected reads during dependency failure                | Freshness or suitability for every resource        |

Stacking all controls can amplify rather than reduce failure. A client retry, gateway retry, service-mesh retry, and application retry multiply work. A circuit breaker that returns quickly may make an upstream look less loaded while users still fail. A stale cache can preserve reads but violate a freshness or revocation requirement.

## Use APISIX Health Checks with the Pool Boundary in Mind

Apache APISIX 3.19.0 supports [active and passive upstream health checks](https://apisix.apache.org/docs/apisix/3.19/tutorials/health-check/). Active checks use HTTP, HTTPS, or TCP probes. Passive checks classify proxied outcomes.

Passive checks alone cannot mark an unhealthy node healthy again because the node no longer receives requests; combine them with active checks when automatic recovery detection is required. APISIX also documents a pool-level fail-open boundary: if no healthy node can be selected, it continues to access the upstream. That behavior avoids treating health classification as an absolute traffic stop, but it can send requests to a failing pool.

Decide whether that behavior matches the service contract. If contacting an unhealthy node is worse than a controlled gateway response, add an explicit admission or breaker policy and test the exhausted-pool path. Do not claim that enabling health checks guarantees that traffic reaches only healthy nodes.

Probe design matters. A shallow TCP probe can show that a process accepts connections while its database is unavailable. A deep probe can create load or take the service out because of one non-critical dependency. Use separate liveness, readiness, and diagnostic signals, and keep the routing probe bounded.

## Use a Circuit Breaker as a Route Policy, Not a Diagnosis

APISIX 3.19.0's [`api-breaker` plugin](https://apisix.apache.org/docs/apisix/3.19/plugins/api-breaker/) considers configured upstream response statuses and failure counts, then returns a configured response while the circuit is open. It probes recovery on an exponential schedule up to `max_breaker_sec` and closes after the configured healthy successes.

This excerpt deliberately chooses values rather than relying on defaults:

```
{
  "api-breaker": {
    "break_response_code": 503,
    "break_response_body": "Service temporarily unavailable",
    "max_breaker_sec": 30,
    "unhealthy": {
      "http_statuses": [500, 502, 503, 504],
      "failures": 5
    },
    "healthy": {
      "http_statuses": [200, 204],
      "successes": 2
    }
  }
}
```

This is a plugin fragment, not a complete Route. It uses HTTP status as evidence; it does not inspect application correctness, distinguish every failure cause, or replace per-node health checks. Choose statuses from application semantics. A `500` caused by one invalid input should not necessarily remove a whole service, while a successful probe may not prove that the original operation recovered.

The synthetic `503` is generated by the gateway when the circuit is open. Preserve response-source telemetry so on-call engineers do not attribute it to the upstream.

## Shed Load Before Collapse

When demand exceeds safe capacity, an early bounded rejection is often better than letting queues grow until every request times out. Define:

- the protected resource: connections, gateway workers, upstream concurrency, database capacity, or model queue;
- the admission key and its trusted source;
- priority classes and starvation safeguards;
- rejection status and retry guidance;
- maximum queue and deadline;
- the signal that ends degraded mode.

Rate limiting protects a rate window; concurrency limiting protects simultaneous work. Neither knows business priority unless policy supplies it. A caller-supplied priority header is not authoritative until authenticated identity and server-side entitlement map it to a class.

## Govern Changes with Error Budgets

An error budget is the failure allowance implied by an SLO, not spare outage time to consume casually. Use burn rate and remaining budget to decide whether to:

- continue or pause a rollout;
- reduce canary weight;
- disable a costly optional plugin;
- stop feature work and repair reliability;
- invoke a documented degraded mode.

APISIX 3.19.0's [`traffic-split` plugin](https://apisix.apache.org/docs/apisix/3.19/plugins/traffic-split/) can direct traffic to weighted upstreams. Its documentation warns that ratios may be less accurate because weighted round robin is used, especially after state resets. Treat a weight as routing intent, not a guarantee that every short interval receives that exact percentage.

Compare the canary and baseline on the same valid request population. Gate promotion on user-visible error, latency, saturation, and semantic checks; a healthy gateway process is not enough.

## Build Incident Evidence into the Gateway Path

During an incident, answer these questions quickly:

1. Which user journey and SLO are burning?
2. Which gateway clusters, Routes, Services, and revisions are affected?
3. Did APISIX, NGINX, or the upstream generate the response?
4. How much latency occurred in the gateway versus upstream waiting?
5. Which health, breaker, retry, cache, or load-shed state was active?
6. How many upstream attempts came from one client request?
7. Did a configuration, deployment, certificate, policy, or dependency change precede the event?
8. Is the mitigation preserving security, consistency, and evidence obligations?

Keep identifiers useful for correlation without treating them as identity. A request ID can join metrics, logs, and traces; if the caller can supply or preserve it, it is still correlation context rather than proof of who acted.

## Test the Reliability Contract

Run controlled exercises for:

- one gateway instance failing while others remain healthy;
- control-plane unavailability and stale configuration;
- slow, refused, reset, and application-error upstreams;
- every upstream node classified unhealthy;
- policy and quota dependencies failing open or closed as designed;
- a mutation timing out after it may have been applied;
- metrics, logging, and tracing loss during a user-visible incident;
- canary regression and rollback under live load;
- recovery without a traffic surge or retry storm.

Assert end-to-end invariants: maximum attempts, maximum latency, allowed degraded behavior, configuration age, and absence of duplicate effects. Do not assert only one status code if the failure phase can legitimately produce several.

## Reliability Engineering Checklist

- Does each workload have a user-centered SLI, objective, window, and valid population?
- Are application correctness and gateway status measured separately?
- Is ownership defined for every dependency and degraded mode?
- Are health checks, retries, breakers, and load shedding used for distinct purposes?
- Is APISIX's no-healthy-node fail-open behavior tested and acceptable?
- Are policy dependencies explicitly fail-open or fail-closed with a time bound?
- Are retries limited to replay-safe work within the remaining deadline?
- Do rollout gates use observed canary outcomes rather than configured weight alone?
- Can telemetry distinguish gateway-, NGINX-, and upstream-generated responses?
- Does the incident runbook preserve required security and audit controls?

## FAQ

### Does a highly available gateway guarantee API reliability?

No. It removes one infrastructure failure mode. End-to-end reliability also depends on policy services, upstreams, data stores, networks, clients, and semantic correctness.

### Should APISIX fail closed when a policy dependency fails?

It depends on the policy. Authorization and entitlement checks often require fail-closed behavior; optional enrichment may safely fail open. Record the decision, scope, duration, and alert for each dependency.

### Are health checks and circuit breakers interchangeable?

No. Health checks classify node reachability or responses. A breaker stops calls according to a failure policy at its configured scope. They observe different evidence and have different recovery behavior.

## Next Steps

Instrument the contract with [Prometheus and Grafana gateway monitoring](https://api7.ai/learning-center/api-gateway-guide/monitor-api-gateways-prometheus-grafana), bound attempts through [safe timeout and retry policies](https://api7.ai/learning-center/api-gateway-guide/api-gateway-timeouts-retries), and connect capacity to [elastic gateway scaling](https://api7.ai/learning-center/api-gateway-guide/elastic-api-gateway-scaling).

[Previous article**Monitoring API Gateways with Prometheus and Grafana: Metrics, Dashboards, and Alerts**](https://api7.ai/learning-center/api-gateway-guide/monitor-api-gateways-prometheus-grafana)[Next article**API Gateways for Edge AI Inference: Routing, Constraints, and Observability**](https://api7.ai/learning-center/api-gateway-guide/api-gateways-edge-ai-inference)

On this page

- [Key Takeaways](#key-takeaways)

- [Start with a User-Centered Reliability Contract](#start-with-a-user-centered-reliability-contract)

- [Map Failure Ownership Before Adding Controls](#map-failure-ownership-before-adding-controls)

- [Match the Control to the Failure](#match-the-control-to-the-failure)

- [Use APISIX Health Checks with the Pool Boundary in Mind](#use-apisix-health-checks-with-the-pool-boundary-in-mind)

- [Use a Circuit Breaker as a Route Policy, Not a Diagnosis](#use-a-circuit-breaker-as-a-route-policy-not-a-diagnosis)

- [Shed Load Before Collapse](#shed-load-before-collapse)

- [Govern Changes with Error Budgets](#govern-changes-with-error-budgets)

- [Build Incident Evidence into the Gateway Path](#build-incident-evidence-into-the-gateway-path)

- [Test the Reliability Contract](#test-the-reliability-contract)

- [Reliability Engineering Checklist](#reliability-engineering-checklist)

- [FAQ](#faq)

  - [Does a highly available gateway guarantee API reliability?](#does-a-highly-available-gateway-guarantee-api-reliability)
  - [Should APISIX fail closed when a policy dependency fails?](#should-apisix-fail-closed-when-a-policy-dependency-fails)
  - [Are health checks and circuit breakers interchangeable?](#are-health-checks-and-circuit-breakers-interchangeable)

- [Next Steps](#next-steps)
