Partial API Outages Need Retry Budgets, Not Retry Storms

During a partial outage, some requests succeed while others time out. A client may not know whether a write happened before the connection disappeared. A “retry three times” rule adds traffic and can duplicate effects.

GitHub offered a current example on November 4, 2026. Its permanent incident record reported elevated errors and intermittent failures affecting Git Operations, API Requests, Issues, and Pull Requests during a morning incident. The accompanying Hacker News discussion shows the user experience: results differed by operation and region, and repeated commands became an improvised recovery strategy.

GitHub said a root-cause analysis would follow; this article does not infer a cause. When a dependency is partly available, which layer may retry an operation, for how long, and with what protection against duplicate work?

Key Takeaways

• A transport failure does not prove that an upstream operation failed before taking effect.
• Retry policy starts with operation semantics and failure phase, not a universal attempt count.
• Give one layer ownership of retries and enforce a total deadline and attempt budget.
• Use exponential backoff with jitter for retryable transient failures.
• A gateway can bound upstream attempts and timeouts; it cannot make a non-idempotent business operation safe.
• Test delayed success, lost responses, overload, and recovery—not only immediate connection refusal.

Classify the Operation Before the Error

HTTP Semantics defines an idempotent method as one whose intended effect is the same after one or several identical requests. PUT, DELETE, and safe methods are idempotent by definition; POST is not generally idempotent. A client can retry an idempotent request automatically after a communication failure, while retrying a non-idempotent request requires knowledge that the request is safe or a way to detect whether it already succeeded.

Method names are only the starting point. A nominally idempotent handler can still append duplicate audit events or send repeated notifications. An idempotency key can make a particular POST operation retry-safe when the service binds the key to the caller and request payload, coordinates concurrent uses, and reliably records the effect and outcome.

For example, atomically reserve a key before processing and make concurrent duplicates wait or return an in-progress result; reject reuse with a different payload. A reservation alone is not enough: if the service crashes after charging a payment but before recording the outcome, blindly executing again can charge twice. Couple the reservation, effect, and outcome transactionally where possible, or use a durable recovery and reconciliation protocol with an idempotent downstream operation. Define expiry and recovery for abandoned in-progress records. The AWS idempotent API guidance explains the atomicity requirement.

Create an operation registry rather than a retry rule per status code:

Operation class	Example	Automatic retry condition
Read with no side effect	Fetch deployment state	Within deadline after a retryable transport or service failure
Idempotent state change	Replace configuration at a known resource URI	Same precondition and payload; stop on conflict
Deduplicated command	Create payment with an idempotency key	Key reservation, effect, and recovery prevent concurrent or crash-induced duplicates
Non-idempotent command	Trigger an unkeyed deployment	Do not automatically retry after an ambiguous send

Record the owner, attempt and concurrency limits, deadline, and reconciliation method.

Locate the Failure Phase

“Timeout” compresses several different states:

flowchart LR
    Client -->|request| Gateway[API gateway]
    Gateway -->|attempt| Dependency[Dependency API]
    Dependency -->|effect may commit| Store[(State)]
    Dependency -. response lost or delayed .-> Gateway
    Gateway -. bounded result .-> Client

• A connection may fail before the request is sent.
• The body may be partly or fully sent before the connection drops.
• The dependency may commit the operation and lose the response.
• A queue may accept work but delay its webhook or status update.
• The gateway may time out while the dependency continues processing.

Only the first case clearly indicates that the dependency did not receive the request. Even then, name resolution, proxies, and reused connections can complicate what the client observes. Log the phase you can establish, but do not turn “no response” into “no side effect.”

For ambiguous writes, reconcile. Query by idempotency key, operation ID, resource version, or business key before sending another command. If the API offers no reconciliation key, the safe answer may be to stop and require operator or application-specific recovery.

Give Retries One Owner

A client, SDK, service mesh, gateway, and backend library can each retry. Three attempts at three layers can multiply upstream calls, particularly when an overloaded dependency receives synchronized retries.

Choose the layer with the best view of operation semantics. An SDK may understand idempotency and client deadlines; a gateway may see upstream availability and enforce a fleet-wide attempt ceiling. Other layers should either disable retries or consume a shared budget.

Express the budget as constraints rather than one magic number:

• maximum upstream attempts, including the first;
• end-to-end deadline;
• per-attempt connect, send, and read timeouts;
• maximum concurrent retries per route or dependency;
• retryable failure classes;
• backoff schedule with jitter;
• maximum stale or fallback response age, if a fallback exists.

The AWS Builders' Library guidance explains why retries can amplify overload and why backoff plus jitter spreads recovery traffic. A retry budget should shrink when the original request volume or error rate rises, not remain an unlimited percentage of a failing dependency's capacity.

Configure APISIX Without Inventing Idempotency

Apache APISIX 3.19 exposes upstream retries, retry_timeout, and separate connect, send, and read timeouts in the Admin API upstream schema. Its documentation says retries defaults according to the number of available backend nodes; setting it to 0 disables retries. Setting retry_timeout to 0 disables that retry timeout.

Those knobs bound gateway behavior. They do not establish whether a request is safe to repeat, whether an upstream committed a side effect, or whether another client layer is also retrying. APISIX configures retry attempts, while the effective NGINX proxy settings and request phase determine whether another upstream attempt is eligible.

Under the NGINX HTTP proxy defaults, error and timeout are eligible failure types; configured settings can differ. A POST, LOCK, or PATCH already sent upstream is normally not passed to another server unless non_idempotent is enabled. Switching is possible only before anything has been sent to the client; with request buffering disabled, a body that has begun sending upstream cannot be passed to another server. Before enabling retries on a write route, test the effective configuration, method, body behavior, failure phase, and upstream implementation against the pinned APISIX and NGINX versions you operate.

Use route-specific policy. A GET route backed by several equivalent replicas may justify a small retry budget. A POST command without an idempotency key may justify retries: 0, a short deadline, and an explicit unknown-outcome response from the application. A large streamed upload has different send and replay costs from a small metadata read.

Health checks and circuit breakers solve adjacent problems, not the same one. A health check can remove a failing node after evidence accumulates; it does not explain an already ambiguous request. A breaker can reduce calls after configured failures; it does not deduplicate committed work. API7's earlier overview of API resilience patterns describes those mechanisms broadly. The retry-budget layer connects them to one operation's semantics.

Make Overload Responses Actionable

A dependency that knows it is overloaded can return 429 or 503 and may include Retry-After. Clients should honor a valid instruction within their own deadline and retry budget, rather than layering an unrelated rapid loop on top. Absence of Retry-After is not permission for immediate synchronized retries.

Separate permanent from transient failures:

• most authentication, authorization, schema, and business-rule 4xx responses should not be retried unchanged;
• a conflict may require reading the new resource version and making a new decision;
• 429 or 503 may be retryable after delay when the operation is safe;
• a timeout or connection loss needs the failure-phase and idempotency analysis above;
• a malformed response is not automatically safer to repeat than an explicit error.

Return a stable error class and operation identifier without exposing secrets or raw upstream bodies.

Observe the Budget, Not Just the Error Rate

An outage dashboard needs to show how recovery policy changes load. Record:

• original requests and additional attempts separately;
• attempt number and retry owner;
• remaining deadline at each attempt;
• failure phase and selected upstream;
• idempotency-key presence, without logging sensitive key material;
• outcome: succeeded, rejected, timed out, reconciled, or unknown;
• time spent in backoff and time spent waiting on the dependency.

Use a gateway-generated or sanitized request identifier for traffic correlation. A caller-supplied identifier is useful for joining events only after collision and spoofing risks are handled; it is not trusted identity.

Alert on retry amplification—the ratio of upstream attempts to original requests—alongside latency and error rate. A falling client error rate can hide a large increase in dependency load if retries eventually succeed.

Test the Ambiguous Paths

Failure testing should include more than returning 503 immediately. Use a staging dependency or controlled fault injection to simulate:

1. connection refusal before send;
2. slow connect, slow response headers, and slow body;
3. a committed write followed by a dropped response;
4. a 429 or 503 with and without Retry-After;
5. duplicate delivery with the same idempotency key;
6. simultaneous recovery by many clients;
7. an unhealthy node re-entering service;
8. delayed webhook or job-status events after the synchronous call fails.

Assert the total upstream attempt count, not only the final response. Confirm that a committed command produces one business effect, that retry traffic stays within its concurrency and deadline limits, and that the final state can be reconciled after an unknown outcome.

The fault-injection plugin can generate configured delays or aborts for APISIX test routes. It cannot reproduce every after-commit failure inside a dependency, so combine gateway-layer faults with an instrumented test service.

Budget Recovery Before the Next Incident

Retries are borrowed capacity. They can recover from a short partial failure, but they spend additional dependency work and caller time. Without idempotency, a clear owner, and an end-to-end budget, they can convert uncertainty into duplicate effects or a larger outage.

Start with one high-value dependency. List its operations, mark which are idempotent or deduplicated, disable competing retry layers, and test a lost response after commit. Then set gateway attempts and timeouts from that evidence rather than from a universal default.

For centralized traffic policy, review API7 Enterprise and the API7 Gateway upstream documentation, while keeping idempotency and reconciliation in the application contract that owns the business effect.