AISIX 1.4.0: Redis Startup Resilience and Bounded Usage Retry

When an AI application is receiving requests, its gateway may be waiting on more than a model provider. Redis can hold shared rate-limit counters or cache state; etcd can hold configuration; and the control plane receives usage data. An outage in one of those dependencies should not automatically turn into an outage for every caller—but keeping traffic flowing is useful only if operators can see what is degraded and what delivery guarantees have changed.

AISIX 1.4.0, released September 22, 2026, makes that operational boundary much clearer. Its main story is not “dependencies no longer matter.” It is that the gateway can keep serving through selected startup and delivery failures while exposing the limits of that degraded path.

The deployment boundary still matters: AISIX processes gateway traffic in your environment. AISIX Cloud provides the control plane, dashboard, organizations, and centralized usage views; it can be deployed as On-Premises or Hybrid Cloud. The gateway and its dependencies remain your operating responsibility in either model.

Key Takeaways

AISIX 1.4.0 lets a gateway bind and serve when its Redis rate-limit or cache backend is unreachable at startup, but the resulting behavior is deliberately degraded rather than equivalent to a healthy shared backend.
Usage retries are bounded and deduplication-aware, but a failure with no response inherits the sender's last observed deduplication signal. Complete the 1.4.0 control-plane rollout before relying on retry; mixed-version replicas or a rollback do not carry an idempotency guarantee.
Credentialed etcd deployments now have a 5-second default dial timeout; check any legitimate slower path before upgrading, or set etcd.dial_timeout_ms explicitly.
Configuration errors, a full telemetry queue, an expired retry budget, and an older control plane remain meaningful failure paths. AISIX Cloud budget checks can also deny traffic during a control-plane outage. Treat the release as a resilience upgrade with verification work, not as a blanket availability promise.

Start the Gateway, Then Identify What Has Degraded

Consider a production deployment whose Redis instance is temporarily unreachable when a gateway restarts. Before 1.4.0, a Redis rate-limit or cache connection could leave the gateway without bound listeners during startup. In 1.4.0, the gateway binds and serves while it retries the shared backend in the background.

flowchart LR
  R[Redis unavailable at startup] --> G[AISIX gateway binds and serves]
  G --> L[Rate limiting: per replica]
  G --> C[Redis cache: cache misses]
  G --> B[Background reattach]
  B -->|Redis answers| H[Shared backend resumes]

The two degraded paths are not interchangeable:

With ratelimit.backend: redis, counting becomes per replica while Redis is unavailable. Cluster-wide limits are not enforced for that period; AISIX does not permanently switch the backend to memory.
With a Redis-backed cache, policies serve misses while Redis is unreachable. In that state, semantic policies do not make an embedding call or Redis round trip. If the exact Redis connection recovers while the vector-search probe is still pending, exact lookups can use Redis; semantic vector matching remains unavailable until the probe succeeds.
Redis replies that refuse credentials or a selected database also take the degraded path, but are identified as reason=refused; an unreachable backend is reason=unreachable. Correcting credentials on the Redis side can therefore be adopted without restarting the gateway.

This behavior does not make boot permissive for malformed configuration. An unparsable Redis url, unreadable TLS material, missing mode-specific fields, or timeout_secs: 0 is rejected locally before a network call and still prevents startup. That distinction preserves the useful signal from a configuration mistake while allowing a recoverable remote dependency outage.

Keep Usage Retries Bounded and Deduplication-Aware

Serving traffic is only half of the recovery story. If a control-plane request fails after a gateway has accumulated usage events, AISIX 1.4.0 can resend the same batch with the same batch ID. A retryable response must advertise batch deduplication for a resend. If there is no response, the sender instead uses the deduplication signal from its most recent response: without a prior signal it drops the batch, while a later response without the signal clears the remembered capability.

flowchart LR
  E[Usage batch] --> S[Send to control plane]
  S -->|Accepted| I[Delivered]
  S -->|Retryable response with dedup signal| R[Resend same batch ID]
  S -->|No response; prior signal retained| R
  S -->|No signal or non-retryable response| D[Drop with reason metric]
  R -->|Compatible control plane accepts| I
  R -->|Signal absent or retry budget exhausted| D

The 1.4.0 control plane claims a batch ID in the same transaction that writes its rows, so it can answer a repeated batch without writing those rows again. This does not make a mixed-version control-plane fleet safe for retries. After a newer replica advertises deduplication, a request to an older replica can fail without a response; the gateway may then resend based on the earlier signal even though that older replica cannot deduplicate. The sender's remembered signal is not proof of which replica processed the failed attempt. Do not rely on duplicate-free ingestion during a mixed-version rollout or rollback.

The retry window is intentionally finite: its 30-minute budget is measured from the oldest event in the batch, and eight consecutive failures that carried a response also exhaust the retry budget. A final attempt can start at the deadline and finish shortly afterward. A batch that the control plane says it can never store receives 422 and is dropped immediately. Later batches wait behind the batch being resent, so console usage may arrive minutes late during an outage. Track aisix_usage_event_drops_total and its new send_failed and retry_budget_exhausted reasons instead of assuming every event was eventually delivered.

Duplicate-free retry requires a 1.4.0 gateway and a control-plane fleet whose every replica supports batch deduplication. A response from an older replica lacks the signal and stops retry; a no-response failure after an earlier signal is the mixed-version exception described above. Do not treat a configuration change as a substitute for upgrading all replicas.

The same release distinguishes a temporary control-plane inability to read the revocation list from an unacceptable certificate: gateway /dp routes receive retryable 503 MTLS_UNAVAILABLE for the former and 401 for the latter. Neither case fails open.

Usage resend is a telemetry-delivery path, not a pre-request authorization decision; it does not guarantee that every request will pass a control-plane outage. For AISIX Cloud budget checks, the gateway denies a request if the control plane is unreachable and no cached decision exists. It can reuse a cached decision within the stale ceiling (600 seconds by default); after that, sticky and fail-closed modes deny traffic, while fail-open permits it. This budget behavior is separate from the 1.4.0 usage-delivery change.

Bound Configuration Recovery Before the Upgrade

AISIX 1.4.0 also prevents a credentialed etcd dial from holding startup indefinitely. etcd.dial_timeout_ms now defaults to 5000. One etcd configuration-provider dial gets a budget of dial_timeout_ms × max(1, nonempty endpoints). Startup dials two such providers sequentially—the environment prefix and shared pricing catalog—so their dial waits can total twice that budget when the cluster is unreachable. This is not a bound on separate etcd requests.

This default affects deployments that set etcd.user; without credentials, the dial performs no I/O. An expired dial follows the existing recovery path: it warns, binds, serves from a snapshot cache when one exists, and retries in the background. Set etcd.dial_timeout_ms: 0 to retain the previous unbounded dial behavior. request_timeout_ms is separate and remains unbounded by default.

The resilience path has an important precondition: a snapshot must be available to serve. Do not translate “binds after an etcd timeout” into “a newly started gateway has configuration without etcd.” Test a restart with the actual cache state and expected outage before relying on it in a recovery procedure.

Make the Operational Contract Visible

Several supporting changes reinforce the same goal: avoid turning a dependency symptom into an opaque gateway outage.

Situation	AISIX 1.4.0 behavior	Operational check
Stalled stderr consumer	Log events use a bounded queue; when it fills, new log lines are dropped and counted instead of blocking request workers.	Alert on `aisix_log_lines_dropped_total`; a full queue is data loss, not a healthy log path.
Redis circuit-breaker recovery	A background probe checks recovery after the 30-second window.	Do not assign the recovery probe's timeout cost to the next business request.
Unknown configuration resource kind	The resource is reported separately from rejected resources.	Update alerts that treated all `aisix_config_rejected_resources` as “unloadable.”
New metric families	Four new metric families exist in 1.4.0.	Do not add them to `observability.metrics.labels` until every gateway is on 1.4.0; an older gateway refuses unknown families at startup.

These are not a substitute for capacity planning or external dependency monitoring. They make the degradation explicit enough to observe and test.

Run an Upgrade Drill That Matches Your Deployment

Before rollout, verify the paths this release changes:

For credentialed etcd, measure the legitimate connection time for each configured endpoint. Set etcd.dial_timeout_ms explicitly if five seconds is too short; use 0 only when retaining an unbounded dial is intentional.
Review redis.username and redis.password in single mode. These fields now apply and override URL credentials as a pair, so a stale explicit value can turn a previously working connection into a refused, degraded one.
Test a Redis-startup outage. Confirm that listeners bind, that rate-limit behavior is per replica, that cache policies miss, and that your alerts differentiate reason=unreachable from reason=refused.
Finish upgrading every control-plane replica to 1.4.0 before relying on usage resend, then upgrade the gateways. Do not assume duplicate-free delivery during a mixed-version rollout or rollback. Test response-bearing and no-response failures separately; verify delayed ingestion, batch deduplication, and the drop metrics.
Update alerts for the separate unknown-kind metric and stage new metric-label configuration only after the whole environment is at 1.4.0. Account for the extra one-time cp-api index work and ongoing storage/write maintenance on a large dpmgr_usage_events table.
If AISIX Cloud budgets are enabled, test control-plane loss with no cached budget decision and after the stale ceiling under the configured outage mode. Do not use usage-retry behavior as evidence that budget-gated requests will continue.

Read the AISIX 1.4.0 release notes alongside the upgrade procedure, including intervening releases when your upgrade skips versions.

Keep Serving—Without Hiding the Trade-Offs

AISIX 1.4.0 is valuable because it narrows the gap between a dependency outage and a gateway outage. A temporary Redis loss no longer has to prevent startup, a credentialed etcd dial has a defined bound, and a compatible control plane can accept a resent usage batch without duplicating it.

Those outcomes have conditions: shared rate limits and Redis caching degrade, duplicate-free usage resend requires a fully compatible control-plane fleet, AISIX Cloud budget checks can still deny traffic, and local configuration errors still stop boot. Make those conditions part of the runbook, then test them with the dependencies and recovery targets you actually operate. Start with the AISIX release notes and the AISIX deployment documentation.