API7 Gateway 3.10.3: Safer Enterprise Upgrades by Design

Key Takeaways

• Enterprise API gateway upgrades are operational changes, not image-tag changes. Capacity, proxy trust, identity, session behavior, secrets, and component order all affect whether a rollout is safe.
• API7 Gateway 3.10.3 documents about 365 MiB of additional default Data Plane shared memory, giving Kubernetes operators a concrete capacity check before rollout.
• Client-supplied X-Forwarded-Proto, X-Forwarded-Host, X-Forwarded-Port, and RFC 7239 Forwarded headers are now accepted only from addresses explicitly trusted through apisix.trusted_addresses; other values are replaced or cleared.
• Built-in Dashboard users gain TOTP two-factor authentication and login-failure restrictions, while new or changed passwords must contain at least 12 characters.
• Additional AI Cache credentials are encrypted at rest, making Control Plane and Data Plane upgrade order especially important during the mixed-version window.
• Safer configuration validation, secret resolution, compatibility reporting, and configuration resynchronization reduce the chance that an invalid or stale state quietly reaches production traffic.

An API gateway upgrade changes one of the most connected components in the platform. The gateway sits between clients and services, receives identity and network context from load balancers, applies security policy, and exports the signals operators use to understand production traffic. Even a release with no application-code changes can therefore affect capacity, authentication, request metadata, and incident response at the same time.

That makes a simple rollout question—“Does the new container start?”—too narrow. A healthy process also asks whether the container has enough memory, whether forwarded headers still reflect the real client, whether administrative access has changed, whether mixed-version components agree on encrypted configuration, and whether every gateway instance received valid state.

API7 Gateway 3.10.3 addresses this broader upgrade problem. The release strengthens trust boundaries and identity controls while making several operational consequences explicit. It also improves the validation and synchronization paths that help a distributed gateway fleet converge on the intended configuration.

The result is a useful model for enterprise API management: treat upgrade safety as a property of the whole traffic platform, not a final checkbox after deployment.

flowchart LR
    release[3.10.3 Release] --> capacity[Capacity Boundary]
    release --> network[Network Trust Boundary]
    release --> identity[Identity Boundary]
    release --> secrets[Secret Boundary]

    capacity --> memory[Shared Memory and Kubernetes Limits]
    network --> headers[Trusted Forwarded Headers]
    identity --> admin[2FA, Lockout, and Tokens]
    secrets --> versions[Control Plane and Data Plane Order]

    memory --> rollout[Safer Enterprise Rollout]
    headers --> rollout
    admin --> rollout
    versions --> rollout

Upgrade Risk Is Distributed Across the Gateway Platform

Enterprise gateways rarely run as one process on one host. A typical deployment combines a Control Plane, multiple Data Plane instances, an external database, load balancers or reverse proxies, identity providers, monitoring systems, and automation that updates configuration. Kubernetes adds resource requests, limits, node pressure, eviction behavior, and rolling replacement to that system.

A change at any boundary can produce symptoms somewhere else. If a pod is OOM-killed, clients see availability problems. If a proxy is not recognized as trusted, an upstream application may see a different scheme or host. If a login policy becomes stricter, an automation using Basic Auth may stop working. If the Control Plane encrypts a value that an older Data Plane cannot decrypt, the affected plugin can fail during rollout even though both components work independently at steady state.

API7 Gateway 3.10.3 makes these dependencies visible in its upgrade notes. That visibility matters because it turns hidden assumptions into deployment checks. Platform teams can then test the actual contract between components instead of validating each component in isolation.

Capacity Planning Starts Before the First New Pod

The most immediate infrastructure change in 3.10.3 is higher default Data Plane shared memory. According to the release notes, the default lua_shared_dict allocations together reserve about 365 MiB more memory at startup than 3.10.2.

The increases support advanced Prometheus metrics, Kubernetes and other service-discovery integrations, SkyWalking tracing, and Developer Portal API-call tracking. These dictionaries are allocated when the gateway starts, even when the corresponding capability is not actively used. The additional reservation therefore applies to every Data Plane using the defaults, not only to instances with all of those features enabled.

For Kubernetes teams, this is a rollout dependency. A pod can pass configuration validation yet still be terminated when its memory limit leaves no room for the new reservation plus normal runtime usage. Node memory pressure may also change scheduling and eviction behavior when several gateway replicas are replaced together.

Before upgrading, teams should compare the 3.10.3 defaults with current pod requests and limits, measured workload headroom, replica count, and node capacity. The release notes also explain that deployments not using a particular feature can lower the relevant dictionary toward its earlier value through gateway or Helm configuration. That is a deliberate capacity decision: right-size the feature after confirming it is unused, rather than reducing memory blindly to make a pod start.

This is also why a canary should receive representative traffic. Startup proves that shared memory can be allocated; sustained requests, metrics, discovery updates, and tracing prove that the pod has enough headroom to operate.

Proxy Trust Should Be Explicit, Not Inherited from the Request

Forwarded headers look like ordinary metadata, but upstream applications often use them for security-sensitive decisions. X-Forwarded-Proto may determine whether a secure cookie is issued. X-Forwarded-Host may influence redirects or generated links. The standardized Forwarded header can carry client and proxy context. If arbitrary clients can supply these values, the application may act on an invented view of the request path.

API7 Gateway 3.10.3 introduces apisix.trusted_addresses as the decision point for client-supplied X-Forwarded-Proto, X-Forwarded-Host, X-Forwarded-Port, and RFC 7239 Forwarded headers. When the setting is absent or the request does not come from a trusted address, the gateway overwrites the forwarded protocol, host, and port with values it observed and clears the Forwarded header before proxying upstream.

If a known load balancer or reverse proxy supplies authoritative values, operators can add its IP address or CIDR to trusted_addresses. This changes the model from “trust the header because it exists” to “trust it because an approved intermediary supplied it.”

flowchart TD
    request[Incoming Request with Forwarded Headers] --> source{Source address trusted?}
    source -- Yes --> preserve[Accept proxy-supplied forwarding context]
    source -- No --> replace[Overwrite X-Forwarded protocol, host, and port]
    replace --> clear[Clear RFC 7239 Forwarded header]
    preserve --> upstream[Proxy to Upstream]
    clear --> upstream

This safer default still requires preparation. Inventory every component immediately in front of the gateway, resolve the addresses the gateway actually sees, and account for network address translation. Then test redirect URLs, secure-cookie behavior, host-based application logic, and request logs from both trusted and untrusted paths. The goal is not merely to preserve old behavior; it is to preserve only the behavior backed by an intentional trust relationship.

Administrative Identity Needs Human and Machine Paths

Gateway administrators and automation do not authenticate in the same way. Human users benefit from interactive security controls such as a second factor and login throttling. Automation needs revocable, non-interactive credentials that do not depend on a browser flow.

API7 Gateway 3.10.3 strengthens both paths. Built-in Dashboard users can enroll in TOTP two-factor authentication, store recovery codes, and complete an OTP step during sign-in. Administrators can reset a user's 2FA state when recovery is necessary. The release also adds login-failure restrictions: by default, five consecutive failures temporarily block the built-in user and source IP for 15 minutes and generate an audit event. New passwords and passwords changed after the upgrade must be at least 12 characters, while existing passwords are not revalidated at sign-in.

The machine path is deliberately separate. Once a built-in user enables 2FA, HTTP Basic Auth for that user is rejected because Basic Auth cannot carry the second factor. Programmatic integrations should use access tokens sent through X-API-KEY. The Dashboard token guidance recommends separate, revocable tokens for routine access instead of depending on the initial administrator account.

Developer-facing identity also becomes more enforceable. The Developer Portal can require two-factor enrollment before a developer accesses protected pages, with the requirement checked at sign-in and at the proxy layer. This is important because a UI-only prompt is not a complete access boundary; server-side enforcement prevents a client from bypassing the intended enrollment gate.

Finally, teams using the OpenID Connect plugin should review session behavior. The plugin no longer enables a 900-second refresh_session_interval by default. Silent periodic re-authentication happens only when the interval is configured explicitly. If an environment depended on the old behavior, setting refresh_session_interval: 900 preserves it; otherwise, the new default avoids an unrequested session refresh policy.

Together, these changes encourage a clearer identity architecture: interactive controls for people, scoped tokens for automation, and explicit configuration for session refresh.

Mixed-Version Secret Handling Makes Upgrade Order Part of Security

Encryption at rest protects credentials stored in gateway configuration, but introducing encryption creates a compatibility boundary. The component writing an encrypted value and the component reading it must understand the same format.

In 3.10.3, the Control Plane encrypts the semantic AI Cache embedding keys for OpenAI and Azure OpenAI. API7 Gateway upgrades the Control Plane before the Data Plane. During that interval, a 3.10.3 Control Plane can encrypt these fields while a 3.10.2 Data Plane cannot decrypt them. An affected ai-cache configuration may therefore fail until the Data Plane is upgraded.

The mitigation is operationally simple and important: upgrade the Data Plane promptly after the Control Plane and avoid editing affected AI Cache configurations until both sides run 3.10.3. This reduces the time in which a configuration write can cross the compatibility boundary.

The release also fixes secret resolution behavior beyond encryption. Consumer authentication now fails closed when a referenced secret cannot be resolved, rather than indexing the unresolved literal value as a usable credential. Updates and deletions under /secrets invalidate the secret cache so old values are re-resolved instead of remaining active until an unrelated expiry or restart.

Those fixes reinforce the same principle: an unresolved or changed secret should produce an explicit safe outcome, not silently preserve ambiguous authentication behavior.

Safer Rollouts Depend on Valid and Converged Configuration

Configuration is only useful when it is accepted consistently and reaches every gateway instance. Version 3.10.3 improves both sides of that equation.

The Control Plane now rejects several invalid core-resource and runtime-service plugin configurations that the Data Plane previously could discard silently. Batch SSL validation uses the correct schema, IPv6 upstream hosts are accepted, and compatibility reports retain the complete sorted result instead of an unordered 200-item subset. For large estates, this gives teams a more reliable picture of whether a target configuration is actually compatible.

The Data Plane also responds more safely when configuration history moves backward. If a restored Control Plane database reports a lower revision than a gateway has already seen, the gateway now forces a full resynchronization rather than continuing to serve stale state until a worker restart. Separately, a grace period prevents healthy, recently heartbeating instances from being marked OutOfSync during the short interval after an ordinary revision change.

SQL Server deployments have one additional first-start consideration. The Control Plane enables READ_COMMITTED_SNAPSHOT to prevent configuration reads from blocking behind writes. On an existing database where the setting is disabled, the first 3.10.3 startup applies the change with ROLLBACK IMMEDIATE, which can disconnect active transactions and sessions once before connection pools reconnect. Teams using SQL Server should schedule and observe that event as part of the Control Plane upgrade.

What to Review Before Upgrading to 3.10.3

Use the rolling upgrade guide together with the version-specific release notes. A practical review sequence is:

1. Capacity: calculate memory headroom using the new shared-dictionary defaults; adjust Kubernetes requests, limits, and node capacity before replacing pods.
2. Proxy topology: list trusted load balancer and reverse-proxy addresses, configure apisix.trusted_addresses, and test forwarded-host, scheme, port, and redirect behavior.
3. Administrative access: identify built-in users, Basic Auth automation, token owners, lockout monitoring, password policy, 2FA recovery, and break-glass procedures.
4. Session policy: decide whether OIDC silent re-authentication should remain enabled and configure the interval explicitly if required.
5. Secret compatibility: locate semantic AI Cache embedding credentials, minimize the mixed-version window, and freeze affected edits until Data Planes are current.
6. Database behavior: for SQL Server, prepare for the one-time snapshot-isolation change and possible connection interruption.
7. Configuration confidence: review complete compatibility reports, test invalid configurations, and confirm every gateway instance converges after a revision change.
8. Canary and observation: upgrade a representative gateway group first, send real test traffic, inspect proxy and resource metrics, and continue only after the new nodes remain stable.

flowchart TD
    review[Read 3.10.3 Upgrade Notes] --> size[Size Memory and Nodes]
    size --> trust[Map Trusted Proxy Addresses]
    trust --> access[Separate Human and Automation Access]
    access --> cp[Upgrade Control Plane]
    cp --> db{SQL Server snapshot change?}
    db -- Yes --> observe[Observe connection recovery]
    db -- No --> freeze[Freeze affected AI Cache edits]
    observe --> freeze
    freeze --> canary[Upgrade Data Plane Canary]
    canary --> verify[Verify traffic, metrics, headers, auth, and config sync]
    verify --> fleet[Continue Rolling Upgrade]

Make the Safe State the Operable State

The most valuable security default is one that operators can understand and maintain. API7 Gateway 3.10.3 moves several boundaries in that direction: forwarded context comes from trusted intermediaries, administrative login has stronger controls, automation has a token-based path, unresolved consumer-credential secrets fail closed, and invalid configuration is rejected earlier.

The release also makes operational costs visible. More shared memory requires explicit capacity. At-rest encryption requires coordinated component versions. A stronger authentication posture requires a recovery plan. A reliable compatibility report must include the entire configuration estate.

That is what safer enterprise upgrades look like in practice: not an absence of change, but a rollout in which each change has an owner, a test, and a clear failure boundary.

Read the full API7 Gateway 3.10.3 release notes, follow the rolling upgrade procedure, and validate capacity, trust, identity, secrets, and configuration convergence in a representative gateway group before updating the fleet.