API7 Gateway 3.10.2 Released: Govern AI Traffic End to End

Key Takeaways

API7 Gateway 3.10.2 connects AI efficiency, safety, observability, traffic control, and developer access in one release rather than treating them as separate platform concerns.
The new AI Cache plugin can serve identical LLM requests from Redis, while the new AI Lakera Guard plugin can inspect prompts and responses for prompt injection and other unsafe content.
Richer llm_summary fields, additive log formatting, and request logs embedded in Debug Session traces give operators more context for model behavior and incident response.
Rate-limit fixes close variable-validation and concurrency gaps, so configured quotas remain meaningful across routes and under Redis-backed load.
Developer Portal improvements strengthen email verification, organization administration, documentation delivery, and server-side SSO enforcement.
Upgrading requires coordinated Control Plane and Data Plane versions, explicit review of HMAC body limits, and migration of Developer Portal signup consent settings.

AI applications rarely stay as a single model call. Once they reach production, the request path expands: authenticate a user, enforce a quota, screen a prompt, select a model, handle a streaming response, record token and tool activity, investigate failures, and expose the resulting capability to developers.

If each stage is owned by a different application library or service, the platform becomes difficult to govern. Cache policy varies between teams. Prompt protection is applied inconsistently. Provider errors disappear behind adapters. Quotas behave differently under concurrency. Operations teams see HTTP status and latency but not enough LLM context to explain what happened. Developers then consume those APIs through a portal that has its own identity and documentation lifecycle.

API7 Gateway 3.10.2, released on June 29, 2026, addresses this as one end-to-end traffic problem. The release adds gateway-level AI caching and prompt protection, expands request-level observability, makes rate limiting more reliable, strengthens secret handling and identity controls, and gives platform teams more ways to operate the Developer Portal.

This is not only an AI feature release. It is a step toward governing the complete path from prompt to provider, from incident to trace, and from API product to developer.

flowchart LR
    developers[Developers and AI Applications] --> portal[Developer Portal]
    portal --> gateway[API7 Gateway 3.10.2]
    gateway --> identity[Identity and Session Policy]
    gateway --> limits[Quotas and Rate Limits]
    gateway --> guard[Prompt and Response Protection]
    gateway --> cache[AI Cache]
    gateway --> models[LLM Providers]
    gateway --> telemetry[Logs, Metrics, and Debug Sessions]
    telemetry --> operations[Platform Operations]

Production AI Traffic Is a Lifecycle, Not a Proxy Call

An AI gateway can provide a stable endpoint in front of multiple model providers, but connectivity alone does not make AI traffic production-ready. Platform teams also need to answer operational questions across the request lifecycle:

Is this application or developer allowed to use the model?
Which quota applies across all routes for the same consumer?
Is the prompt safe to send, and should the response be checked before delivery?
Can an identical request be served without paying for another model invocation?
Did the request stream, call tools, use reasoning tokens, or consume cached provider tokens?
If a provider returns HTTP 429 or 5xx, will the client receive enough detail to react correctly?
Can an operator move from an incident to the relevant request logs without searching several disconnected systems?

The gateway is a useful control point because it sees the authenticated identity, request body, policy decisions, selected upstream, response, and final status. Controls applied there can be shared across applications instead of being rebuilt for every service.

API7 Gateway 3.10.2 develops that control point in three connected directions. It adds preventive controls before a model call, diagnostic context during and after the call, and governance around the developers and applications consuming the API. The value comes from treating those capabilities as one operating model.

Control AI Cost and Risk Before the Upstream Call

Repeated LLM requests create an obvious efficiency problem. Support assistants, classification services, summarization pipelines, and internal tools can send the same normalized request many times. Calling the upstream model for every identical request adds latency and token cost without changing the answer.

API7 Gateway 3.10.2 introduces the AI Cache plugin. It creates an exact-match cache key from the normalized request body and the selected AI instance configuration, then stores successful responses in Redis. On a hit, the gateway serves the cached response without calling the model and returns X-AI-Cache-Status: HIT together with X-AI-Cache-Age. A miss goes upstream and can populate the cache. Streaming requests are bypassed in 3.10.2 and return X-AI-Cache-Status: BYPASS.

Centralizing this behavior makes cache policy visible to the platform team. Operators can select routes where exact reuse is appropriate, set the time to live, control maximum cached response size, isolate or share cache keys deliberately, and use response headers to distinguish hits, misses, and bypasses.

Caching is not automatically safe for every prompt. User-specific context, tenant data, model-version changes, and freshness requirements all influence whether two requests should share a response. The gateway does not eliminate that decision. It gives the organization one place to make and audit it.

Cost control also needs a safety boundary. API7 Gateway 3.10.2 adds the AI Lakera Guard plugin to screen AI traffic through Lakera Guard. Operators can inspect input, output, or both. Flagged traffic can be blocked or recorded in alert mode, supporting a staged path from observation to enforcement.

The AI Aliyun Content Moderation plugin also gains request_check_mode. Teams can check only the last user turn or all user turns in a multi-turn conversation, depending on the risk and cost profile of the application. Long content is split in linear time with UTF-8-safe handling.

flowchart LR
    request[AI Request] --> gateway[API7 Gateway 3.10.2]
    gateway --> cache[AI Cache Control]
    gateway --> safety[Input and Output Safety Control]
    gateway --> upstream[LLM Upstream]
    cache --> cacheOutcome[Hit, Miss, or Bypass]
    safety --> safetyOutcome[Allow, Alert, or Block]
    upstream --> modelOutcome[Model Response or Error]
    cacheOutcome --> governed[Governed AI Traffic]
    safetyOutcome --> governed
    modelOutcome --> governed

This diagram shows the control responsibilities at the gateway; it does not prescribe the execution order for a combined plugin configuration. Teams should validate the effective behavior of their selected plugins and policies in staging.

Together, caching and safety controls answer two different questions before traffic leaves the gateway: “Do we need another upstream call?” and “Should this content cross the boundary?” Keeping both decisions close to the traffic path reduces duplicated application logic and creates a more consistent AI governance model.

Make Model Behavior Explainable During Incidents

Traditional API monitoring begins with request rate, error rate, and latency. Those signals remain important for AI traffic, but they do not explain token usage, streaming experience, tool activity, or provider-specific failures.

API7 Gateway 3.10.2 expands the structured llm_summary emitted to logger plugins when summary logging is enabled. The new fields include whether a request streamed, how many tools were available, whether tool calls occurred, an end-user identifier, tokens read from or written to a provider cache, and reasoning tokens. These fields help teams distinguish requests that look similar at the HTTP layer but have different cost and execution paths.

For example, two requests may both return HTTP 200 in eight seconds. One may be a short non-streaming response; the other may begin streaming quickly, invoke a tool, and consume many reasoning tokens. An operator investigating cost or latency needs those differences in the telemetry.

The Data Plane now supports log_format_extra, which adds fields to the default rich log format instead of forcing teams to replace it. This matters operationally because a custom log_format can accidentally remove useful default context. The new $upstream_unresolved_host variable also records the configured upstream host before DNS resolution, helping teams connect a resolved backend address to the model or service name they intended to call.

Debug Sessions also become more self-contained. Each request's logs can be captured as OpenTelemetry span events on the root span, so an operator inspecting a Debug Session can see per-request logs in the trace without requiring a separate external log collector for that investigation.

Several reliability fixes make this telemetry more actionable. AI Proxy now preserves the response body and content type when an upstream model returns an error such as HTTP 429 or 5xx, including when AI Proxy Multi fallback is involved. Streaming Anthropic requests no longer wait until timeout when an upstream omits the final completion chunk. A malformed tool call no longer causes the rest of a response to disappear, and a failure while building an AI Proxy Multi working pool no longer clears pool state destructively.

flowchart TD
    request[LLM Request] --> summary[llm_summary Context]
    request --> trace[Debug Session Root Span]
    request --> upstream[LLM Upstream]
    upstream --> outcome[Response or Provider Error]
    summary --> logs[Additive Gateway Logs]
    trace --> events[Per-request Log Events]
    outcome --> details[Preserved Error Body and Content Type]
    logs --> triage[Incident Triage]
    events --> triage
    details --> triage

This shortens the path from symptom to explanation. Instead of correlating an alert, a generic gateway log, a provider response, and application-side LLM metadata manually, operators can collect more of the context at the gateway where the routing decision happened.

Keep Quotas and Identity Correct Under Real Load

AI traffic can be expensive and bursty, so a configured limit must remain correct when requests are concurrent and when one consumer uses multiple routes.

API7 Gateway 3.10.2 fixes three important rate-limit behaviors. First, variable-derived count and time_window values in Limit Count are now validated. Invalid, non-positive, or unsafe values are rejected instead of being ignored, which previously could disable the intended limit.

Second, Redis-backed sliding-window counting uses an atomic script. Without an atomic check-and-increment, concurrent requests can observe the same previous count and collectively exceed the quota. The fix makes the result match the configured policy under load.

Third, Limit Request now keys a shared-resource limit by its parent resource. A limit attached to a Consumer is therefore enforced across that Consumer's routes instead of creating an independent bucket for every route.

These are implementation details with direct governance impact. Platform teams use limits to protect model capacity, allocate tenant budgets, and reduce noisy-neighbor behavior. If the counter scope or concurrency behavior differs from the policy, the quota is only nominal.

Identity boundaries receive similar attention. The Attach Consumer Label plugin now strips configured identity headers from the client request even when the matched Consumer has no label value, preventing the client from supplying a value that appears to come from the gateway. Developer Portal email-domain SSO policies are enforced on the server rather than only in the UI, so direct calls to password, magic-link, or reset endpoints cannot bypass a domain that is required to use SSO.

OpenID Connect can now store sessions in Redis instead of putting session state in the cookie. This gives teams a server-side session option for distributed deployments while leaving cookie storage as the default. The release also supports custom authentication headers for Elasticsearch Logger, giving observability pipelines an alternative to basic authentication.

Because these capabilities introduce more credential-bearing fields, 3.10.2 expands encryption at rest for Redis passwords used by rate-limiting plugins, AI Cache, AI Lakera Guard API keys, OpenID Connect Redis session passwords, and Elasticsearch custom authentication headers. The security improvement comes with an important upgrade-order requirement, covered below.

Connect Runtime Governance to the Developer Portal

The traffic path is only one half of an API program. Developers also need a controlled way to discover documentation, authenticate, join organizations, and consume API products.

API7 Gateway 3.10.2 expands the Developer Portal in several practical ways:

Operators can disable the API Hub completely when a deployment should not expose a public catalog. The navigation, pages, and sitemap entries are removed together.
Email verification can be required before signup or sign-in completes.
Platform administrators can take ownership of or delete organizations, improving recovery and lifecycle management when the original owner is unavailable.
The portal can use a custom PostgreSQL schema instead of public, which helps organizations isolate portal data within an existing database service.
Documentation is available through Markdown and LLM-friendly text endpoints for AI tooling, while selected pages can be excluded from those endpoints without being removed from the human-facing docs site.

The release also fixes security and usability gaps around developer identity. Enabling or disabling two-factor authentication now verifies the account password, backup codes display correctly, and an incorrect TOTP code produces an error instead of navigating to the home page.

These changes connect API governance with developer experience. Platform teams can choose whether the catalog is public, require verified identities, preserve organization boundaries, and decide which documentation can be consumed by AI tools. Developers still receive a self-service experience, but the policy is enforced by the platform rather than assumed from the interface.

Improve Reliability Below the Application Layer

Several 3.10.2 Data Plane changes address problems that applications should not have to solve themselves.

After a transient DNS or service-discovery failure, a domain-name upstream now recovers when resolution succeeds instead of remaining stuck on HTTP 503. Consistent-hash routing keeps the ring stable when a node becomes unhealthy, remapping only keys associated with the failed node rather than moving traffic for healthy nodes too. The Prometheus library update removes duplicate metric series that could cause an entire scrape to be rejected.

The new max_post_args_readable_size setting bounds how much JSON or multipart request content the gateway reads when evaluating post_arg.* route conditions. Its default is 64 MiB, and it can be set to 0 to disable the limit. This gives operators a direct control over memory exposure from large request bodies used in route matching.

Deployment options also broaden. API7 Gateway 3.10.2 adds an RPM installation path for Data Plane instances, including an offline script that provisions the gateway-group client certificate, writes configuration, and joins the instance to the Control Plane. This is useful for air-gapped or host-based environments that do not standardize on Docker or Kubernetes.

These updates share the same goal as the AI and portal changes: keep infrastructure behavior predictable so application teams can depend on the gateway contract.

What to Review Before Upgrading to 3.10.2

API7 Gateway 3.10.2 adds important controls, but three upgrade notes require deliberate preparation.

Coordinate Control Plane and Data Plane Secret Handling

The 3.10.2 Control Plane encrypts additional plugin fields at rest. An older 3.10.1 Data Plane cannot decrypt fields newly encrypted by the 3.10.2 Control Plane. During that mixed-version window, affected rate-limit, AI Cache, AI Lakera Guard, Elasticsearch Logger, or OpenID Connect configurations can fail.

Upgrade the Data Plane promptly after the Control Plane and avoid editing affected plugins until both sides are on 3.10.2. Inventorying those fields before rollout reduces the chance of an unexpected configuration change during the compatibility window.

Recheck the HMAC Request Body Limit

When hmac-auth validates the request body, its default max_req_body_size is 64 MiB in 3.10.2. This raises the 3.10.1 default of 512 KiB and aligns the behavior with Apache APISIX. Bodies over the limit now return the accurate HTTP 413 status instead of a misleading HTTP 401.

If an environment relied on the lower limit as a security boundary, configure it explicitly. If legitimate signed requests can approach 64 MiB, test gateway and upstream memory behavior rather than treating the higher default as a capacity recommendation.

Signup consent now uses signUpConsentLabel, an HTML snippet rendered beside the consent checkbox. The previous tosURL and beforeSignUpButtonHtml settings have been removed. Move existing consent content before upgrading; otherwise, the consent text will no longer appear, and the consent check is enforced only when a label is configured.

Beyond those required items, test a representative set of production behavior:

Cache hits, misses, bypasses, tenant boundaries, and Redis failure behavior.
Input and output guard policies in both alert and blocking modes.
Streaming completions, malformed tool calls, fallback, provider 429 responses, and 5xx errors.
Dynamic and Consumer-level rate limits under concurrent traffic.
Debug Session traces with embedded logs and the required privacy controls for end_user_id and other LLM context.
Developer Portal email verification, SSO-domain enforcement, 2FA recovery, and signup consent.
DNS recovery, consistent-hash behavior, Prometheus scraping, and large post_arg.* route matching.

Govern the Complete Path, Not Isolated Features

API7 Gateway 3.10.2 matters because production AI and API management are converging. The same platform must control model cost, prompt safety, identity, quotas, telemetry, incident response, developer access, and infrastructure reliability.

The release brings those responsibilities closer together. AI Cache reduces repeated model work where exact reuse is appropriate. AI Lakera Guard and content moderation create a policy boundary around prompts and responses. Richer LLM logs and Debug Session events make failures easier to explain. Atomic and correctly scoped limits keep quotas enforceable. Developer Portal controls extend governance to the people and tools consuming the APIs.

Most importantly, these capabilities do not need to become separate application projects. They can be operated at the gateway layer as one end-to-end traffic system.

Read the full API7 Gateway 3.10.2 release notes, review the API7 AI Gateway documentation, and validate the upgrade notes against your Control Plane, Data Plane, Redis, identity, and Developer Portal configuration before rollout.