AISIX 1.6.0: Make Streaming SLOs Reflect the Outcome

An AI stream can send its response headers successfully and still end because the client disconnects, an upstream breaks, or the read times out. If request metrics are counted at response start, a stream that never completed can look like a successful 200.

AISIX 1.6.0, released October 12, 2026, changes that measurement point for supported streaming traffic. AISIX now records those requests in request metrics when the stream ends, under the status it ended on. The change makes a production SLO more representative of what the caller experienced; it does not change provider billing, nor does it make a stream's initial response head a promise of completion.

AISIX processes gateway traffic in your environment. This article describes AISIX gateway telemetry; AISIX Cloud has its own supported control-plane and dashboard deployment options.

Key Takeaways

• Supported streams that finish normally are recorded as 200; a mid-stream client departure is 499, an upstream break or in-stream error is 502, and a read timeout is 504.
• The affected request-counter and duration families now describe the terminal stream outcome. Time to first token remains on the dedicated TTFT metrics.
• Existing success-rate, percentile, and rate()-based alerts can change after the upgrade. Long streams can make rate signals lag by roughly one stream length.
• This is a telemetry-semantic change. It does not infer provider charges, rewrite a response already delivered to a client, or cover every endpoint outside the documented stream families.

Why Response Start Was the Wrong End of the Story

Consider an application that streams an assistant response to a browser. Headers arrive, the browser renders the first token, then the upstream connection breaks. Counting the request at response start says 200; counting it at the end preserves the observable outcome for the request metric.

flowchart LR
  C[Client starts stream] --> G[AISIX gateway]
  G --> H[Response head and tokens]
  H -->|stream completes| S[Record 200 at stream end]
  H -->|disconnect, upstream error, or timeout| F[Record terminal status at stream end]

In 1.6.0, the documented coverage includes streams on /v1/chat/completions, /v1/messages, /v1/responses, transcription and translation event streams, /v1/audio/speech, and streamed /a2a calls. An in-stream 4xx keeps that status. A caller that leaves before the response head remains a separate client-cancellation metric case rather than being retroactively represented as a completed stream.

The distinction is important: a 200 response head and a completed stream are different events. For a stream refused after a 200 head, release notes also retain 200 while recording error_class: content_filter where applicable. Treat the status, error class, and endpoint-specific semantics together rather than reading any one field as a universal completion signal.

What Changes in Prometheus

AISIX lists these affected families: aisix_requests_total, aisix_request_duration_seconds, aisix_proxy_requests_total, aisix_proxy_failed_requests_total, aisix_proxy_request_duration_seconds, aisix_llm_requests_total, and aisix_llm_request_duration_seconds. For streaming traffic, success rates can decrease and duration percentiles can increase because a request is accounted for at its terminal outcome and over its full lifetime.

The end-to-end and TTFT families—aisix_request_e2e_latency_seconds and aisix_request_ttft_seconds—now carry the stream's real status_class rather than a fixed 200. TTFT itself remains a first-token measurement; it is not a substitute for full-stream duration.

This is not a license to compare a pre-upgrade dashboard directly with a post-upgrade one. A higher error ratio can be an accurate correction to stream accounting rather than an immediate regression. Likewise, a higher duration percentile can reflect the newly measured full stream rather than slower time to first token.

Upgrade the Alert Contract, Not Just the Gateway

Before upgrading production traffic, make the change observable on purpose:

1. Inventory dashboards, recording rules, SLOs, autoscaling signals, and alerts that query the affected metric families for streaming traffic.
2. Run a completed stream, a client cancellation after delivery begins, an upstream interruption, and a read-timeout case in a non-production environment. Confirm the terminal statuses your queries display.
3. Keep first-token and full-stream questions separate: use the TTFT metrics for the first and the request-duration families for the second.
4. Revisit rate()-based automation, including any KEDA query on aisix_llm_requests_total. AISIX documents that a burst of long streams can delay the signal by about one stream length.
5. Review the rest of the 1.6.0 upgrade notes, especially renamed error_kind and error_class values, before changing parsers or alert routing.

Do not use this change to estimate provider billing. AISIX gateway metrics and usage records are operational evidence; the provider's invoice remains a separate source of truth. Similarly, do not assume every endpoint, an unlisted protocol, or a stream that has not received a response head follows the same lifecycle rule.

Measure the Result the Caller Reached

AISIX 1.6.0 gives streaming teams a more useful question for request metrics: not whether the gateway began a response, but how that stream ended. That makes success-rate and duration signals closer to the caller's experience, provided teams update their baselines and keep TTFT, full-stream duration, status, and error-class signals distinct.

Read the AISIX 1.6.0 release notes before upgrading, then validate the stream outcomes and alert behavior you actually operate.