LLM Monitoring with Prometheus: SLOs for AI Traffic

Useful LLM monitoring starts with the caller-visible request. The application needs to know whether the request finished successfully, how long it waited for the first streamed frame, how long the full response took, and whether fallback hid an unhealthy upstream.

Those questions use different units. One client request can create several upstream attempts. A stream can send 200 headers and fail later. A gateway can queue a usage event even when the remote telemetry sink never stores it. If a dashboard mixes these units, it can look precise while answering the wrong question.

This guide builds an initial Prometheus SLO dashboard with AISIX AI Gateway 1.6.0. Version pinning matters: 1.6.0 changed streamed request metrics to record the terminal status when the stream ends. The queries and upgrade notes below are not a promise about earlier deployments.

Key Takeaways

• Define whether each metric counts client requests, upstream attempts, streams, or usage events.
• Build availability from terminal request outcomes, then use deployment metrics to diagnose target attempts.
• Separate streaming from non-streaming latency and TTFT from full stream duration.
• Filter the AISIX side label; TTFT and end-to-end latency use opposite compatibility filters.
• Aggregate histogram buckets across gateway instances; never average summary quantiles into a fleet percentile.
• Monitor policy bypass, rate-limit rejection, telemetry loss, and configuration health beside model outcomes.

Start with Four Measurement Units

An SLO usually starts with the client request because that is the unit the application experienced. Attempts, policy decisions, logs, traces, and usage events explain why the request succeeded or failed.

This is narrower than a complete AI Gateway observability program. Output quality, provider billing, prompt evaluation, and business outcomes need other systems and should not be forced into one availability ratio.

Expose Metrics on a Private Path

AISIX exposes Prometheus metrics on a dedicated listener. The AISIX Metrics Reference documents this default:

observability:
  metrics:
    prometheus:
      enabled: true
      addr: 0.0.0.0:9090
      path: "/metrics"

The endpoint is unauthenticated by design. Keep it inside a private monitoring network, restrict ingress, and do not publish it directly to the internet.

AISIX registers some traffic families only after the corresponding activity occurs. An empty traffic query immediately after startup does not prove that scraping failed. Send a controlled request, scrape again, and check configuration status separately.

Define Request Success Explicitly

AISIX 1.6.0 samples request counters when a request or stream ends. For model-inference traffic, aisix_llm_requests_total uses outcome values including success, client_error, rate_limited, and upstream_error.

A starting success ratio is:

sum(rate(aisix_llm_requests_total{outcome="success"}[5m]))
  /
sum(rate(aisix_llm_requests_total[5m]))

The expression is not a universal availability definition. Decide what spends error budget:

• an upstream or gateway 5xx usually represents a service failure;
• a malformed caller request may be excluded from the platform SLO but still indicate a broken client contract;
• a policy block may be the correct outcome;
• a 429 may be intended quota enforcement or unexpected shared-capacity exhaustion.

If quota rejections are excluded, apply the same filter to numerator and denominator and keep a separate rate-limit panel. Never remove failures merely to improve the percentage.

AISIX 1.5.0 and earlier sampled most streamed model requests when response headers started, often under 200. AISIX 1.6.0 waits for stream termination and records outcomes such as client cancellation, upstream break, or timeout. Annotate the upgrade because success rates and request-rate timing can move even if workload behavior did not suddenly change.

Measure Full Streaming Latency

LLM latency has at least two user-visible milestones:

1. Downstream TTFT: time from AISIX receiving the request until the first frame reaches the caller.
2. Downstream end-to-end duration: time until the complete response or stream ends.

For a fleet-wide P90 of successful streaming end-to-end duration:

histogram_quantile(
  0.90,
  sum by (le) (
    rate(aisix_request_e2e_latency_seconds_bucket{
      side!="upstream",
      streaming="true",
      status_class="2xx"
    }[5m])
  )
)

The negative side!="upstream" selector is deliberate. It selects downstream observations from current gateways and also matches the unlabeled downstream series emitted by AISIX 1.4.0 or earlier during a mixed-version rollout.

Do not substitute a request-duration summary for a fleet-level streamed latency histogram. AISIX 1.6.0 changed aisix_request_duration_seconds, aisix_llm_request_duration_seconds, and aisix_proxy_request_duration_seconds to record whole-stream duration; in AISIX 1.5.0 and earlier, those families stopped when the response started. aisix_request_e2e_latency_seconds remains the better input for aggregating a streamed end-to-end percentile across a fleet because it is a histogram with explicit downstream observations.

Measure TTFT Without Blending Two Clocks

AISIX TTFT histograms carry side="upstream" and side="downstream":

• upstream starts when the serving attempt begins and stops at the first upstream frame;
• downstream starts when AISIX receives the request and stops when the first frame reaches the client.

The client-perceived P90 is:

histogram_quantile(
  0.90,
  sum by (le) (
    rate(aisix_request_ttft_seconds_bucket{side="downstream"}[5m])
  )
)

To compare the upstream clock while retaining AISIX 1.4.0 series during an upgrade, use:

histogram_quantile(
  0.90,
  sum by (le) (
    rate(aisix_request_ttft_seconds_bucket{side!="downstream"}[5m])
  )
)

The compatibility filters are opposite: E2E uses side!="upstream" for the downstream view, while TTFT uses side!="downstream" for the upstream view. A query with no side filter blends two intervals and produces a misleading distribution.

The difference between TTFT sides is not always pure gateway overhead. It may include request parsing, input guardrails, failed attempts and backoff, output hold-back, and protocol-bridge behavior. On bridged Responses traffic, the first downstream event may wait for content rather than the first upstream framing event.

Use Histograms Across Instances

Prometheus histograms expose bucket counters that can be summed before histogram_quantile() calculates a percentile. That is why the queries use:

sum by (le) (rate(metric_bucket[5m]))

The official Prometheus histogram guidance distinguishes histograms from summaries. Summary quantiles are calculated inside each gateway process. Averaging two instances' P90 values does not produce a fleet P90.

Bucket boundaries are part of the monitoring contract. If users regularly wait beyond the highest useful bucket, the percentile becomes coarse. If a team changes buckets, it changes the emitted series and can break recording rules that select an exact le value. Coordinate bucket changes with dashboards and alerts.

Separate Requests from Attempts

Fallback can produce one successful client request and several upstream attempts. The request counter answers “what did the caller receive?” Deployment metrics answer “what happened on each selected target?”

AISIX documents deployment families such as:

• aisix_deployment_requests_total;
• aisix_deployment_success_responses_total;
• aisix_deployment_failure_responses_total;
• aisix_deployment_state;
• successful and failed fallback counters.

A target-attempt failure ratio by model can begin with:

sum by (model) (rate(aisix_deployment_failure_responses_total[5m]))
  /
sum by (model) (rate(aisix_deployment_requests_total[5m]))

This is not caller availability. A high attempt-failure ratio can coexist with a healthy request SLO while fallback rescues requests. It remains an important leading indicator because capacity, latency, and cost now depend on the remaining targets.

The reverse also occurs: a request can fail before dispatch because of authentication, policy, body limits, or configuration. Attempt metrics will not explain that failure. Join request outcomes with access logs, policy counters, and configuration health.

Monitor Policy and Telemetry Degradation

A healthy model response does not prove that policy and evidence pipelines are healthy.

Rate Limits and Budgets

Track rejection counters by their documented scope. A rise can mean correct protection, a changed caller pattern, or a bad configuration. The alert needs the quota owner and intended threshold. Budget gauges represent the latest gateway state, not an independent billing ledger.

Guardrails

Track blocks and fail-open or bypass signals. A low block count can mean safe traffic, an unattached guardrail, a narrow matcher, or an unavailable detector. Streaming output policy can also run after the provider has performed work, so a blocked delivery does not reverse a provider charge or external side effect.

Usage Delivery

Alert on exporter failure, rejected events, and queue drops separately from request availability. A model path can remain healthy while the organization loses cost or audit records. Successful local enqueueing does not prove that a remote warehouse stored the event.

Configuration

Use configuration-reload success and timestamp metrics to detect a gateway serving an older accepted snapshot. A control-plane update is not evidence that every data plane loaded it.

Control Cardinality and Identity Claims

Every label multiplies time series. Keep bounded dimensions such as normalized endpoint, protocol, provider, model alias, status, stream mode, and side on SLO dashboards. Put request IDs, raw URLs, prompts, completions, and unbounded caller values in logs or traces instead.

AISIX 1.6.0 can add selected human-readable names beside ID labels. Treat a rename as a new series and add such labels only when the operational query justifies them. A client classification or caller-supplied header is metadata, not verified identity, unless a trusted authentication mapping establishes it.

The OpenTelemetry semantic conventions provide a useful vocabulary, but naming consistency does not make a high-cardinality value safe for Prometheus.

Build the First Dashboard and Test It

Start with four primary panels:

1. terminal request success ratio under a documented denominator policy;
2. downstream streaming E2E P90;
3. downstream TTFT P90, compared with upstream TTFT;
4. attempt failures and fallback pressure.

Add rate-limit rejections, guardrail blocks and bypasses, usage-export failures, and configuration status as supporting panels. Then inject controlled failures: revoke a test key, exceed a test quota, fail a primary target, interrupt a stream, and disable a telemetry receiver.

The request SLO and diagnostic panels should move differently and predictably. That is the practical goal described in the Google SRE monitoring guidance: alerts should lead from user impact toward a cause an operator can investigate.

Use the AISIX Metrics Reference to adapt labels and endpoint coverage to the exact deployed release. A small, versioned monitoring contract is more trustworthy than a dashboard that mistakes every available metric for reliability.