Native Histograms for API Gateway SLOs: A Safe Migration Guide

API gateway latency rarely fits neat bucket boundaries. Most requests may complete in milliseconds while streaming AI calls, cross-region traffic, or failing upstreams extend the distribution into seconds. Classic Prometheus histograms force metric authors to choose buckets before seeing the full production range.

Kubernetes 1.37 made that tradeoff newly relevant by enabling native histogram exposition in its shared metrics subsystem by default. The Kubernetes announcement describes higher-resolution latency data, a bounded bucket model, and dual exposition for migration. Prometheus has separately marked native histograms stable since version 3.8, although scraping them still requires explicit configuration.

For API gateway teams, native histograms can improve percentile analysis and reduce the time-series multiplier created by classic buckets. They do not automatically fix bad labels, vague SLOs, or incompatible dashboards. The migration must cover the complete path from instrumentation through storage, remote write, PromQL, alerts, and long-range reports.

Key Takeaways

• A classic histogram stores each configured le bucket as a separate time series; a native histogram stores a structured histogram sample.
• Native histograms offer dynamic exponential buckets and bounded relative error across a wide latency range.
• Prometheus 3.8 made the feature stable, but scrape_native_histograms: true is still required; remote write has its own setting.
• Classic and native histogram samples should be collected in parallel during migration because queries and historical ranges behave differently.
• Cardinality control remains essential: native histograms reduce bucket-series overhead, not the number of route, service, tenant, or status label combinations.
• Do not claim a gateway exports native histograms until its instrumentation and exposition path have been verified.

Why Classic Buckets Struggle with Gateway Latency

A classic Prometheus histogram might define buckets such as 5 ms, 10 ms, 25 ms, 50 ms, 100 ms, 250 ms, 500 ms, 1 second, and 5 seconds. Each observation increments every cumulative bucket above its value. The resulting _bucket series include an le label for the upper bound.

This works well when boundaries reflect the decision a team needs to make. A 300 ms user-facing objective deserves a bucket at or near 300 ms. Problems appear when one metric must cover very different traffic:

• cached reads and local health checks;
• ordinary REST requests;
• uploads and downloads;
• cross-region upstream calls;
• server-sent events or model streams;
• timeout and retry tails.

Adding more classic buckets increases the number of time series for every label combination. Keeping fewer buckets reduces cost but makes quantile interpolation coarser. Static boundaries also age poorly when the workload changes.

Prometheus native histograms promote the histogram to a first-class sample type. One sample contains count, sum, and dynamically populated bucket spans. Standard exponential schemas provide relative rather than fixed absolute resolution, making one configuration useful from milliseconds to seconds.

What Changed in Kubernetes 1.37

Kubernetes implements native histograms in k8s.io/component-base/metrics, so control-plane and node components can expose both classic and native representations. The 1.37 feature is Beta and enabled by default in Kubernetes, but that does not mean Prometheus will ingest the native representation automatically.

Prometheus still needs a scrape job configured for native histograms:

scrape_configs:
  - job_name: kubernetes-apiservers
    scrape_native_histograms: true
    always_scrape_classic_histograms: true

The second setting is a migration guard. It retains classic series while teams update dashboards, recording rules, and alerts. Prometheus documents the current behavior in its native histogram specification. From Prometheus 3.9 onward, the old feature flag is a no-op; explicit scrape configuration is the durable control.

Kubernetes metrics are a timely proof point, not evidence that every application metric has changed. Apache APISIX users should verify the actual exposition returned by their deployed Prometheus plugin and the capabilities of the Prometheus client library in that release. If the gateway emits only classic histograms, the collector cannot invent true native resolution after the fact without a defined conversion strategy.

Map the Full Telemetry Path

Native histogram readiness is an end-to-end property:

flowchart LR
    G[Gateway instrumentation] --> E[Metrics exposition]
    E --> P[Prometheus scrape]
    P --> R[Rules and queries]
    P --> W[Remote write]
    R --> D[Dashboards and alerts]
    W --> L[Long-term store]

Check each component:

1. Instrumentation: Does the metric library create a native histogram, a classic histogram, or both?
2. Exposition: Does the endpoint negotiate a format that carries native samples?
3. Scraping: Is scrape_native_histograms enabled for the intended jobs?
4. Remote write: Is send_native_histograms enabled, and does the receiver preserve the sample type?
5. Queries: Do PromQL expressions use the native metric name rather than _bucket series?
6. Visualization: Does the dashboard datasource and panel handle histogram samples correctly?
7. Retention: Can the long-term store query data consistently across the migration date?

A green scrape target proves only that Prometheus fetched something. It does not prove that native samples survived remote write or that an SLO report uses them.

Update PromQL Deliberately

A classic percentile query aggregates bucket series while preserving le:

histogram_quantile(
  0.99,
  sum by (le, service) (rate(http_request_duration_seconds_bucket[5m]))
)

The native equivalent operates on the histogram metric itself:

histogram_quantile(
  0.99,
  sum by (service) (rate(http_request_duration_seconds[5m]))
)

The shorter expression is not a license for a blind search-and-replace. Prometheus warns that classic and native histograms cannot simply be aggregated together, and range queries that cross a format transition can be incomplete. Functions and binary operations may annotate or omit incompatible samples.

Build parallel recording rules with distinct names. Compare p50, p95, p99, request counts, and known threshold fractions for several weeks. Unit-test the rules with promtool, and verify that alert state changes match the old rules under both ordinary and pathological traffic.

Percentiles are also not sufficient for an SLO. If the objective is “99.9% of requests below 300 ms,” calculate the fraction under the threshold and define which requests belong in the denominator. Keep exact classic buckets around when a contractual threshold benefits from a precise boundary that an exponential schema cannot represent exactly.

Native Histograms Do Not Fix Cardinality

Native histograms reduce the series multiplication caused by le buckets, but every other label still multiplies the metric. A gateway metric labeled by route, upstream, service, method, status class, region, and tenant can still explode.

Use labels for bounded operational dimensions. Put request IDs and raw paths in traces or logs. Normalize templated routes rather than exporting /users/12345. Treat tenant labels cautiously; for a large or unbounded tenant population, aggregate at a service tier and use logs for per-tenant investigation.

This division of labor matches the broader API observability model:

• metrics show fleet-level rates, errors, latency, and saturation;
• traces explain a sampled request's path through gateway and upstream;
• logs preserve discrete policy decisions and error context.

Native histograms make one metrics dimension more efficient. They do not turn metrics into an audit database.

Run a Dual-Exposition Migration

A conservative migration follows five stages.

1. Inventory

List histogram metrics used by dashboards, alerts, recording rules, SLO reports, autoscalers, and capacity models. Record bucket boundaries and label counts. Separate gateway processing latency from upstream latency and total request latency.

2. Lab Validation

Enable native scraping for a non-production target. Confirm negotiated exposition, stored sample types, query results, remote-write behavior, and memory or storage impact. Send a distribution that includes near-zero values, normal traffic, and outliers.

3. Parallel Collection

Collect classic and native forms long enough to cover the longest dashboard and alert lookback. Keep separate rule names and compare results. Watch ingestion CPU, storage, query latency, and scrape payload size instead of assuming every dimension improves.

4. Consumer Cutover

Move dashboards first, then non-paging alerts, recording rules, SLO reports, and finally paging alerts. Document the format boundary so long-range reports do not silently compare incomplete periods.

5. Classic Retirement

Stop classic ingestion only after every consumer is migrated and rollback has been tested. Some metrics with important exact bucket boundaries may remain classic by design.

API Gateway SLO Checklist

Before using native histograms for a gateway SLO, confirm:

• the metric measures the correct latency boundary;
• success, errors, cancellations, and streams are classified explicitly;
• route and service labels are bounded;
• native samples reach the long-term store;
• rules do not aggregate incompatible schemas;
• dashboards avoid ranges that cross an unhandled migration boundary;
• threshold-based objectives retain enough precision;
• classic and native alerts were compared under replayed or canary traffic;
• rollback preserves the old rule names and data.

Conclusion

Kubernetes 1.37 moves native histograms from an observability experiment toward an ecosystem default, while Prometheus 3.8 provides a stable storage and query foundation. API gateway teams should use that momentum to revisit latency telemetry—but not to rush a format toggle.

Start with the SLO decision, verify the gateway's real instrumentation, map every telemetry hop, and run dual exposition through the longest operational lookback. When the path is ready, native histograms can provide sharper tail-latency visibility with fewer bucket series. Explore API7 Gateway and its metrics guidance to build the surrounding metrics, logs, and operational controls.