Kubernetes Memory QoS for API Gateway Reliability

An API gateway can look healthy by CPU and request-rate metrics while memory pressure is already stretching its tail latency. TLS state, connection pools, request buffers, caches, telemetry queues, and plugin data all compete inside the gateway process. When the node starts reclaiming memory aggressively, the first visible symptom may be slow requests rather than an immediate out-of-memory kill.

Kubernetes 1.37 makes this problem more actionable. Memory QoS graduated to Beta and its feature gate is enabled by default on kubelets. The important detail is easy to miss: the default configuration remains behaviorally inert. Kubernetes does not write memory.high, memory.min, or memory.low until an operator explicitly enables throttling or tiered reservation on Linux nodes using cgroup v2.

For gateway operators, Memory QoS is not a switch that automatically makes latency predictable. It is a node-level mechanism that must be combined with realistic requests and limits, capacity isolation, gateway telemetry, and a staged rollout.

Key Takeaways

• Kubernetes 1.37 enables the MemoryQoS feature gate by default, but throttling and reservation still require explicit kubelet configuration.
• memory.high can slow allocation before a container reaches its hard limit; it does not replace a correctly sized limit.
• Tiered reservation is node-wide, so a gateway cannot opt in independently of neighboring pods.
• Gateway p99 latency, Linux pressure stall information, cgroup events, and restart data should be reviewed together.
• The safest rollout uses a dedicated node pool, representative traffic, and a reversible canary.

Why Gateways Are Sensitive to Memory Pressure

API gateways are long-running, connection-heavy services. Their memory profile is rarely a flat function of requests per second.

Traffic shape changes allocation behavior:

• large request or response bodies increase buffering;
• slow clients keep buffers and connections alive longer;
• TLS handshakes and certificate chains add per-connection work;
• retries and slow upstreams increase concurrent in-flight requests;
• high-cardinality metrics and logs expand queues and labels;
• plugins may maintain caches, dictionaries, or per-route state.

A cache can improve latency while also creating memory risk if keys, TTLs, and entry sizes are not bounded. The lessons in API gateway cache memory efficiency apply beyond response caching: cardinality and object lifetime matter as much as the nominal cache capacity.

Under node pressure, page reclaim and throttled allocation can increase response time before Kubernetes reports a restart. That makes a pure “container OOM count” alert too late for a latency-sensitive edge component.

What Kubernetes 1.37 Actually Changes

Memory QoS uses the cgroup v2 memory controller to give the Linux kernel more information about protection and throttling.

Kubernetes 1.37 introduces two explicit operator choices:

1. Set memoryThrottlingFactor to enable memory.high for Burstable and BestEffort containers.
2. Set memoryReservationPolicy: TieredReservation to configure memory.min and memory.low protections by Kubernetes QoS class.

The upstream post explains that memoryThrottlingFactor now defaults to null. This avoids silently changing workload behavior when a cluster upgrades. If a previous kubelet configuration explicitly set a factor, that setting is preserved; clusters relying on the old implicit behavior should audit their rendered kubelet configuration instead of assuming.

memory.high: A Throttle, Not a Hard Wall

When a cgroup exceeds memory.high, the kernel applies reclaim pressure and throttles further allocation. The process can keep running, but requests may slow while memory is reclaimed. The hard container limit still determines the eventual OOM boundary.

This creates a useful buffer between normal operation and an abrupt kill, but only if the buffer is large enough for the workload. A value that is too aggressive can turn transient bursts into sustained latency.

The Linux kernel's cgroup v2 memory controller documentation is the authoritative reference for memory.current, memory.events, memory.high, memory.low, and memory.min. Kubernetes translates pod resource settings and kubelet policy into those primitives; it does not change their underlying semantics.

memory.min and memory.low: Protection Under Contention

Tiered reservation uses protection rather than throttling. Guaranteed pods receive stronger protection through memory.min; Burstable pods receive best-effort protection through memory.low.

That makes the Kubernetes QoS class operationally important. Under container-level resource configuration, a Pod is Guaranteed only when every container has non-zero CPU and memory requests equal to its limits; equal memory values alone are insufficient. With Pod-level resources, a Pod can instead be Guaranteed when it defines both Pod-level CPU and memory requests and limits, with each request equal to its corresponding limit. If neither Guaranteed path is met and at least one CPU or memory request or limit is set, the Pod is Burstable. The correct choice depends on workload and node economics, but it should be deliberate.

Protection is not free capacity. Reserving memory for one workload leaves less reclaimable memory for its neighbors. The upstream documentation also notes that page cache charged to the cgroup is covered, so a workload reading large files can protect memory the node might otherwise reclaim.

Start With Requests and Limits

Memory QoS cannot correct arbitrary resource settings. Begin with a representative load test and a production profile.

Measure memory at several operating points:

• steady request rate with normal payloads;
• expected traffic bursts and connection concurrency;
• slow upstreams that increase in-flight requests;
• largest accepted request and response bodies;
• configuration reloads and plugin changes;
• telemetry-backend interruption that grows local queues.

Set the request from sustained working-set evidence, not idle memory. Set the limit high enough for valid bursts while remaining below a level that threatens the node. Leave headroom for the operating system, kubelet, runtime, and daemon workloads.

Do not raise the limit until OOMs disappear without checking payload and concurrency controls. An unbounded body, cache key space, or telemetry queue can consume any larger limit eventually.

Gateway-level safeguards should remain in place:

• cap request-body sizes where the API contract allows it;
• bound caches by entries or memory and use finite TTLs;
• control downstream concurrency and upstream retries;
• reject overload before queues become memory storage;
• sample or aggregate high-cardinality telemetry.

Apache APISIX provides traffic and upstream controls at the gateway layer, while Kubernetes governs the process and node resources. Neither layer replaces the other.

Design a Safe Memory QoS Rollout

Memory QoS configuration is applied at the kubelet, not per Deployment. The 1.37 tiered reservation policy applies to every pod on a node, and individual pods cannot opt out. This makes node-pool design the first safety boundary.

Use a Dedicated Canary Node Pool

Create a small node pool with cgroup v2 and the candidate kubelet configuration. Place a limited number of gateway replicas there using normal scheduling controls. Keep a control group on nodes without the new behavior.

A throttling-only configuration can look like:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
memoryThrottlingFactor: 0.9

Tiered reservation adds:

memoryReservationPolicy: TieredReservation

These values are examples from the Kubernetes feature documentation, not universal recommendations. Managed Kubernetes services may expose kubelet settings differently or may not expose them at all. Verify support with the cluster provider before planning a rollout.

Test Contention, Not Only Peak Traffic

A gateway load test on an otherwise idle node does not exercise Memory QoS. Add controlled neighboring memory pressure and observe how the gateway behaves as the node approaches contention.

The test should answer:

• Does p99 latency rise before memory.high events?
• Does the gateway remain ready while allocation is throttled?
• Do upstream timeouts cause retries that amplify pressure?
• Does protection for one QoS class starve important node services?
• Does a configuration rollback remove stale cgroup values after reconciliation?

Keep health checks lightweight. If a readiness check allocates significant memory or calls a slow upstream, memory pressure can make the probe fail and remove capacity precisely when the fleet is stressed. The guidance in API gateway health-check best practices helps separate process readiness from upstream health.

Observe the Right Signals

No single metric explains a memory-related latency event. Correlate four layers.

Gateway Signals

Track request rate, active connections, body-size distribution, upstream latency, gateway-generated errors, retries, and p50/p95/p99 latency. Break down results by route or workload class without creating unbounded labels.

Container and cgroup Signals

Watch working set and resident memory alongside memory.current. Use memory.events to identify high, max, oom, and oom_kill events. A rising high counter paired with worse tail latency is different from a steady counter paired with stable service.

Node Pressure

Linux pressure stall information reports time that tasks are delayed by resource contention. Memory PSI can reveal system-wide pressure that an individual container metric misses.

Kubernetes Lifecycle

Correlate pod restarts, eviction events, readiness changes, rescheduling, and node conditions. A replacement pod can hide an OOM in aggregate gateway availability while still disrupting long-lived connections.

Create alerts around user impact and mechanism together. For example, page on sustained p99 degradation plus memory-pressure evidence, while treating an isolated memory.high event with stable latency as a warning for investigation.

Avoid Common Misconfigurations

Assuming Beta Means Fully Automatic

The feature gate is on by default; the behaviors are not. Audit the effective kubelet configuration and cgroup files before concluding that Memory QoS is active.

Treating Throttling as Extra Capacity

memory.high can delay allocation, but it cannot make an undersized pod safe. If normal traffic regularly crosses the threshold, adjust workload controls or capacity rather than accepting permanent throttling.

Mixing Incompatible Workloads on One Node

Tiered reservation is node-wide. Separate latency-sensitive gateways from memory-heavy batch jobs when their reclaim priorities conflict.

Watching Average Latency

Memory reclaim often appears first in the tail. Average latency can remain calm while a small but important request segment exceeds its SLO.

Rolling Out Without a Reversal Plan

Store the previous kubelet configuration, document how the provider applies changes, and verify rollback on the canary pool. The upstream post describes how kubelet reconciliation resets stale protection in relevant cases, but operators should still test their exact upgrade path.

Production Checklist

• Confirm nodes use cgroup v2 and support the required kubelet settings.
• Measure the gateway working set under realistic payloads and concurrency.
• Bound bodies, caches, retries, queues, and telemetry cardinality.
• Choose Guaranteed or Burstable QoS intentionally.
• Test memoryThrottlingFactor on a dedicated canary node pool.
• Evaluate tiered reservation only after reviewing every workload on the node.
• Correlate gateway p99, cgroup events, memory PSI, and Kubernetes lifecycle events.
• Test upstream slowdown, telemetry interruption, and neighboring memory pressure.
• Preserve a control group and a documented rollback.

Kubernetes 1.37 gives platform teams a better vocabulary for memory behavior, but reliable gateways still depend on disciplined resource design. Use Memory QoS to shape what happens under contention, then use gateway controls and telemetry to prevent ordinary traffic from reaching that cliff.

For teams standardizing API traffic across Kubernetes environments, API7 Enterprise and Apache APISIX provide the gateway layer for routing, protection, and observability while Kubernetes manages workload isolation.