Elastic API Gateway Scaling: Signals, Capacity, and Safe Scale-In

Elastic scaling works only when the signal predicts gateway saturation early enough for new capacity to become ready. CPU can be useful, but it is not a complete capacity model: connection count, request cost, event-loop delay, upstream latency, and downstream limits can become the real constraint first.

A safe design therefore combines a demand signal with minimum warm capacity, a tested startup budget, admission controls, and conservative scale-in. The autoscaler changes replica count; it does not create database connections, third-party quota, regional addresses, or upstream capacity.

Key Takeaways

• Define one capacity unit from load tests before choosing an autoscaling target.
• Keep enough ready replicas to survive startup delay, node loss, and normal traffic variation.
• Use a leading demand or saturation signal and treat latency and errors as guardrails, not blind instructions to add replicas.
• Configure resource requests before using CPU utilization with Kubernetes HPA.
• Delay scale-in, drain connections, and verify endpoint removal before terminating a gateway pod.
• Test missing metrics, cold starts, downstream saturation, burst traffic, and recovery.

Start With a Capacity Envelope

Measure a representative traffic mix rather than a single lightweight endpoint. Record the point at which service-level objectives begin to fail and identify the first constrained resource.

Do not publish a universal requests-per-second number. A result is valid for the tested hardware, configuration, plugin set, protocol, payload, upstream, and latency objective.

Build a Closed-Loop Scaling System

flowchart LR
    T[Client traffic] --> G[Ready gateway replicas]
    G --> U[Upstream capacity]
    G --> M[Gateway and platform metrics]
    U --> M
    M --> A[Metrics adapter and autoscaler]
    A -->|scale out or in| G
    P[Policy: bounds, rate limits, stabilization] --> A
    O[SLO and cost review] --> P

The request path is client → gateway → upstream. Metrics flow to the adapter and autoscaler; they are observations, not authorization or routing decisions. The policy owner sets bounds and reviews outcomes.

Apache APISIX 3.19.0's prometheus plugin exposes signals including current client connections, HTTP status counts, and request, upstream, and APISIX latency histograms. These metrics can support capacity analysis. They do not become Kubernetes custom metrics until a Prometheus adapter or another metrics pipeline publishes the required metric through the Kubernetes metrics APIs.

Choose Signals by Failure Mode

Use latency and error signals to decide whether to hold, shed load, or stop a rollout. If the database is saturated, multiplying gateway replicas and connections can make the incident worse.

Configure HPA as an Explicit Policy

Kubernetes autoscaling/v2 supports resource, pod, object, and external metrics plus separate scale-up and scale-down behavior. For CPU utilization, HPA calculates utilization against container CPU requests, so missing or unrealistic requests make the target misleading.

The following is a runnable-shape example that still requires cluster-specific values. It assumes a metrics adapter already publishes gateway_inflight_requests as an average per-pod custom metric; it is not a native APISIX metric name.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: apisix-gateway
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: apisix-gateway
  minReplicas: 4
  maxReplicas: 20
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 60
    - type: Pods
      pods:
        metric:
          name: gateway_inflight_requests
        target:
          type: AverageValue
          averageValue: "200"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Percent
          value: 25
          periodSeconds: 60

The values are examples, not recommendations. With multiple metrics, HPA calculates a desired replica count for each and uses the largest. Kubernetes also documents an important failure path: if a metric cannot produce a desired count and the available metrics recommend scaling down, scale-down is skipped; a valid scale-up recommendation can still proceed.

The behavior fields above are explicit. If scaleDown.stabilizationWindowSeconds is omitted, the documented default is 300 seconds; the scale-up default is zero. Pin cluster version and controller settings because operators can change cluster-wide behavior.

Make New Replicas Count Only When Ready

A process that has started is not necessarily ready to receive production traffic. Readiness should cover the conditions needed to serve safely: configuration loaded, listeners active, required local state initialized, and critical dependencies within the chosen failure policy.

Kubernetes HPA handles missing metrics and not-yet-ready pods conservatively, but that does not replace a correct readiness probe. Measure image pull, scheduling, configuration propagation, key loading, and warm-up separately. Set minimum replicas and headroom so traffic survives that complete delay.

Avoid a circular readiness rule that requires a healthy optional dependency while the intended policy is to serve degraded traffic without it. The probe must match the actual availability contract.

Scale In Without Dropping Useful Work

Scale-in removes capacity while traffic still exists. Kubernetes pod termination marks terminating endpoints not ready for regular traffic and sends a stop signal within the configured grace period. Applications still need enough time to stop accepting new work and finish or transfer eligible connections.

For gateway workloads:

1. choose a scale-down stabilization window longer than normal metric noise;
2. remove the pod from new-traffic selection through readiness and endpoint state;
3. allow HTTP keep-alive, streams, and upgraded connections to drain according to protocol policy;
4. set terminationGracePeriodSeconds from observed drain time;
5. verify what happens when the grace period expires and processes are force-killed.

A PodDisruptionBudget limits compatible eviction-based voluntary disruptions. It does not replace HPA minimums, rollout surge/unavailable settings, topology spread, or a connection-drain test.

Keep Admission Control in Front of Elasticity

Autoscaling is delayed feedback. A sudden burst can arrive before the next metric sample, replica calculation, scheduling decision, and readiness transition. Use bounded request rate, concurrency, body size, timeouts, and queueing to protect the warm fleet and its dependencies during that interval.

Scale-out must also respect external limits:

• node and IP address capacity;
• load-balancer target registration;
• control-plane and configuration propagation;
• certificate and secret distribution;
• upstream connection pools and database limits;
• third-party quotas and cost ceilings.

Verification Checklist

• Does the capacity test match the production protocol, plugin chain, payload, and upstream latency?
• Are CPU requests set before CPU utilization drives HPA?
• Can the custom metric be traced from source query to metrics API and HPA status?
• Does a missing metric prevent unsafe scale-in as expected?
• Can the warm fleet survive a burst until new replicas become ready?
• Does scale-out stop or shed load when the downstream is saturated?
• Are terminating pods removed from new traffic and given enough drain time?
• Do long-lived connections, retries, and forced termination behave as designed?
• Are minimum replicas, topology, and disruption settings tested during node maintenance?
• Can operators reconstruct which metric and policy caused each scaling event?

FAQ

Is CPU enough for an API gateway HPA?

Sometimes for a stable, compute-bound workload. Add a signal for the first real saturation mode when connections, request cost, queueing, or downstream capacity diverge from CPU.

Should latency trigger scale-out directly?

Only after separating gateway saturation from upstream delay. Otherwise the autoscaler can add gateway pressure to an already failing dependency.

Can an API gateway scale to zero?

That is rarely appropriate for an internet-facing synchronous gateway because the first request needs a listener before it can trigger useful work. Keep tested warm capacity unless another front door safely queues or redirects traffic and the platform explicitly supports the chosen scale-to-zero path.

Next Steps

Combine elasticity with concurrency budgets and load shedding, safe timeout and retry policy, and evidence-based gateway optimization.