Kubernetes HPA Scale-to-Zero for API-Driven Workers

Kubernetes 1.37 makes a long-requested capability much easier to use: the HorizontalPodAutoscaler can scale a workload down to zero replicas and bring it back when an object or external metric changes. The feature is Beta and enabled by default.

The headline sounds like a universal cost optimization. It is not. The Kubernetes announcement is explicit that Services do not buffer requests when no Pods are ready. Scale-to-zero works naturally for queue consumers and batch processors; synchronous HTTP services need another design.

For API-driven background work, the reliable pattern is to keep the request boundary available, authenticate and limit callers at an API Gateway, acknowledge accepted work through an ingestion service, place it in a durable queue, and scale only the worker pool to zero.

Key Takeaways

• Kubernetes 1.37 enables HPA minReplicas: 0 by default, but only with a suitable object or external metric.
• CPU and memory cannot wake a zero-replica workload because no Pod exists to report them.
• A Kubernetes Service does not queue HTTP requests while its backends are absent.
• Keep the API Gateway and ingestion path available; scale asynchronous workers from durable queue depth or lag.
• Design idempotency, backpressure, metrics failure, cold starts, and rollback before claiming the cost saving.

What Kubernetes 1.37 Adds

Before Kubernetes 1.37, teams typically needed an add-on, an external controller, or an Alpha feature gate to scale an HPA-managed workload down to zero. The Beta implementation now accepts minReplicas: 0 when the HPA includes at least one object or external metric.

The distinction is fundamental. CPU and memory are resource metrics produced by running Pods. At zero replicas, those signals disappear. Queue depth, queue lag, or another external demand signal continues to exist when consumers are absent, so the HPA can observe demand and calculate a nonzero replica count.

Kubernetes also tracks whether the controller created the zero state. A ScaledToZero=True condition tells later reconciliation loops that the HPA should continue evaluating the wake-up metric. If an operator manually scales a Deployment to zero, the existing pause behavior remains; the HPA does not automatically wake it.

That prevents automation from silently undoing an intentional pause, but it also creates an operational check: teams need to distinguish “scaled down normally” from “manually stopped.”

Why Direct HTTP Backends Are the Wrong First Target

Imagine routing /reports directly to a Deployment with zero Pods. The first request reaches the Kubernetes Service, but there is no ready endpoint. The Service does not hold that request until the HPA notices demand, schedules a Pod, pulls an image, starts the process, and passes readiness checks.

CPU utilization cannot help, because there is no Pod to consume CPU. A gateway request counter might be exported as an external metric, but that still does not make the original request durable. Retrying at several layers can create a thundering herd or duplicate work.

This is why scale-to-zero fits asynchronous operations better than ordinary low-latency APIs. Suitable examples include document conversion, report generation, media processing, webhook fan-out, data import, and non-interactive AI batch work. The request can be accepted quickly, represented as a durable job, and processed after capacity starts.

Keep synchronous latency-sensitive services at one or more ready replicas unless a purpose-built request activation layer provides the buffering and semantics you need.

A Safer API-to-Queue Architecture

A practical design separates request admission from background execution:

flowchart LR
    C[API client] --> G[API Gateway]
    G --> I[Ingestion API]
    I --> Q[Durable queue]
    Q --> W[Worker pool 0..N]
    Q --> M[Queue depth metric]
    M --> H[Kubernetes HPA]
    H --> W
    W --> R[Result store]

The gateway and ingestion API remain available. The gateway authenticates the caller, enforces route-level limits, rejects oversized or invalid requests where configured, and records the admission decision. The ingestion service writes the job to the queue before returning an identifier. The HPA reads queue demand through the External Metrics API and starts workers.

The gateway is not the durable queue. Apache APISIX routes and governs API traffic; a message broker or job system owns persistence, delivery, and acknowledgement. Keeping that boundary clear avoids implying that an HTTP retry is equivalent to durable job delivery.

The client can poll a status endpoint, receive a webhook, or consume an event when the result is ready. Each option needs its own authorization and retention design.

Choose a Wake-Up Metric That Represents Work

Queue length is a common starting point, but it is not always sufficient. Ten tiny jobs and ten hour-long jobs create the same depth. Depending on the system, a better signal may be:

• ready jobs, excluding delayed or permanently failed items;
• oldest-job age;
• queue lag or unprocessed partitions;
• estimated work units;
• a weighted combination of depth and age.

The Kubernetes example uses an external metric exposed through a metrics adapter. A simplified HPA looks like this:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: report-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: report-worker
  minReplicas: 0
  maxReplicas: 20
  metrics:
    - type: External
      external:
        metric:
          name: report_jobs_ready
          selector:
            matchLabels:
              queue: reports
        target:
          type: Value
          value: "30"

Treat this as a structural example, not a universal threshold. Here, Value compares the global queue metric with a fixed target, so "30" represents roughly one replica per 30 ready jobs. Validate the threshold, adapter query, update frequency, and worker startup behavior for your own queue.

Kubernetes rejects minReplicas: 0 when the HPA contains only CPU or memory metrics. It also reports a failed scaling condition if the external metric cannot be retrieved. Alert on that condition; zero replicas plus an unavailable metric can otherwise look like healthy idleness.

Keep the Gateway Available and Bound Admission

Scaling workers to zero saves their reserved CPU, memory, or GPUs. Scaling the shared API Gateway to zero usually defeats the design because callers lose the stable admission point.

The gateway should have an independent availability and scaling policy. For Apache APISIX, Prometheus metrics can expose traffic and latency signals, while Kubernetes readiness and multiple replicas protect the data path. API7's earlier guide to APISIX, Prometheus, and KEDA autoscaling covers event-driven scaling with an external controller. The Kubernetes 1.37 change is different: this design uses the core HPA scale-to-zero path for the worker, with a queue metric that remains observable at zero.

Admission limits matter because worker capacity is temporarily absent. Use per-caller and global rate limits to bound how quickly the queue can grow. Set request-size limits before large payloads consume ingestion resources. Apply authentication and authorization before creating a job. If the queue or result store is unavailable, fail explicitly instead of accepting work that was not persisted.

Backpressure should be part of the API contract. A 429 Too Many Requests with a meaningful retry policy is safer than accepting an unbounded backlog that violates completion objectives.

Design for Cold Starts

Scale-to-zero exchanges idle cost for activation latency. The wake-up path includes metric collection, adapter availability, HPA reconciliation, scheduling, image availability, container startup, dependency connection, and readiness.

Measure the entire path from the first durable job to the first completed job. Then decide whether the service objective can absorb it.

Ways to reduce risk include:

• keep worker images small and available in a nearby registry;
• avoid startup migrations or other global mutations;
• use readiness probes that test actual dependencies;
• pre-pull large images on specialized GPU nodes where appropriate;
• set queue visibility or lease timeouts longer than expected processing startup;
• configure a downscale stabilization window to avoid rapid oscillation;
• keep minReplicas: 1 for workloads whose first-job latency is critical.

The default HPA downscale stabilization window is five minutes. That is a useful guard against short idle gaps, but it should be tested against workload frequency and cost goals.

Make Jobs Idempotent and Observable

Queues normally provide at-least-once delivery, which means a job can be processed more than once after a timeout or worker failure. Require an idempotency key at admission or generate a stable job ID, store execution state, and make result writes conditional.

Observability should connect four stages:

1. The gateway accepted or rejected the API request.
2. The ingestion service persisted the job.
3. The HPA observed demand and changed replicas.
4. A worker started, completed, retried, or failed the job.

Useful metrics include accepted requests, rejected requests, queue depth, oldest-job age, HPA desired replicas, ScaledToZero, metric retrieval failures, Pod startup time, processing duration, retries, dead-letter volume, and end-to-end completion latency.

Propagate a correlation ID from the gateway into the job metadata, but do not put sensitive payloads into metric labels or general-purpose logs.

Upgrade and Rollback Safely

Kubernetes 1.37 enables HPAScaleToZero on both the API server and controller manager. During a version-skewed control-plane upgrade, wait until both components support and enable the feature before applying an HPA with zero minimum replicas.

Before disabling the feature gate or downgrading, set affected HPAs back to minReplicas: 1 or higher and scale any zero-replica workloads up. Otherwise, an older or disabled controller may interpret zero as a manual pause and leave the workload asleep.

Roll out one queue at a time. Keep dashboards open through at least one full scale-down and wake-up cycle. Test a metrics-adapter outage, queue outage, image-pull delay, worker crash, duplicate delivery, and backlog spike before relying on the configuration for production savings.

Production Checklist

• Is the workload asynchronous and backed by a durable queue?
• Does the wake-up metric exist independently of worker Pods?
• Are the gateway and ingestion path kept available?
• Is work acknowledged only after durable persistence?
• Are rate limits, backlog limits, and overload responses defined?
• Are jobs idempotent and safe to retry?
• Is cold-start time included in the completion objective?
• Do alerts cover unavailable external metrics and a growing oldest-job age?
• Is the ScaledToZero condition visible to operators?
• Is the downgrade procedure documented and tested?

Conclusion

Kubernetes 1.37 turns native HPA scale-to-zero into a practical option, but the best first targets are workloads whose demand survives while no Pods run. API-triggered background jobs fit when the request boundary stays available and a durable queue carries the work across the cold start.

Use the API Gateway for identity, policy, traffic limits, and a stable public contract. Use the queue for durability. Use an external metric for activation. Use the HPA for worker capacity. That separation gives platform teams a real idle-cost reduction without pretending that a zero-endpoint Kubernetes Service can hold live HTTP requests.

To build the always-available traffic boundary around this pattern, explore Apache APISIX and the API7 Gateway documentation.