API Gateway Concurrency Control: Capacity Budgets, Backpressure, and Load Shedding

API gateway concurrency control protects an upstream by limiting how many requests may be in flight at once. Set the limit from measured sustainable capacity, allow only a small and time-bounded waiting allowance, and reject excess work before the upstream enters a latency and timeout spiral. Use request-rate limits alongside concurrency limits: rate controls arrivals, while concurrency controls occupied capacity.

This article focuses on the capacity boundary itself. For broader service classes and objectives, see API gateway quality of service. For retry behavior after an admission decision, see API gateway timeouts and retries.

Key Takeaways

• Requests per second and concurrency measure different risks; a modest request rate can exhaust a slow dependency.
• Estimate a starting budget from arrival rate and service time, then establish the real safe value with load tests and production signals.
• Prefer a small bounded delay or an early rejection to an unbounded queue at the gateway.
• Place limits at the narrowest resource boundary: route, tenant, upstream pool, or expensive operation.
• Use 429 for a caller-specific policy and 503 when shared service capacity is unavailable; document the retry contract.
• Test slow-upstream and partial-failure cases, not only a healthy throughput benchmark.

Why Request Rate Is Not Enough

Suppose two endpoints each receive 100 requests per second. One normally completes in 20 milliseconds; the other waits two seconds on a reporting database. The first has roughly two requests in flight on average, while the second has roughly 200. Their request rates are identical, but their demand on connections, memory, database slots, and other finite resources is not.

A useful planning relationship is:

average in-flight requests ≈ arrival rate × average service time

This is a form of Little's Law when the observed system is stable. It is a starting estimate, not a safe configuration by itself. Tail latency, burstiness, retries, unequal request cost, and downstream pool limits can make the required headroom very different from the average.

Concurrency control addresses the question, “How much work may occupy this dependency now?” A rate limit instead asks, “How quickly may new work arrive?” Mature admission control normally uses both.

Find the Capacity Boundary Before Setting the Number

The useful concurrency limit is not “the most connections the gateway can accept.” It is the maximum admitted work that the protected path can complete while meeting its latency and error objectives.

Identify the first scarce resource for the route:

• upstream worker or thread slots;
• database connections or transaction contention;
• sockets to a partner API;
• CPU for cryptography, compression, or transformation;
• memory for large request and response buffers;
• a paid downstream service with its own quota.

Then run a step test with representative payloads. Increase offered concurrency gradually and observe throughput, queueing time, latency percentiles, errors, and resource saturation. Sustainable capacity normally ends before the absolute throughput peak: once latency rises sharply while completed throughput changes little, accepting more in-flight work mostly creates a queue.

Set an initial gateway budget below that knee and leave capacity for health checks, operational traffic, failover, and variance. Re-test after application, database, instance-size, or dependency changes. A limit copied from another route is not evidence that the protected resource can sustain it.

Convert Excess Load into Explicit Backpressure

When all admission slots are occupied, the gateway has three choices:

1. Delay briefly. This can absorb a small scheduling mismatch, but the wait must have a strict bound.
2. Reject immediately. This preserves capacity and gives the caller a clear backpressure signal.
3. Accept durably for later work. This requires an application-level asynchronous contract and durable queue; holding an HTTP request open is not durable acceptance.

An unbounded gateway queue is the dangerous fourth choice. It consumes connections and memory, raises tail latency, causes callers to time out, and can trigger retries while the original work still waits. Google SRE guidance on overload recommends load shedding when serving every request would threaten the service.

Use status codes to communicate the boundary:

• 429 Too Many Requests is usually appropriate when a consumer, credential, or tenant exceeded its assigned policy. A server can include Retry-After when it has a meaningful retry time.
• 503 Service Unavailable is usually clearer when the shared service is temporarily unable to accept more work.

The status does not control retry behavior by itself. Publish whether retries are allowed, require backoff and jitter at clients, and ensure retry attempts do not bypass the same admission boundary.

Apply Budgets at More Than One Scope

A single global limit can stop total collapse but still allow a noisy tenant or expensive operation to consume every slot. Layer the smallest number of budgets that correspond to real resources:

flowchart LR
    C[Clients] --> G[Gateway fleet safety limit]
    G --> T[Per-tenant or consumer budget]
    T --> R[Per-route cost class]
    R --> U[Upstream pool capacity]
    U --> D[(Database or dependency)]

• Fleet or instance safety limit: prevents the gateway layer itself from exhausting file descriptors, memory, or CPU.
• Tenant budget: provides fairness and contains one customer's burst.
• Route budget: distinguishes an expensive export from a fast metadata read.
• Upstream budget: protects the actual shared pool even when several routes reach it.

Be precise about whether a gateway counter is local to one process, one instance, or shared across the fleet. A per-instance limit of 100 across ten independently counted instances can admit far more than 100 requests. Either divide the global budget conservatively, use a supported shared counter where available, or enforce the definitive limit at the resource owner as well.

Configure Apache APISIX limit-conn

Apache APISIX 3.18 provides the limit-conn plugin to limit concurrent requests. conn is the normal concurrency threshold. Requests above it and within the burst allowance are delayed; requests beyond the hard boundary are rejected. default_conn_delay controls the delay calculation.

The following illustrative route protects a slow report service. The values are examples, not recommended production defaults:

curl "http://127.0.0.1:9180/apisix/admin/routes/report-export" \
  -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "uri": "/reports/export",
    "methods": ["POST"],
    "plugins": {
      "limit-conn": {
        "conn": 40,
        "burst": 8,
        "default_conn_delay": 0.1,
        "only_use_default_delay": true,
        "key_type": "var",
        "key": "server_addr",
        "rejected_code": 503,
        "rejected_msg": "Report service is at capacity"
      },
      "prometheus": {}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "reports.internal:8080": 1
      }
    }
  }'

In this example, server_addr creates a gateway-server-scoped key. It is useful for demonstrating a safety boundary but is not a cluster-wide capacity guarantee. Select the key and deployment topology deliberately. For consumer fairness, attach authentication and use a verified consumer-related key supported by the APISIX release you operate; do not trust an arbitrary client-supplied tenant header.

The small burst value is waiting allowance, not extra sustainable capacity. With only_use_default_delay: true, admitted excess requests receive the configured delay. If waiting would violate the caller's deadline, set burst to zero and reject immediately.

APISIX also offers limit-req and limit-count for rate and quota controls. Combine them only after defining which boundary each one owns; overlapping arbitrary limits are hard to explain and operate.

Validate the Policy Under Failure

A healthy benchmark can produce a limit that is unsafe during the event that matters. Test at least these cases in staging:

1. Increase offered concurrency until the upstream reaches its measured knee; verify admission keeps it below the chosen safety point.
2. Make upstream processing progressively slower without changing arrival rate; confirm rejections rise while admitted-request latency stays bounded.
3. Remove part of the upstream pool and verify the budget contracts or is already safe for the reduced capacity.
4. Combine a client retry policy with rejection and verify attempts remain bounded.
5. Send one expensive payload class among many cheap requests and check whether a separate route or cost budget is needed.
6. Run the test across the full gateway fleet to reveal local-counter multiplication.

Assert invariants rather than one peak RPS number: maximum in-flight work, maximum waiting time, bounded total latency, acceptable completion rate, and recovery after load falls.

Monitor Saturation and Rejection Together

The APISIX prometheus plugin exports gateway request and latency metrics. Correlate those with upstream and dependency telemetry. A useful dashboard includes:

• admitted, delayed, and rejected requests by route and trusted consumer class;
• gateway and upstream latency percentiles;
• upstream active requests, worker utilization, connection-pool use, and queue depth;
• retry attempts and request amplification;
• gateway CPU, memory, active connections, and event-loop health;
• capacity-limit changes and deployment events.

Interpret rejection as a protection signal, not automatically as a gateway failure. A rising rejection ratio with stable admitted-request latency can mean the boundary is working. Zero rejections alongside exploding latency and full downstream pools can mean the boundary is absent or too high.

Rollout Checklist

• Name the scarce resource and its owner.
• Measure the sustainable point with realistic payloads and tail latency.
• Decide whether the budget is per instance, route, consumer, pool, or fleet.
• Keep waiting bounded; use early rejection when no useful deadline remains.
• Pair concurrency with a separately justified arrival-rate limit.
• Publish 429 or 503 semantics and the retry contract.
• Test slow, partial-failure, and recovery cases across the full fleet.
• Recalibrate after material capacity or workload changes.

Concurrency control is successful when admitted work remains useful during overload—not when every request enters a queue. With an evidence-based budget, explicit backpressure, and failure-oriented tests, the gateway becomes a capacity firewall for the upstream instead of another place where overload accumulates.

Summary

API gateway concurrency control limits in-flight work at the point where an upstream can still remain healthy. Derive the budget from the protected resource, use only bounded delay, shed excess load explicitly, and layer limits where tenants or routes consume different capacity. Apache APISIX limit-conn can enforce the boundary, while rate controls and Prometheus telemetry complete the operating loop. The final number must come from your workload and failure tests.

FAQ

Is a concurrency limit the same as a rate limit?

No. A rate limit controls how quickly work arrives; a concurrency limit controls how much work is currently occupying capacity. Slow requests can exhaust an upstream even at a modest request rate.

Should excess requests wait or fail immediately?

Wait only when the queue is small, strictly bounded, and still fits the caller's deadline. Otherwise reject early so clients receive explicit backpressure and admitted work can finish.

Does one APISIX limit-conn value protect an entire cluster?

Only if the selected policy and key provide the required sharing semantics. With local counters, each APISIX node enforces its own value. Validate the aggregate behavior across the full fleet.

Next Steps

Use the APISIX limit-conn documentation to verify the attributes supported by your release, then test the policy against a slow upstream. For centralized policy management and operational support around Apache APISIX, explore API7 Enterprise.