Asynchronous API Gateway Patterns: Queues, Callbacks, and Performance Boundaries

An API gateway supports asynchronous processing by securing, validating, limiting, and routing an acceptance request—but the application must durably record the work before returning success. A reliable design returns 202 Accepted with a job identifier and status location, makes duplicate submissions safe, exposes a terminal outcome, and applies backpressure when the queue or workers are saturated.

Returning 202 from the gateway without a durable handoff does not make work asynchronous. It only makes failure harder for the client to see.

Key Takeaways

A 202 Accepted response means processing has not finished; it does not promise eventual success.
Persist the job or message before acknowledging acceptance.
Give clients a status resource, callback, event, or another explicit completion channel.
Make submission idempotent because clients cannot always distinguish a lost response from a lost request.
Limit queue age, depth, request size, and per-tenant intake; asynchronous work can move overload rather than remove it.
Keep gateway and application responsibilities separate: policy at the edge, workflow state in a durable owner.

When an Asynchronous API Helps

The synchronous request-response model is appropriate when work can predictably finish within the caller's deadline. An asynchronous contract is useful when an operation:

takes seconds or minutes, such as media conversion or report generation;
experiences bursts that workers should absorb gradually;
depends on slow or rate-limited external services;
needs independent retry and recovery after the client disconnects;
produces a result too late for one open HTTP connection.

The performance benefit is specific: the client and gateway release the acceptance connection sooner, and a durable queue can decouple intake from worker throughput. Total work does not disappear. Queue storage, scheduling, status reads, callbacks, and worker processing add their own latency and cost.

Do not convert a fast operation to asynchronous merely to conceal an overloaded dependency. First determine whether the operation needs a long-running workflow or simply needs concurrency control and capacity correction.

Define the Acceptance Contract

RFC 9110 section 15.3.3 defines 202 Accepted for a request accepted for processing whose processing is not complete and might ultimately be disallowed. The response ought to describe the request's current status and point to a status monitor when one is available.

A practical submission looks like this:

POST /v1/report-jobs HTTP/1.1
Host: api.example.com
Authorization: Bearer <token>
Idempotency-Key: 6d28a698-11e4-47bc-a85b-5427fbd89261
Content-Type: application/json

{"account_id":"acct-42","format":"csv"}

HTTP/1.1 202 Accepted
Location: /v1/report-jobs/job-8f31
Retry-After: 3
Content-Type: application/json

{
  "id": "job-8f31",
  "status": "queued",
  "status_url": "/v1/report-jobs/job-8f31"
}

The application should issue this response only after it has committed enough state to recover and process the job. A safe sequence is:

sequenceDiagram
    participant C as Client
    participant G as API Gateway
    participant A as Intake Service
    participant Q as Durable Queue/Store
    participant W as Worker
    C->>G: POST job + idempotency key
    G->>G: Authenticate, validate, admit
    G->>A: Forward accepted request
    A->>Q: Atomically record job/message
    Q-->>A: Durable acknowledgement
    A-->>G: 202 + job ID + status URL
    G-->>C: 202 Accepted
    W->>Q: Claim and process job
    C->>G: GET status URL
    G->>A: Read authorized status
    A-->>G: queued/running/succeeded/failed
    G-->>C: Authorized status response

If the service sends 202 before the durable acknowledgement, a crash can silently lose accepted work. If it waits for the worker to finish, it has recreated a synchronous operation.

Choose a Completion Pattern

Poll a job-status resource

Polling is simple for clients that can make outbound HTTP requests but cannot expose a public callback endpoint. Model the job as a resource with a small state machine:

queued -> running -> succeeded
                  -> failed
queued/running -> cancelled

Return stable terminal states, timestamps, a safe error category, and an authenticated result link. Use Retry-After, exponential polling backoff, or server-documented intervals to prevent the status endpoint from becoming the dominant load.

Deliver a callback or webhook

A callback reduces polling but creates another distributed-system boundary. Require HTTPS, authenticate the delivery, sign the body, include a timestamp and delivery identifier, and define replay protection. Treat redirects and DNS changes carefully to reduce server-side request forgery risk. Use bounded retries and a dead-letter path; do not retry forever.

Callback delivery is normally at least once. The receiver must deduplicate events and query the authoritative status resource if event order is uncertain.

Publish an event

For controlled service-to-service environments, publish a completion event to a broker or event stream. Define the message schema, partitioning/order expectation, retention, consumer authentication, and replay behavior. The public HTTP gateway can still govern job intake and status access; it does not need to impersonate the broker.

One API may offer both polling and callbacks, but one durable job record should remain the source of truth.

Make Duplicate Submission Safe

After the application commits a job, the response can be lost. The client sees a timeout and cannot know whether to submit again. An idempotency key gives the server a way to return the existing job instead of creating a second one.

A robust implementation:

scopes the key to the authenticated tenant and operation;
stores a fingerprint of the relevant request payload;
atomically records the key, job identifier, and response state;
returns the same job for an exact replay;
rejects reuse of the key with a different payload;
retains the record longer than the documented client retry window.

Forwarding an Idempotency-Key header through the gateway is useful, but it is not enforcement. The application or durable acceptance component must provide the atomic behavior. Workers also need idempotent effects or deduplication because queue delivery can be repeated after a crash.

Put Backpressure at Intake

A queue smooths a burst only while its depth and job age remain acceptable. If intake continuously exceeds completion, the queue is an expanding outage.

Define limits for:

accepted jobs per tenant and per operation;
concurrent jobs and maximum queued jobs;
maximum payload and result size;
maximum queue age before work is no longer useful;
worker attempts and total processing deadline;
external-service calls and spending;
retention for job records, results, and dead letters.

Reject before durable acceptance when the system cannot honor the contract. A tenant-specific intake limit generally maps to 429. Shared saturation can map to 503. Do not return 202 for a job placed into a queue that has no plausible processing capacity.

Large inputs are often better uploaded directly to object storage with a short-lived, scoped upload credential. Submit a reference plus integrity metadata to the job API rather than buffering the entire object through the gateway and queue.

Configure the APISIX Intake Route

Apache APISIX should enforce edge policy while an intake service owns durable workflow state. The following APISIX 3.18 example authenticates the caller, validates a small JSON body, bounds the arrival rate, and exports metrics. It does not claim that APISIX itself enqueues the job.

Prerequisites are Apache APISIX 3.18, an Admin API key, a reachable job-intake.internal:8080 service, and an APISIX Consumer with a key-auth credential. The block is a parseable route example; replace the upstream address, schema, and numeric policy with values from your environment.

curl "http://127.0.0.1:9180/apisix/admin/routes/report-jobs" \
  -X PUT \
  -H "X-API-KEY: ${admin_key}" \
  -d '{
    "uri": "/v1/report-jobs",
    "methods": ["POST"],
    "plugins": {
      "key-auth": {},
      "request-validation": {
        "body_schema": {
          "type": "object",
          "required": ["account_id", "format"],
          "properties": {
            "account_id": {"type": "string", "maxLength": 80},
            "format": {"type": "string", "enum": ["csv", "json"]}
          },
          "additionalProperties": false
        }
      },
      "limit-req": {
        "rate": 5,
        "burst": 10,
        "key_type": "var",
        "key": "consumer_name",
        "rejected_code": 429,
        "policy": "local"
      },
      "prometheus": {}
    },
    "upstream": {
      "type": "roundrobin",
      "nodes": {
        "job-intake.internal:8080": 1
      }
    }
  }'

The request-validation plugin checks the configured schema before the request reaches the intake service. Keep business invariants in the application as well. The numeric limits above are illustrative and must be replaced with measured tenant and worker capacity.

This example explicitly uses policy: local, so each APISIX node keeps its own counter. In a multi-node fleet, the aggregate admitted rate can exceed five requests per second for one Consumer. Divide the budget conservatively across nodes or evaluate a supported Redis-backed policy when the intake service requires a shared fleet-wide limit; include that dependency's latency and availability in the design.

With a test Consumer credential stored in consumer_api_key, send a schema-valid request:

curl "http://127.0.0.1:9080/v1/report-jobs" \
  -X POST \
  -H "apikey: ${consumer_api_key}" \
  -H "Idempotency-Key: 6d28a698-11e4-47bc-a85b-5427fbd89261" \
  -H "Content-Type: application/json" \
  -d '{"account_id":"acct-42","format":"csv"}'

The success condition is a 202 response with the intake service's job identifier and status location after durable acceptance. An invalid body should be rejected before reaching the service, and an invalid credential should fail authentication. Requests above the steady rate but within the burst allowance may be delayed; requests that exhaust the configured burst allowance should return 429. Exact response bodies depend on the application and plugin configuration.

Create a separate route for status reads so its authorization, cache behavior, and rate policy can differ from submission. Verify that a caller can access only jobs in its own tenant. Never place credentials or sensitive input in a predictable job identifier or status URL.

Clients can express a preference for asynchronous handling with Prefer: respond-async, defined by RFC 7240. It is a preference, not a command. Only advertise or return Preference-Applied: respond-async when the application actually implements that negotiation.

Operate the Whole Workflow

Gateway latency alone is a misleading success metric. Track:

acceptance rate, rejection rate, and acceptance latency;
queue depth and age of the oldest ready job;
time from acceptance to start and to terminal outcome;
worker concurrency, success, failure, retry, and dead-letter rate;
duplicate submissions and idempotency conflicts;
status-read and callback delivery load;
expired, cancelled, and abandoned results;
cost per accepted and completed job.

Propagate a correlation identifier from the submission to the durable job, worker attempts, result, and callback. Do not propagate one live tracing span for hours; link workflow events using stable identifiers and trace relationships supported by your telemetry system.

Test Failure at Every Handoff

Before launch, test these ambiguous moments:

Crash before durable commit: the client must not receive 202.
Crash after commit but before response: a retry with the same idempotency key must return the existing job.
Deliver the same queue message twice: externally visible effects must not duplicate.
Lose or reorder callbacks: the receiver must deduplicate and reconcile from status.
Stop workers while intake continues: queue-age alerts and admission limits must activate.
Expire a result while a client polls: the API must return a documented terminal representation rather than an ambiguous absence.
Revoke a tenant while jobs remain: define whether work is cancelled, quarantined, or allowed to finish.

Also load-test status reads and callbacks. Moving work out of the original request often creates two or more new API traffic streams.

Decision Checklist

Does the operation truly outlive a reasonable HTTP deadline?
What durable event allows the service to return 202?
How does a client discover success, failure, cancellation, and expiration?
What makes resubmission and worker redelivery safe?
Who may read the status or result?
When does intake reject because the queue is too deep or too old?
What is the retention and deletion policy?
Which gateway metrics join to queue, worker, and application signals?

Summary

An asynchronous API is a workflow contract, not a status-code shortcut. The API gateway should authenticate, validate, meter, and observe intake, while a durable application component records the job before acknowledging it. Combine 202 Accepted with a status or notification channel, atomic idempotency, bounded queueing, and explicit overload behavior. That design releases request connections quickly without turning lost work and growing queues into hidden failure modes.

FAQ

Can an API gateway return `202` and enqueue the work itself?

Only when a documented gateway capability provides a durable handoff with the required delivery and recovery semantics. In the common design, the gateway forwards to an intake service, and that service commits the job before returning 202.

Is polling worse than a webhook?

Not universally. Polling is easier for clients that cannot receive inbound requests; webhooks reduce repeated reads but require authenticated delivery, replay protection, retry, and dead-letter handling.

Does a queue remove the need for rate limits?

No. A queue absorbs a bounded mismatch between intake and completion. If arrivals continuously exceed workers, job age and storage grow until the workflow fails or becomes too expensive.

Next Steps

Define the durable-acceptance point and job state machine before configuring the gateway. Then use the APISIX request-validation documentation to validate the intake schema supported by your release. For centrally governed Apache APISIX deployments, explore API7 Enterprise.