Operating Serverless APIs: Cold Starts, Idempotency, and Asynchronous Handoffs

An API gateway can authenticate and bound a serverless request, route it to a function or submission service, and expose a stable client contract. It cannot remove cold starts, make a timed-out invocation safe to repeat, or turn queue acceptance into proof that business work completed.

Design synchronous and asynchronous APIs as different contracts. Synchronous calls need a deadline that includes gateway and function startup. Asynchronous calls need a durable operation identity, status model, duplicate handling, retention policy, and terminal failure path.

Key Takeaways

• Choose synchronous execution only when the complete latency distribution fits the client deadline.
• Treat a gateway timeout as an unknown outcome if function execution may already have started.
• Put idempotency in the application state transition, not only in a gateway header.
• Return 202 Accepted only after durable handoff and provide an operation-status contract.
• Separate gateway admission, cloud queue acceptance, function execution, and business completion.
• Test cold, warm, throttled, duplicate, expired, and replayed requests.

Choose the Interaction Contract First

Do not hide a long-running synchronous function behind a larger gateway timeout. That consumes client, gateway, and function capacity while making recovery ambiguous.

Separate the Actors

flowchart LR
    C[Authenticated client] --> G[API gateway]
    G -->|short synchronous request| F[Function endpoint]
    G -->|validated submission| S[Submission service]
    S -->|durable enqueue| Q[Managed queue or event service]
    Q --> W[Function worker]
    W --> D[Idempotent state transition]
    D --> O[Operation status store]
    C -->|poll operation ID| G
    G --> O
    F -. execution result .-> G
    Q -. acceptance and delivery evidence .-> M[Operations telemetry]
    W -. execution and terminal outcome .-> M

The gateway admits and routes. The submission service owns durable enqueue and operation creation. The queue or cloud service decides delivery. The function executes. The application state store decides whether a duplicate changes state. Keep these responsibilities distinct in prose, metrics, and incident response.

Budget Cold Starts Explicitly

A cold invocation may include environment allocation, runtime initialization, dependency loading, network setup, and application initialization before handler work begins. Measure each function version with production-like package size, memory, networking, region, and dependencies.

For a synchronous route, budget:

client deadline > gateway processing + connection + platform queue/startup + function execution + response transfer + recovery margin

Provisioned concurrency can prepare AWS Lambda execution environments ahead of invocation, but it adds cost and can spill over when demand exceeds the provisioned level. Pin callers to the version or alias that owns the provisioned capacity; invoking another version does not use that pool. Treat this as a measured latency/cost choice, not a promise that every request is warm.

Example: Proxy a Bounded Function Endpoint

Apache APISIX 3.19.0's pinned aws-lambda plugin source uses the shared generic-upstream implementation to call a configured function URL or API endpoint. The following Route body is a runnable-shape example; replace the placeholder URI and configure cloud authentication through an approved secret path.

{
  "uri": "/reports/preview",
  "plugins": {
    "aws-lambda": {
      "function_uri": "https://example-id.lambda-url.us-east-1.on.aws/preview",
      "timeout": 2500,
      "max_req_body_size": 1048576,
      "ssl_verify": true,
      "keepalive": true,
      "keepalive_timeout": 60000,
      "keepalive_pool": 20
    }
  }
}

The plugin buffers the request body up to max_req_body_size; a body-read error, including an oversized body, returns HTTP 400. If the upstream request itself fails, the generic implementation returns HTTP 503. A function response with status 400 or above is passed through rather than automatically retried by this plugin path.

Its behavior-changing Booleans are explicit:

The plugin source also preserves a caller-supplied Authorization or x-api-key header instead of overwriting it with configured cloud authorization. Strip or reject untrusted client cloud-credential headers before this plugin, or prove that passing them is the intended contract. Do not let the caller select function_uri, region, or cloud service.

This proxy path is still synchronous HTTP from APISIX's perspective. It does not by itself create a durable asynchronous operation.

Design a Durable Asynchronous Handoff

A safe submission sequence is:

1. authenticate the caller and map it to server-side policy;
2. validate payload size, schema, and operation type;
3. require or issue a stable idempotency key;
4. create an operation record with a unique ID;
5. durably enqueue the work and commit the accepted state;
6. return 202 Accepted with the operation ID and status URL;
7. process the message idempotently;
8. write a terminal success or failure state;
9. expire payload and status data under a documented retention policy.

If operation creation and queue publication cannot be atomic, use a transactional outbox or another recovery design. Do not return 202 after merely starting an in-memory task.

Example response:

HTTP/1.1 202 Accepted
Content-Type: application/json
Location: /operations/op_01JEXAMPLE
Retry-After: 3

{"operation_id":"op_01JEXAMPLE","status":"accepted"}

accepted means the platform durably accepted responsibility under the stated retention and retry contract. It does not mean the function ran or the business action succeeded.

Make Idempotency an Application Invariant

An idempotency key is useful only when the application binds it to:

• authenticated caller or tenant;
• operation type and canonical request fingerprint;
• stored result or terminal state;
• retention window;
• conflict behavior when the same key carries a different payload.

The gateway may validate presence and format, but the state-changing service must atomically enforce the key with the business transition. A caller-supplied key is a correlation value, not trusted identity. Namespace it with authenticated context.

For a duplicate request, return the original operation or result when the fingerprint matches. Return a conflict when the same key is reused for different work. Test concurrent duplicates, not only sequential retries.

Understand Cloud Retry Semantics

For AWS Lambda Event Functions, asynchronous invocation queues the event and returns 202 when Lambda accepts it. AWS documents that function errors are retried twice by default, while throttling and system errors may be retried for up to six hours by default. The queue is eventually consistent, so duplicate delivery is possible even without a function error.

These are Lambda service semantics, not APISIX plugin enforcement. Configure maximum event age, retries, and an on-failure destination or dead-letter path for the selected function. Keep the handler idempotent even when retries are reduced.

Handle Synchronous Unknown Outcomes

When the gateway times out, ask whether the function could still be running or could have committed a side effect. If yes, the result is ambiguous. Do not automatically replay a payment, message, or provisioning request without an idempotency invariant and status lookup.

Return an operation identifier early when the workflow can outlive the connection. For genuinely synchronous operations, keep the total retry budget inside the client deadline and retry only when the method and application semantics permit it.

Observe Every Stage

Track separately:

• gateway admits, rejects, timeouts, and upstream statuses;
• submission acceptance and enqueue latency;
• queue age, backlog, redelivery, and expiry;
• cold/warm initialization and function duration;
• concurrency throttling and provider errors;
• idempotency hits, conflicts, and duplicate suppression;
• business success, failure, and ambiguous state;
• callback attempts and terminal delivery.

Use bounded labels such as API, operation type, function alias, environment, and terminal status. Keep request IDs, user IDs, and idempotency keys out of Prometheus labels; store them in access-controlled logs or traces when policy permits.

Serverless API Checklist

• Is the API explicitly synchronous, asynchronous, event-based, or callback-based?
• Does the total synchronous deadline include cold startup and recovery margin?
• Can a timeout occur after side effects begin, and how is that ambiguity resolved?
• Is durable handoff complete before returning 202 Accepted?
• Is the idempotency key namespaced by authenticated identity and request fingerprint?
• Are queue retries, duplicate delivery, expiry, and terminal failure configured and tested?
• Are ssl_verify and keepalive defaults and both paths documented?
• Are untrusted client cloud-authorization headers stripped or intentionally accepted?
• Can operators correlate gateway, queue, function, and business outcomes?
• Does rollback preserve compatible function aliases, event schemas, credentials, and status records?

FAQ

Can an API gateway eliminate Lambda cold starts?

No. It can keep upstream connections warm and route to a prepared alias, but execution-environment readiness belongs to the serverless platform and function configuration.

Does 202 Accepted mean the function succeeded?

No. It means the chosen submission boundary accepted the work. Provide a status contract and terminal failure path.

Is an idempotency header enough to prevent duplicate effects?

No. The application must atomically bind the key to authenticated context, request meaning, and the business state transition.

Next Steps

Apply safe timeout and retry policy, design asynchronous gateway patterns, and deliver changes through gateway change management.