API Gateway Cache Memory: Lessons from Cloudflare's 100 TB DNS Optimization

An API gateway cache can reduce upstream latency and load, but it also converts traffic diversity into state. Every cache key, identity partition, response body, header, timestamp, and eviction record consumes memory. At low volume, that overhead is easy to ignore. At large scale, a few unnecessary bytes multiplied by millions of entries can determine how many gateway instances a platform needs.

That lesson was unusually visible this week. Cloudflare described how five data-layout changes reduced the per-entry footprint of its 1.1.1.1 DNS cache from 953 bytes to 420 bytes. Across more than 250 billion entries, the production rollout freed roughly 100 TB of memory; insert throughput increased 43%, and lookup latency fell 19%. The accompanying Hacker News discussion drew attention because the result came from careful representation and measurement, not a new cache algorithm.

An API response cache is not a DNS cache, and those numbers should not be transferred to a gateway capacity plan. The engineering method does transfer: control key cardinality, remove unused capacity, keep common data compact, and measure the complete lookup path under a realistic workload.

Key Takeaways

• Cache capacity is the number of entries multiplied by the full per-entry cost, not just the response-body bytes.
• A cache key is both a correctness boundary and a cardinality decision. Adding tenants, headers, queries, or users can multiply memory use.
• Immutable cached values should not pay for growth capacity they will never use.
• Better memory locality can improve latency as well as reduce memory consumption.
• API gateway teams should benchmark hit rate, resident memory, lookup latency, insert cost, and upstream offload together.
• Apache APISIX proxy-cache provides memory and disk strategies, explicit TTLs, response-size limits, and identity isolation that make these tradeoffs configurable.

Why the Cache Entry Matters More Than It Looks

A simple capacity estimate often looks like this:

cache memory = entry count × average response size

The more useful model is:

cache memory = entry count × (key + metadata + object overhead + allocated capacity + response)

The key may include a method, normalized URI, selected query parameters, content negotiation headers, tenant, or authenticated consumer. Metadata may include timestamps, TTLs, status, hit counters, lengths, and eviction structures. The runtime adds object headers, pointers, allocator rounding, and unused capacity. A 500-byte response does not imply a 500-byte cache entry.

Cloudflare's DNS cache optimization report illustrates the multiplier. Its team replaced growable containers with fixed-size representations after insertion, combined separate lists, removed repeated values that could be reconstructed from the key, and packed common record data more tightly. The important observation is not that every gateway should copy a Rust layout. It is that stored data has a different lifecycle from data under construction.

An API gateway usually builds an entry once and reads it many times. That makes the stored representation a first-class performance interface.

Start with Cache-Key Cardinality

Before optimizing bytes, decide how many distinct keys the gateway is allowed to create. A technically correct but overly specific key can destroy reuse.

Consider an endpoint such as:

GET /catalog/items?category=books&page=1

If the key includes every request header, then harmless differences in User-Agent, trace headers, cookies, or request IDs create separate entries. If it omits a header that changes the representation, such as Accept-Language, callers can receive the wrong response. If it includes an authenticated consumer, each identity gets a safe but separate namespace.

Treat each key component as a documented choice:

APISIX proxy-cache enables consumer_isolation by default. When a request resolves to a consumer or remote user, that identity is prepended to the effective cache key. This is a security-conscious default, but platform teams must include it in their memory and hit-rate model. Disabling isolation is safe only when the cached response is genuinely identical and authorized for all callers.

Choose Memory and Disk for Different Jobs

Memory caching minimizes lookup overhead, but its capacity competes directly with gateway workers, TLS state, plugins, buffers, and connection pools. Disk caching can hold more data at a lower RAM cost, but introduces filesystem behavior and different latency characteristics.

Use memory for small, frequently reused responses with a controlled key space. Use disk for larger objects, longer-lived entries, or workloads where a small latency increase is acceptable. Do not choose memory merely because it is faster in a microbenchmark.

Set a response-size ceiling. One unexpectedly large JSON response can displace many small, high-value entries or create allocation pressure. APISIX exposes max_resp_body_size for the memory strategy and streams larger responses without caching them. That limit should reflect observed payload distributions, not the largest response the API can theoretically produce.

TTLs are another capacity control. Long TTLs can improve hit rates but retain cold entries and stale data. Short TTLs reduce residency but increase insert churn and upstream traffic. Align TTLs with the actual change rate and the business cost of staleness. A product catalog, feature configuration, exchange rate, and authorization result should not share one default.

A Bounded APISIX Cache Policy

The following route fragment shows the shape of a deliberately bounded memory policy. It assumes that the named memory cache zone is already configured for the APISIX deployment; verify the syntax and zone configuration against the documentation for the deployed version.

{
  "plugins": {
    "proxy-cache": {
      "cache_strategy": "memory",
      "cache_zone": "memory_cache",
      "cache_method": ["GET"],
      "cache_http_status": [200],
      "cache_ttl": 60,
      "consumer_isolation": true,
      "max_resp_body_size": 1048576
    }
  }
}

This configuration makes several decisions explicit: only successful GET responses are eligible, entries live for a bounded time, authenticated identities do not share entries, and responses larger than 1 MiB bypass the memory cache.

The configuration still needs application knowledge. APISIX always refuses to cache upstream responses marked Cache-Control: private, no-store, or no-cache. For in-memory caching, cache_control governs request-side directives and TTL derivation from max-age or s-maxage; it does not override those response-side restrictions. Teams still need to define which query parameters and headers change the representation and how mutations invalidate related objects. API7.ai's earlier guide to multi-layer caching in an API gateway explains where gateway, local, and upstream caches can complement one another.

Optimize the Whole Lookup Path

Reducing the serialized response size is useful, but the hot path includes more than payload bytes:

1. normalize the request and build the key;
2. hash or compare the key;
3. locate the entry and check freshness;
4. enforce identity or policy boundaries;
5. reconstruct headers and the response;
6. update counters or eviction state;
7. send the result to the client.

Cloudflare found that fewer allocations and more contiguous data improved lookup latency. API gateway teams should look for the same relationship in their own runtime. A representation that is compact but expensive to decode may save memory and lose throughput. A representation with many pointers may be easy to update but unfriendly to CPU caches.

Benchmark using real distributions. Include the common small response, uncommon large response, typical query combinations, authenticated and anonymous traffic, and the actual hit/miss ratio. Measure at a steady state after the cache fills; cold-start memory dips are not representative of normal operation.

At minimum, track:

• process resident memory and cache-zone occupancy;
• entry count, insertion rate, eviction rate, and expiration rate;
• hit, miss, bypass, and stale-response outcomes;
• lookup latency for hits and upstream latency for misses;
• response-size distribution and oversized bypasses;
• upstream request reduction and error behavior;
• memory and latency at p90, p98, and p99, not only the average.

The gateway's metrics and logging capabilities should connect cache outcomes to route, consumer, and upstream health without putting raw cache keys or sensitive query values into high-cardinality metric labels.

Avoid Cache Optimization That Breaks Correctness

Memory efficiency is never a reason to collapse security boundaries. A shared entry must not mix tenants, authorization scopes, personalized content, or data regions. Cache keys containing secrets should be hashed or avoided, and logs should not expose them.

Also define behavior during failures. If the cache is full, does the gateway evict an entry or bypass caching? If storage is unavailable, does the request continue upstream? Can stale content be served, and for which routes? How does a deployment or configuration change invalidate incompatible entries?

The safest sequence is:

1. prove the key is correct;
2. bound the eligible methods, statuses, sizes, and TTLs;
3. observe the real workload;
4. optimize the representation or topology;
5. confirm that latency, hit rate, and upstream load improved together.

Conclusion

Cloudflare's 100 TB result is a reminder that cache performance is a data-structure problem at fleet scale. API gateways face the same multiplier at a smaller but still operationally important level. Key cardinality, identity partitions, response limits, TTLs, and per-entry overhead determine whether a cache reduces infrastructure cost or merely moves it into gateway memory.

Start with correctness and bounded capacity. Then profile the stored representation and the complete hit path under production-like traffic. Apache APISIX and API7 Enterprise provide the policy and observability points needed to make caching an explicit, measurable part of API delivery rather than an invisible memory tax.