HTTP Connection Reuse: The Hidden API Latency Multiplier

An API can be fast at the origin and still feel slow because a client repeatedly pays to establish TCP and TLS connections. That cost is easy to miss: application traces show a short handler, the gateway reports healthy backends, and no single request looks broken. The lost time sits between requests, where a reusable connection should have been.

Go made this failure mode unusually visible in October 2026. Its engineering team explained how Go 1.27 improves HTTP/1 connection reuse when a caller closes a response without reading its body to the end. In the post's local example, 100 sequential requests fell from about 710 milliseconds to about 150 milliseconds. That is an illustration from one machine and workload, not a general five-times performance promise.

The Hacker News discussion focused on the runtime fix, but the operational lesson is larger. API teams need to know which hop owns each connection, why reuse was lost, and whether a proposed fix trades handshake cost for extra body reads, idle sockets, or retry risk.

Key Takeaways

• Draw the full path: client → API gateway → origin. Each hop has its own protocol, pool, limits, and failure modes.
• A successful HTTP response does not prove that its connection returned to the reusable pool.
• Go 1.27's automatic drain is bounded to 50 milliseconds or 256 KiB and applies to unread HTTP/1 response bodies.
• HTTP/2 multiplexing changes the mechanics; do not copy an HTTP/1 body-drain workaround into every protocol.
• Measure handshake rate, connection churn, request latency, and origin latency together before changing pool sizes.
• Treat gateway keep-alive tuning and client-library behavior as related evidence, not interchangeable fixes.

Why an Unread Body Can Discard a Healthy Connection

HTTP/1 responses share a byte stream. Before a client can parse the next response on the same connection, it must know where the current response ends. Content-Length, chunked transfer framing, or connection closure supplies that boundary, but the client still has to consume the framed body.

Through Go 1.26, closing resp.Body with unread bytes generally prevented net/http.Transport from reusing that connection. A program that wanted only a status code could therefore open another connection for its next request. Even JSON decoding could leave one trailing newline unread after the decoder found the closing object delimiter.

Starting in Go 1.27, Transport asynchronously drains an unread HTTP/1 response body for up to 50 milliseconds or 256 KiB, according to the Go 1.27 release notes. It skips work when reuse cannot help, such as a response with Connection: close. The bounds matter: downloading an unlimited or slow body merely to save a socket could cost more than establishing another connection.

This behavior does not make body ownership optional. Applications still need to close bodies, stream large responses deliberately, and set request deadlines. It also does not change an API gateway written on another network stack. It removes one sharp edge in one Go client path.

Map the Two Connection Pools

Teams often say “the gateway connection” as if there were one socket. A reverse proxy terminates one downstream connection and opens or reuses a separate upstream connection.

flowchart LR
    Client[API client] -->|downstream connection| Gateway[API gateway]
    Gateway -->|upstream connection| Origin[API origin]
    Gateway --> Metrics[latency and status evidence]

The two hops can use different protocols. A client may speak HTTP/2 to the gateway while the gateway uses HTTP/1.1 to an origin. A mobile client may create a new downstream connection on every network transition while the gateway maintains a warm upstream pool. Conversely, downstream reuse can look healthy while an origin sends Connection: close or a gateway pool is undersized.

Ask four questions for each hop:

Question	Client → gateway	Gateway → origin
Which protocol is negotiated?	HTTP/1.1, HTTP/2, or HTTP/3	Depends on gateway and upstream configuration
Who owns the pool?	Client library, SDK, browser, or sidecar	Gateway worker and upstream configuration
What prevents reuse?	Unread body, close header, timeout, network change	Close header, pool pressure, idle expiry, origin behavior
Which evidence exists?	Client trace, DNS/TCP/TLS timings	Gateway latency, origin logs, socket and handshake telemetry

This map prevents a common mistake: changing an upstream gateway pool to fix a downstream mobile-client problem.

Diagnose Churn Before Tuning Keep-Alive

Start with a controlled comparison, not a production-wide configuration change.

1. Choose one route and one client version. Mixes of SDKs and protocols blur the result.
2. Record total and origin latency. Apache APISIX's Prometheus plugin exposes request, upstream, and gateway-side latency views. Those signals do not identify every handshake by themselves, but their divergence helps locate the slow hop.
3. Count new connections and handshakes. Use client tracing, origin server metrics, load-balancer logs, or a bounded packet capture in a test environment.
4. Check response framing and completion. Look for unread bodies, early cancellation, streaming responses, Connection: close, and timeouts.
5. Repeat with connection reuse disabled. If latency barely changes, the suspected pool may not be the bottleneck. If it changes sharply, confirm why before increasing limits.

Correlate the result with the broader API observability signals for errors, saturation, and route-level latency. A lower handshake count is useful only if tail latency, error rate, memory, and file-descriptor pressure remain acceptable.

Configure the Gateway Side Deliberately

Apache APISIX lets an Upstream use a separate keep-alive pool. The current APISIX Admin API documentation exposes pool size, idle timeout, and requests-per-connection controls. This article references the APISIX 3.19 documentation available on October 20, 2026; verify fields and defaults against the version you deploy.

There is no universal correct pool size. A larger pool can reduce connection creation, but it also retains more idle sockets per worker and per upstream. Long idle timeouts may preserve warm connections while increasing the chance that an intermediary or origin has already discarded them. A high requests-per-connection value reduces churn but may conflict with an origin's lifecycle policy.

Tune from the path's capacity model:

• gateway workers × upstreams × idle connections per pool;
• expected concurrency and request duration;
• TLS handshake cost and network round-trip time;
• origin limits, load-balancer idle timeout, and deployment churn;
• long-lived streams versus short request-response traffic.

Keep retries separate from reuse. Retrying an idempotent request after a stale pooled connection fails may be safe under a defined policy. Retrying a payment or another non-idempotent write can duplicate a side effect. Connection tuning must not quietly broaden retry semantics.

Design API Responses for Predictable Consumption

Clients and gateways cannot optimize around an API contract that sends unnecessary or unbounded response bodies.

For status-oriented operations, return the smallest representation that still communicates the result. A 204 No Content response may fit an operation with no representation, while a structured error body is valuable when a call fails. Do not remove useful diagnostics merely to save a connection; make success and error shapes intentional.

For large exports and streams, document whether callers may stop early, whether byte ranges or pagination exist, and how cancellation reaches the origin. For ordinary JSON APIs, keep framing correct and avoid middleware that appends unexpected bytes. Test with the actual SDKs your consumers use.

The same principle appears in TLS key exchange planning for API gateway origins: handshake choices, connection lifetime, and compatibility belong to one measured path. Optimizing one component without the path model can move cost rather than remove it.

A Safe Rollout Checklist

Before changing client or gateway behavior, capture a baseline and a rollback trigger:

• protocol on both sides of the gateway;
• new connections and TLS handshakes per request;
• total, upstream, and gateway-attributed latency percentiles;
• client cancellations, resets, and timeout errors;
• gateway and origin socket counts;
• response-body size distribution;
• retry count and method distribution;
• memory and file-descriptor headroom.

Upgrade one client cohort or adjust one Upstream at a time. Confirm that fewer handshakes correspond to better tail latency without increasing stale-connection errors or resource pressure. For Go applications, test on Go 1.27 with the real response sizes and cancellation patterns; do not infer production gains from the blog's small sequential benchmark.

Conclusion

Connection reuse is a path property, not a switch labeled “keep-alive.” Go 1.27 fixes a specific HTTP/1 client sharp edge, but API teams still need to distinguish downstream from upstream pools, protocol behavior from configuration, and lower handshake counts from actual user value.

Use an API gateway as the observable boundary: compare end-to-end and upstream latency, keep pool settings explicit, and roll changes out by route and client cohort. If you operate Apache APISIX or API7 Gateway, start with one high-volume API, draw both connection hops, and test the smallest change that explains the evidence.