API7 Gateway Debug Sessions: Per-Request Tracing That Starts When the Alert Fires

Debug sessions in API7 Gateway capture the complete execution timeline of individual requests: every processing phase, every plugin in the order the gateway ran it, and the log lines that request produced. An alert policy can create the session itself — so the traffic that set off the alert is the traffic you get to inspect.

That last part is the reason we built it this way. Here is the problem it solves.

The 03:14 problem

An alert fires at 03:14. Checkout is returning 502 on roughly one request in four hundred.

You have metrics, so you know the rate and when it started. You have access logs, so you know which routes and which upstreams. What you do not have is the one thing that would end the investigation in a minute: the actual sequence of what the gateway did to one of those failing requests — which plugins ran, in what order, which one took 900 ms, and what the gateway logged while it was happening.

Every serious API gateway can show you exactly that. Kong, Apigee, Azure API Management, Gravitee — all of them ship a per-request debugging tool that renders the full execution timeline of a single request. The capability is not the gap.

The gap is when you are allowed to turn it on.

The ordering problem

We surveyed how five gateway debugging tools are activated, reading only the vendors' own documentation. The result was more uniform than we expected: none of them ships a built-in trigger. You can reach most of these tools from an API, but the alert-to-capture wiring is glue code you write and operate yourself.

Apigee's alerting notifies a human: it sends "a notification with a summary of the event, including links to the Google Cloud console and the Apigee UI, for further investigation."
Kong's Konnect Debugger is started from the Konnect UI — New session, or Start Debugging from a control plane, service, or route. There is no documented public API for it.
Azure API Management traces a request only if that request carries a time-limited debug token in an Apim-Debug-Authorization header, minted from the Azure Resource Manager API and valid for at most an hour. You cannot trace traffic from clients you do not control, and you cannot trace a request that has already happened.
Gravitee has OpenTelemetry tracing of live traffic, but Debug mode — the tool that renders the policy-chain timeline — does not attach to live traffic. You type a method, path, headers, and body into a form, and the Gateway runs that request through the policy chain.

So the industry-standard workflow is:

alert fires → a human is notified → the human opens a debugging session → the session captures subsequent traffic

The request that triggered the alert is never in that capture. You are always debugging the next occurrence, not the one you were told about. For a steady 5% error rate this is a minor annoyance — the next occurrence is seconds away. For the failure modes that actually consume engineering weeks, it is fatal: the ones that fire twice a day, or only during a deploy, or only when one upstream is slow.

Even building the glue yourself does not fully close it. Apigee is honest about this in its own docs: there is "a delay of approximately 15 seconds between starting a debug session and its activation across Apigee. This can cause missed API requests in the initial 15-20 seconds of a debug session."

There is a second ordering problem hiding inside the first. Debug capture is expensive and it records real payload-adjacent data, so nobody leaves it on. That is correct engineering. But it means the tool is unavailable precisely when you need it and available only when you no longer do.

Making the alert do the work

The fix is not a better trace viewer. It is removing the human from the trigger path.

In API7 Gateway, a debug session is an object with a target set of gateway instances, a duration, and a sampling rule. Anything that can create that object can start a capture — including an alert policy.

You attach a debug session configuration to an alert policy: a duration, a sample count, and optionally a sampling rule. When the policy's conditions are met, the control plane creates one session per configuration for each gateway group whose condition actually fired, targeting every instance in that group. The alert history records which session it created, and the console links straight from the alert into the captured traces.

The 03:14 alert now reads differently. You are not being told to go start an investigation. You are being told an investigation already exists, and here are fifty traces from the window in which it happened.

One design detail matters here. If a session created by the same policy is still active on that gateway group, no new one is created. The active session's own duration becomes the cooldown, so a flapping alert produces one capture window, not hundreds.

What a captured trace actually contains

A trace is an OpenTelemetry trace, rendered as a waterfall.

The root span is the request. Under it sit the gateway's processing phases, and under each phase sit the individual plugin invocations, in the order the gateway ran them:

GET /checkout                                       412ms
├─ apisix.phase.access                              380ms
│  ├─ http_router_match                             0.16ms
│  ├─ apisix.phase.rewrite.plugins.key-auth         3.1ms
│  ├─ apisix.phase.rewrite.plugins.proxy-rewrite    0.4ms
│  └─ apisix.phase.access.plugins.limit-count       371ms   ←
├─ apisix.phase.header_filter                       0.25ms
└─ apisix.phase.body_filter                         1.4ms

Read top to bottom, that is the execution order: the two rewrite-phase plugins run before the access-phase one, and within a phase the higher-priority plugin goes first — key-auth at 2500 ahead of proxy-rewrite at 1008. Here the slow one is limit-count, sitting on a Redis round-trip.

That nesting is the answer to two separate questions people usually have to reason about from source code and configuration: which plugins ran on this request, and in what order. Plugin execution order in this class of gateway is computed from each plugin's priority, merged across route, service, and global-rule configuration — reasoning about it statically is genuinely hard. The trace does not require you to reason. It shows you what happened.

The second thing in the trace is the request's own log lines, attached to the root span as events with their level, source location, and message.

This part has a property worth calling out. Log capture is independent of the gateway's configured error_log level. A debug-level message that nginx would discard because you are running at warn still lands in the trace. You do not have to raise the log level in production and then filter a firehose to correlate log lines to one request — the correlation is structural, because the lines are attached to that request's span.

Capture is bounded: 200 entries per request, each message truncated at 2048 bytes.

Where the trace data goes

The other axis worth being precise about is architectural, and here the differences between products are easy to miss because they are not about features.

The three richest visual debuggers all require the vendor's cloud control plane:

Kong's Konnect Debugger requires data planes attached to Konnect; some of its bug-fix entries in the Kong Gateway changelog are labeled "Konnect Only." Kong is careful about what crosses the boundary — sampling and sanitization run on the data plane before anything is transmitted, auth and identity headers are masked by default, storage is region-pinned, and payload capture is an explicit opt-in. But the traces do land in Konnect.
Apigee's debug tool works with Apigee hybrid, where the runtime runs in your own Kubernetes cluster — but the debug data does not stay there. Google's documentation lists "Debug data" as flowing from the runtime plane to the management plane, and describes Message Processors streaming debug, analytics, and deployment-status data "directly to the Apigee control plane using Google Cloud Pub/Sub."
Azure API Management supports tracing on the self-hosted gateway, but the debug token is minted by management.azure.com and the trace is read back from there.

The two tools that do stay entirely on your infrastructure are also the two least capable. Kong's X-Kong-Request-Debug header has no session and no UI, and no traffic sampling — the client must add the header to each request it wants traced, and header output is truncated at 2 KB. Gravitee's Debug mode, when APIM is self-hosted, never sees live traffic at all.

In API7 Gateway, the data plane sends traces to the control plane over the same authenticated channel it already uses for heartbeats and metrics, and the control plane writes them to a Jaeger instance in your deployment — bundled with the Helm chart and the Docker Compose deployment. There is no vendor endpoint in the path. This falls out of the architecture rather than being a feature: the data plane already runs in your environment and already has exactly one outbound channel.

The honest limits

Debug sessions trace HTTP traffic. Stream (L4) proxying and the TLS handshake are not traced.

A capture window is 10 to 600 seconds. That is enough for an alert-triggered capture and for reproducing a known-bad request; it is not a way to sit and wait for something that happens twice a day.

Each request is captured by at most one session, so if several people debug the same gateway group simultaneously, give each session a sampling rule.

And the trace is stored as captured. If your URLs carry sensitive values in the query string, or your plugins log sensitive values, they are in the trace. Use a narrow sampling rule and restrict who holds the permission to read and export sessions.

The shape of the fix

The interesting part of this was not building a trace viewer. Per-request tracing in an OpenTelemetry-shaped gateway is well-trodden work.

The interesting part was noticing that an entire product category had standardized on a workflow whose first step guarantees you miss the event you were told about — and that the fix is not more capability but a different trigger. The capture already knew how to run. It just needed to be started by the thing that noticed the problem, not by the person who read about it afterwards.