API Gateway Log Analytics: Correlation, Queries, and Operational Signals

API gateway log analytics should answer operational questions, not merely collect access records. A useful design connects a client operation to gateway admission, upstream attempts, application outcomes, and configuration changes while making missing or delayed logs visible.

The gateway sees an important boundary, but it does not see everything. Its 200 response does not prove a business transaction completed, and its timeout does not prove the upstream did no work. Join gateway evidence with application-owned outcomes before drawing conclusions.

Key Takeaways

• Design a stable event schema before choosing dashboards or storage technology.
• Separate correlation IDs from authenticated identity and business identifiers.
• Record decision reasons, upstream attempts, latency components, bytes, and configuration version.
• Detect loss, backlog, truncation, and schema drift in the logging pipeline itself.
• Start queries from an operational question and attach an owner and response action.
• Keep request and response bodies disabled by default unless a narrow, reviewed use case justifies them.

Define the Boundary: Storage, Audit, and Analytics

These concerns overlap but are not interchangeable:

• Log delivery and storage move records durably, partition them, retain them, and control access.
• Audit logging preserves evidence about security-relevant actions, integrity, retention, and review.
• Log analytics groups and correlates events to explain behavior, detect change, and guide response.
• Metrics provide bounded numeric time series suited to alerts and trends.
• Traces connect causally related operations across components when propagation and sampling are correct.

An analytics dashboard built on best-effort access logs is not automatically an audit system. Likewise, a durable archive is not useful for operations if fields cannot distinguish a gateway rejection from an upstream failure.

Build an Event Model Around Questions

Start with the questions operators must answer:

Prefer stable IDs for aggregation and keep human-readable names as lookup data. Do not put raw URLs, unbounded user agents, account IDs, or arbitrary error text into metric labels. Logs can carry higher-cardinality fields under access and retention controls.

Correlate Without Inventing Trust

Use several identifiers for different purposes:

• a gateway request ID for one request at the boundary;
• a trace ID for a sampled distributed execution;
• an authenticated subject or consumer reference for security context;
• an application operation or idempotency key for one business action;
• a gateway configuration or release ID for change attribution.

Apache APISIX 3.19.0's request-id plugin preserves a non-empty value already present in the configured request header; otherwise, it generates one. Therefore, the request ID is useful correlation context but is not trusted identity and is not guaranteed globally unique when callers can supply it. Validate length and character set at the edge if it will be indexed.

The plugin's include_in_response option defaults to true. The true path returns the request ID in the configured response header so clients can report it; the false path still forwards and logs the ID but does not expose it in the response. Neither path authenticates the caller.

Emit a Minimal, Bounded Record

The following Apache APISIX 3.19.0 excerpt enables a request ID and sends a deliberately small JSON log to an HTTPS collector. It is not a complete deployment: collector authentication, CA trust, retries, capacity, storage schema, redaction, and alerting must be designed separately.

{
  "uri": "/orders/*",
  "plugins": {
    "request-id": {
      "header_name": "X-Request-Id",
      "include_in_response": true
    },
    "http-logger": {
      "uri": "https://logs.internal.example/v1/gateway",
      "ssl_verify": true,
      "include_req_body": false,
      "include_resp_body": false,
      "log_format": {
        "event_time": "$time_iso8601",
        "request_id": "$apisix_request_id",
        "route_id": "$route_id",
        "consumer_name": "$consumer_name",
        "method": "$request_method",
        "status": "$status",
        "request_time": "$request_time",
        "upstream_time": "$upstream_response_time",
        "bytes_sent": "$bytes_sent"
      }
    }
  },
  "upstream_id": "orders-v2"
}

The http-logger plugin sets include_req_body and include_resp_body to false by default, omitting those payloads. With the default log format, setting either option to true may include an eligible body, subject to conditions, NGINX buffering limits, and the default 524,288-byte per-body cap, after which the logged body is truncated. This excerpt uses a custom log_format: changing either flag to true alone would still not add body fields to its log entry. If a reviewed use case needs a request body, the documentation offers $request_body as a possible explicit format field; verify its availability and limits in the selected deployment. Body logging can capture tokens, personal data, payment data, or large payloads, so enable it only for a narrow route and reviewed condition.

ssl_verify defaults to false for http-logger in APISIX 3.19.0. The false path uses TLS without authenticating the collector certificate; the true path verifies it using the configured trust. This example explicitly selects true. Certificate verification protects the collector connection but does not prove that every log field is authoritative.

The plugin batches delivery. Its global metadata max_pending_entries defaults to 8192; while the backlog exceeds the limit, new entries are discarded to bound worker memory. A gateway response can therefore succeed while its log record is lost. Alert on collector availability and delivery loss, and never describe best-effort transport as complete audit evidence without an independently verified design.

Normalize Before Querying

At ingestion, validate the schema and add controlled metadata such as environment, region, gateway instance, schema version, and configuration version. Preserve the raw event or a cryptographic reference when audit requirements demand it, but query normalized typed fields.

Useful derived fields include:

• status class and normalized gateway error category;
• total, gateway, and upstream latency in consistent units;
• attempt count and selected upstream;
• admitted, denied, throttled, or failed-before-upstream outcome;
• bytes in and out;
• authenticated principal reference and credential type;
• deployment, canary, and policy version;
• log arrival delay and pipeline source.

Keep missing distinct from zero. An absent upstream latency can mean rejection before proxying, connection failure, logging variation, or malformed input. Do not convert every absence into a zero-latency upstream call.

Turn Queries into Operational Signals

The following is an illustrative ClickHouse query for a hypothetical normalized table. Adapt field names and types to the actual schema; it is not an APISIX built-in schema.

SELECT
  toStartOfFiveMinutes(event_time) AS bucket,
  route_id,
  count() AS requests,
  countIf(status >= 500) AS server_errors,
  quantile(0.95)(request_time_ms) AS p95_ms
FROM gateway_logs
WHERE event_time >= now() - INTERVAL 1 HOUR
GROUP BY bucket, route_id
ORDER BY bucket, route_id;

This query can identify a route-level change, but an alert still needs a minimum traffic volume, an error ratio or service-level objective, an owner, and a runbook. A count of five errors means something different at ten requests than at ten million.

Build query families for:

1. Availability: error ratio by route and gateway outcome, separated from client errors.
2. Latency: percentiles split into gateway and upstream components, with traffic volume.
3. Saturation: concurrency, connection pressure, admission rejection, and queue or logger backlog.
4. Security: authentication and policy denial reasons, suspicious credential or source rotation, and bypass attempts.
5. Change impact: before-and-after results by configuration, upstream, region, or canary cohort.
6. Cost: bytes, retry attempts, high-cost route classes, and telemetry volume.

Detect Pipeline Failure and Data Drift

Monitor the logs as a data product:

• compare gateway request counters with received log counts over bounded windows;
• record event time and ingestion time to expose delay;
• reject or quarantine invalid schema versions instead of silently dropping fields;
• track null rates and cardinality changes for critical fields;
• inject a safe synthetic request and verify its record reaches the query layer;
• test collector outage, backlog exhaustion, recovery, and duplicate delivery;
• document whether retries can duplicate a log and how queries deduplicate it.

Absence of an error record is not proof that no error occurred. During a collector outage, dashboards can look healthier precisely because failing events stopped arriving.

Protect Sensitive Data

Collect the least data that answers the operational or audit question. Exclude authorization headers, cookies, secrets, full tokens, and unreviewed bodies. Hashing an email or account number can still leave personal data linkable; apply the organization's privacy classification and retention rules.

Separate access to operational summaries from access to raw security events. Record query and export activity for sensitive datasets. Test deletion, legal hold, and retention transitions rather than assuming the storage policy applied correctly.

Log Analytics Checklist

• Does every field have a type, owner, meaning, and sensitivity classification?
• Are request ID, trace ID, authenticated identity, and business operation kept distinct?
• Can records separate pre-upstream rejection, upstream failure, and application denial?
• Are body logging defaults, true and false paths, custom-format behavior, size caps, and truncation documented?
• Is collector TLS verification explicit?
• Are backlog, discard, delay, duplicate, and schema-failure paths observable?
• Does every alert include traffic context, owner, and response action?
• Are gateway results joined with application outcomes before business conclusions are made?
• Are retention, access, redaction, and deletion tested?

Frequently Asked Questions

Should every request be logged?

Not necessarily. Security or regulatory evidence may require complete records for specific events, while high-volume operational analytics may use bounded sampling or aggregation. Define the requirement per field and event class, then test loss and sampling behavior.

Can a request ID be used as user identity?

No. It is correlation context, and APISIX can preserve a caller-supplied non-empty value. Use identity established by the configured authentication path and keep its provenance separate.

Are logs enough for API observability?

No. Metrics are better for bounded time-series alerting, traces can connect sampled cross-service work, and application state establishes business outcomes. Logs supply detailed events that complement those signals.

Next Steps

Connect this model to API gateway log storage, API access log auditing, and API gateway timeouts and retries.