--- title: "API Gateway Log Storage: Pipelines, Retention, Security, and Cost - API7.ai" description: "Design an API gateway log storage pipeline with safe schemas, bounded delivery, tiered retention, access controls, and predictable cost." url: "https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-storage" --- [Skip to article](#column-content) Column contentsAPI Gateway Guide [API Gateway Guide](https://api7.ai/learning-center/api-gateway-guide) Search this column 01 Gateway Fundamentals [View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=fundamentals)[1. What Is an API Gateway? API Gateway Software and Use Cases](https://api7.ai/learning-center/api-gateway-guide/what-is-an-api-gateway)[2. Why Do Modern Architectures Need an API Gateway?](https://api7.ai/learning-center/api-gateway-guide/why-do-modern-architectures-need-api-gateway)[3. Reverse Proxy vs Load Balancer vs API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-vs-reverse-proxy-vs-load-balancer)[4. API Gateway and Microservices: Decoupling and Traffic Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-and-microservices)[5. Understanding Core API Gateway Features](https://api7.ai/learning-center/api-gateway-guide/core-api-gateway-features)[6. Open Source vs. Commercial API Gateways: How to Choose the Right One?](https://api7.ai/learning-center/api-gateway-guide/open-source-vs-commercial-api-gateway)[7. API Gateway vs Service Mesh: How to Choose?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-vs-service-mesh)[8. Envoy vs NGINX, Java, and Go for API Gateways](https://api7.ai/learning-center/api-gateway-guide/nginx-vs-envoy-vs-java-vs-go)[9. Core Concepts of API Gateway: Service, Route, Upstream, Plugin, and Consumer](https://api7.ai/learning-center/api-gateway-guide/core-concepts-of-api-gateway)[10. API Gateway Comparison: Apache APISIX vs. Kong vs. Traefik vs. KrakenD vs. Tyk](https://api7.ai/learning-center/api-gateway-guide/api-gateway-comparison-apisix-kong-traefik-krakend-tyk) 02 Gateway Architecture and Design [View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=architecture)[11. Deep Dive into API Gateway Architecture: Core Components and Design Principles](https://api7.ai/learning-center/api-gateway-guide/api-gateway-architecture)[12. Choosing the Right API Gateway for Your Architecture](https://api7.ai/learning-center/api-gateway-guide/api-gateway-monolithic-vs-microservices)[13. How to Architect an API Gateway for High Availability (HA)?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-high-availability)[14. API Gateway Plugin Architecture: How to Extend and Customize Your API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-plugin-system)[15. How Event-Driven Architecture (EDA) Works with API Gateway?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-event-driven-architecture)[16. API Gateway Request Lifecycle Explained](https://api7.ai/learning-center/api-gateway-guide/api-gateway-request-lifecycle-explained)[17. Cloud-Native API Gateway on Kubernetes](https://api7.ai/learning-center/api-gateway-guide/cloud-native-api-gateway-kubernetes)[18. API Gateway Control Plane vs. Data Plane](https://api7.ai/learning-center/api-gateway-guide/api-gateway-control-plane-vs-data-plane)[19. How API Gateways Work with GitOps: Automating API Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-gitops-integration)[20. How to Perform A/B Testing and Canary Releases with an API Gateway](https://api7.ai/learning-center/api-gateway-guide/ab-testing-canary-release-api-gateway) 03 Gateway Setup and Traffic Handling [View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=traffic-protocols)[21. How to Deploy an API Gateway on Kubernetes: Step-by-Step Guide for Engineers](https://api7.ai/learning-center/api-gateway-guide/deploy-api-gateway-on-kubernetes)[22. Hands-on Setup with Apache APISIX API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-getting-started-guide)[23. API Gateway Auth: OAuth 2.0, JWT & OIDC with APISIX](https://api7.ai/learning-center/api-gateway-guide/api-gateway-authentication-apache-apisix-oauth-jwt-oidc)[24. Mastering Traffic Control: Efficiently Managing API Requests with Gateways](https://api7.ai/learning-center/api-gateway-guide/api-gateway-traffic-control-strategies)[25. Building Reliable API Gateways with Logging and Monitoring](https://api7.ai/learning-center/api-gateway-guide/api-gateway-logging-monitoring-best-practices)[26. Dynamic Routing in API Gateways: Best Practices for Flexible Traffic Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-dynamic-routing-management)[27. How API Gateways Support WebSocket](https://api7.ai/learning-center/api-gateway-guide/how-api-gateways-support-websocket)[28. API Gateway for GraphQL: Security, Cost, and Caching](https://api7.ai/learning-center/api-gateway-guide/api-gateway-handle-graphql)[29. How API Gateways Handle HTTP/3 QUIC Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-http3-quic)[30. How API Gateways Handle MQTT Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-mqtt) 04 Gateway Integrations, AI, and MCP [View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=ai-mcp)[31. How API Gateways Handle gRPC Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-grpc-support)[32. How API Gateways Implement Custom Plugins](https://api7.ai/learning-center/api-gateway-guide/api-gateway-custom-plugins)[33. How API Gateways Integrate with Kafka](https://api7.ai/learning-center/api-gateway-guide/api-gateway-kafka-integration)[34. How to Implement API Aggregation in API Gateways](https://api7.ai/learning-center/api-gateway-guide/api-gateway-api-aggregation)[35. How API Gateways Enable Serverless Architecture](https://api7.ai/learning-center/api-gateway-guide/api-gateway-serverless-architecture)[36. How API Gateways Proxy LLM Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-proxy-llm-requests)[37. How API Gateways Enhance MCP Servers](https://api7.ai/learning-center/api-gateway-guide/api-gateway-enhance-mcp-server)[38. MCP Gateway vs MCP Proxy vs AI Gateway vs API Gateway](https://api7.ai/learning-center/api-gateway-guide/ai-gateway-vs-mcp-gateway-vs-api-gateway)[39. What Is an AI Gateway? Architecture and Benefits](https://api7.ai/learning-center/api-gateway-guide/what-is-an-ai-gateway)[40. What Is an MCP Gateway? Architecture, Use Cases & How It Works (2026 Guide)](https://api7.ai/learning-center/api-gateway-guide/what-is-mcp-gateway) 05 Gateway Performance [View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=performance)[41. How API Gateway Reduces Latency: Performance Optimization Strategies](https://api7.ai/learning-center/api-gateway-guide/how-api-gateway-reduces-latency)[42. API Gateway Caching Strategy: Comprehensive Analysis and Implementation Guide](https://api7.ai/learning-center/api-gateway-guide/api-gateway-caching-strategy-analysis)[43. How to Optimize API Gateway Load Balancing: Advanced Strategies and Best Practices](https://api7.ai/learning-center/api-gateway-guide/how-to-optimize-api-gateway-load-balancing)[44. How API Gateway Efficiently Handles Large-Scale Traffic: Architecture and Strategies](https://api7.ai/learning-center/api-gateway-guide/how-api-gateway-handles-large-scale-traffic)[45. How to Improve API Gateway QoS: Traffic Classes, Capacity Protection, and SLOs](https://api7.ai/learning-center/api-gateway-guide/api-gateway-quality-of-service)[46. API Gateway Log Storage: Pipelines, Retention, Security, and Cost](https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-storage)[47. API Gateway Timeouts and Retries: Safe Policies and APISIX Configuration](https://api7.ai/learning-center/api-gateway-guide/api-gateway-timeouts-retries)[48. API Gateway Concurrency Control: Capacity Budgets, Backpressure, and Load Shedding](https://api7.ai/learning-center/api-gateway-guide/api-gateway-concurrency-control)[49. Asynchronous API Gateway Patterns: Queues, Callbacks, and Performance Boundaries](https://api7.ai/learning-center/api-gateway-guide/asynchronous-api-gateway-patterns)[50. API Gateway TLS Performance: Handshakes, Session Reuse, and Measurement](https://api7.ai/learning-center/api-gateway-guide/api-gateway-tls-performance) 06 Gateway Security [View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=security)[51. DDoS Defense at the API Gateway: Layered Controls and Failure Planning](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ddos-defense)[52. Distributed API Gateway Rate Limiting: Local vs Redis Counters and Accuracy Trade-offs](https://api7.ai/learning-center/api-gateway-guide/distributed-api-gateway-rate-limiting)[53. API Gateway IP Allowlist and Denylist Management: Trust, Proxies, and Operations](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ip-allowlist-denylist-management)[54. API Gateway Security Scanning: What to Test and How to Operationalize Findings](https://api7.ai/learning-center/api-gateway-guide/api-gateway-security-scanning)[55. SQL Injection and XSS at the API Gateway: Detection Limits and Defense in Depth](https://api7.ai/learning-center/api-gateway-guide/sql-injection-xss-api-gateway)[56. API Access Log Auditing: Evidence, Integrity, Retention, and Review](https://api7.ai/learning-center/api-gateway-guide/api-access-log-auditing)[57. Preventing API Scraping and Abuse: Identity, Behavioral Signals, and Rate Controls](https://api7.ai/learning-center/api-gateway-guide/prevent-api-scraping-abuse)[58. API Gateway mTLS: Client Identity, Certificate Rotation, and APISIX Configuration](https://api7.ai/learning-center/api-gateway-guide/api-gateway-mtls)[59. Fine-Grained API Gateway Authorization: RBAC, ABAC, and External Policy Decisions](https://api7.ai/learning-center/api-gateway-guide/fine-grained-api-gateway-authorization)[60. API Gateway and WAF Reference Architecture: Ownership, Header Trust, and Failure Modes](https://api7.ai/learning-center/api-gateway-guide/api-gateway-waf-reference-architecture) 07 Gateway Ecosystem and Comparisons [View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=ecosystem)[61. Apache APISIX vs Kong: Migration, Operations, and Team Fit](https://api7.ai/learning-center/api-gateway-guide/apache-apisix-vs-kong)[62. Traefik vs Kong vs APISIX: A Reproducible API Gateway Benchmark Method](https://api7.ai/learning-center/api-gateway-guide/traefik-kong-apisix-benchmark-method)[63. Amazon API Gateway Explained: Deployment Models, Trade-offs, and Use Cases](https://api7.ai/learning-center/api-gateway-guide/amazon-api-gateway-explained)[64. API Gateway and Istio: Clear North-South and East-West Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-istio)[65. API Gateway with NGINX: Layering, Ownership, and Migration Patterns](https://api7.ai/learning-center/api-gateway-guide/api-gateway-nginx)[66. API Gateway with Envoy: Deployment Patterns and Policy Ownership](https://api7.ai/learning-center/api-gateway-guide/api-gateway-envoy)[67. API Gateway with Cloudflare API Shield: Edge and Origin Security Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-cloudflare-api-shield)[68. Apigee vs MuleSoft vs Amazon API Gateway: A Decision Framework](https://api7.ai/learning-center/api-gateway-guide/apigee-mulesoft-amazon-api-gateway)[69. Should an API Gateway Connect Directly to a Database? Patterns and Risks](https://api7.ai/learning-center/api-gateway-guide/api-gateway-database-access-patterns)[70. API Gateway Architecture for Large-Scale SaaS Platforms](https://api7.ai/learning-center/api-gateway-guide/api-gateway-large-scale-saas-architecture) 08 Industry Applications and Integrations [View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=applications)[71. API Gateway CI/CD: Validation, Promotion, and Safe Rollback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ci-cd)[72. API Gateways for AI and ML Inference: Traffic, Security, and Observability](https://api7.ai/learning-center/api-gateway-guide/api-gateways-ai-ml-inference)[73. API Gateway Architecture for Financial Services: Controls and Failure Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-financial-services)[74. API Gateway Architecture for E-Commerce: Checkout Reliability and Traffic Peaks](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ecommerce)[75. API Gateways for Mobile Applications: Authentication, Payloads, and Versioning](https://api7.ai/learning-center/api-gateway-guide/api-gateways-mobile-applications)[76. API Gateways for Web3 Applications: RPC Traffic, Identity, and Abuse Controls](https://api7.ai/learning-center/api-gateway-guide/api-gateways-web3-applications)[77. OpenAPI-Driven API Gateway Operations: Import, Validation, and Drift Control](https://api7.ai/learning-center/api-gateway-guide/openapi-driven-api-gateway-operations)[78. gRPC-Web and gRPC Transcoding Through an API Gateway](https://api7.ai/learning-center/api-gateway-guide/grpc-web-grpc-transcoding-api-gateway)[79. API Gateway Pricing and Cost Optimization: A Workload-Based Model](https://api7.ai/learning-center/api-gateway-guide/api-gateway-pricing-cost-optimization)[80. Evaluating API Gateway Directions: Evidence and Architectural Impact](https://api7.ai/learning-center/api-gateway-guide/api-gateway-trends) 09 Advanced Gateway Applications [View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=operations)[81. API Gateways in Zero Trust Architecture: Identity, Policy, and Limits](https://api7.ai/learning-center/api-gateway-guide/api-gateways-zero-trust-architecture)[82. API Gateway Log Analytics: Correlation, Queries, and Operational Signals](https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-analytics)[83. From API Sprawl to Governed Delivery: The Gateway's Role in Modernization](https://api7.ai/learning-center/api-gateway-guide/api-gateway-modernization)[84. API Monetization at the Gateway: Plans, Metering, Entitlements, and Billing Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-monetization-gateway)[85. Safe AI-Assisted API Gateway Optimization: Recommendations, Guardrails, and Approval](https://api7.ai/learning-center/api-gateway-guide/ai-assisted-api-gateway-optimization)[86. API Gateways for Low-Code and No-Code Integration: Governance and Safety](https://api7.ai/learning-center/api-gateway-guide/api-gateways-low-code-no-code-integration)[87. API Gateway and CIAM Integration: Token Validation, Consent, and Identity Context](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ciam-integration)[88. Elastic API Gateway Scaling: Signals, Capacity, and Safe Scale-In](https://api7.ai/learning-center/api-gateway-guide/elastic-api-gateway-scaling)[89. Multi-Tenant SaaS API Gateways: Isolation, Quotas, and Tenant-Aware Policy](https://api7.ai/learning-center/api-gateway-guide/multi-tenant-saas-api-gateway-policy)[90. API Gateway DevSecOps: Policy as Code, Testing, and Runtime Feedback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-devsecops) 10 Platform Engineering and Planning [View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=platform-governance)[91. API Discovery and Documentation at the Gateway: Inventory, Ownership, and Drift](https://api7.ai/learning-center/api-gateway-guide/api-discovery-documentation-gateway)[92. WebAssembly Plugins for API Gateways: Portability, Isolation, and APISIX Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-webassembly-plugins)[93. Monitoring API Gateways with Prometheus and Grafana: Metrics, Dashboards, and Alerts](https://api7.ai/learning-center/api-gateway-guide/monitor-api-gateways-prometheus-grafana)[94. API Gateway Reliability Engineering: SLOs, Failure Isolation, and Incident Response](https://api7.ai/learning-center/api-gateway-guide/api-gateway-reliability-engineering)[95. API Gateways for Edge AI Inference: Routing, Constraints, and Observability](https://api7.ai/learning-center/api-gateway-guide/api-gateways-edge-ai-inference)[96. Reducing Cloud Cost with API Gateway Controls: Measurement Before Optimization](https://api7.ai/learning-center/api-gateway-guide/reducing-cloud-cost-api-gateway-controls)[97. Improving API Developer Experience with Gateway Standards and Self-Service](https://api7.ai/learning-center/api-gateway-guide/improving-api-developer-experience-gateway-self-service)[98. API Gateway Change Management: Compatibility, Progressive Delivery, and Rollback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-change-management-rollback)[99. Operating Serverless APIs: Cold Starts, Idempotency, and Asynchronous Handoffs](https://api7.ai/learning-center/api-gateway-guide/operating-serverless-apis-gateway)[100. Planning an API Gateway Roadmap for the Next Five Years](https://api7.ai/learning-center/api-gateway-guide/planning-api-gateway-roadmap-next-five-years) [API Gateway Guide](https://api7.ai/learning-center/api-gateway-guide)[Gateway Performance](https://api7.ai/learning-center/api-gateway-guide?topic=performance) On this page On this page - [Key Takeaways](#key-takeaways) - [The Gateway Is a Producer, Not the Archive](#the-gateway-is-a-producer-not-the-archive) - [Choose the Delivery Pattern Deliberately](#choose-the-delivery-pattern-deliberately) - [Design a Minimum Useful Schema](#design-a-minimum-useful-schema) - [Configure a Bounded APISIX HTTP Log Pipeline](#configure-a-bounded-apisix-http-log-pipeline) - [Separate Operational Logs from Audit Records](#separate-operational-logs-from-audit-records) - [Set Retention and Cost by Data Class](#set-retention-and-cost-by-data-class) - [Test the Failure Path, Not Only the Happy Path](#test-the-failure-path-not-only-the-happy-path) - [API Gateway Log Storage Checklist](#api-gateway-log-storage-checklist) - [FAQ](#faq) - [Should an API gateway write directly to Elasticsearch?](#should-an-api-gateway-write-directly-to-elasticsearch) - [How long should API gateway logs be retained?](#how-long-should-api-gateway-logs-be-retained) - [Are API access logs the same as audit logs?](#are-api-access-logs-the-same-as-audit-logs) - [Next Steps](#next-steps) Article 46 # API Gateway Log Storage: Pipelines, Retention, Security, and Cost API7.ai September 7, 2026 An API gateway should produce structured events and deliver them through a bounded pipeline; it should not be the long-term system of record. Store searchable operational logs in a log platform, move older data to cheaper archive storage when justified, and keep security audit records under stricter access and integrity controls. That separation protects request processing from a slow log destination, gives each data class an explicit retention policy, and makes cost visible. It also prevents a common mistake: collecting every header and body first and deciding whether the data was safe only after it reached several systems. ## Key Takeaways - Separate log generation, transport, indexing, archive, and deletion responsibilities. - Prefer an allowlisted schema; do not log credentials, cookies, or bodies by default. - Give the delivery buffer a hard bound and decide what happens when the destination is unavailable. - Treat operational access logs and security audit logs as different products with different controls. - Set retention by purpose, investigation window, legal requirements, and cost—not by a universal number of days. - Monitor the logging pipeline itself, including queue depth, discarded entries, delivery errors, and ingestion lag. ## The Gateway Is a Producer, Not the Archive The gateway is well placed to record the route, authenticated identity context, upstream result, and timing for each request. It is poorly placed to retain months of data: local disks are finite, gateway instances are replaced, and a storage outage must not consume all memory or block API traffic. A production pipeline normally separates five stages: ``` flowchart LR A[API gateway\nstructured event] --> B[Bounded batch or buffer] B --> C[Collector or broker] C --> D[Hot searchable store] D --> E[Lower-cost archive] E --> F[Expiration or deletion] C --> G[Security analytics] ``` The bounded buffer absorbs short interruptions. A collector or broker normalizes delivery and decouples gateway availability from the storage backend. The hot store supports recent troubleshooting and dashboards. Archive storage is optional and should contain only data that has a defined future use. Expiration is part of the design, not an afterthought. This article focuses on storage and delivery. For choosing logs, metrics, and traces together, see [API gateway logging and monitoring best practices](https://api7.ai/learning-center/api-gateway-guide/api-gateway-logging-monitoring-best-practices). ## Choose the Delivery Pattern Deliberately Apache APISIX provides logger plugins for several delivery models, including [`http-logger`](https://apisix.apache.org/docs/apisix/plugins/http-logger/), [`kafka-logger`](https://apisix.apache.org/docs/apisix/plugins/kafka-logger/), [`elasticsearch-logger`](https://apisix.apache.org/docs/apisix/plugins/elasticsearch-logger/), [`file-logger`](https://apisix.apache.org/docs/apisix/plugins/file-logger/), and [`syslog`](https://apisix.apache.org/docs/apisix/plugins/syslog/). The existence of a connector does not determine the right architecture. | Pattern | Good fit | Main trade-off | | --------------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------- | | Standard output or file plus node agent | Container or VM platforms with an established collector | Depends on local rotation, disk limits, and agent health | | HTTP(S) collector | Simple centralized ingestion with few moving parts | Collector capacity and TLS configuration become critical | | Kafka or another durable broker | High volume, multiple consumers, or replay requirements | Adds an operational system and end-to-end lag | | Direct search-store delivery | Small, controlled environments | Couples gateways more closely to indexing availability and schema changes | For most distributed deployments, a local collector or durable broker creates a cleaner failure boundary than sending every gateway instance directly to a search cluster. Direct delivery may still be reasonable when scale is modest and the team has tested failure behavior. ## Design a Minimum Useful Schema Start from the questions responders and security teams actually need to answer. A useful access event often contains: | Field group | Examples | Design note | | ----------- | --------------------------------------------- | ------------------------------------------------------------------ | | Correlation | timestamp, request ID, trace ID | Use stable identifiers across services | | Routing | route ID, service ID, method, route template | Prefer a template over a raw URI with identifiers or query strings | | Identity | consumer or tenant pseudonym, credential type | Avoid the credential itself | | Outcome | gateway status, upstream status, bytes sent | Distinguish gateway rejection from upstream failure | | Timing | total, upstream, and gateway latency | Use consistent units and definitions | | Network | trusted client address or region | Define proxy trust before recording a forwarded address | Do not record `Authorization`, API keys, session cookies, secrets in query strings, or request and response bodies by default. The [OWASP Logging Cheat Sheet](https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html) recommends sanitizing event data and protecting logs during transport and storage. Even fields that appear harmless can become personal or confidential when combined, so document purpose and access for the whole event. OpenTelemetry's [logs data model](https://opentelemetry.io/docs/specs/otel/logs/data-model/) can help normalize timestamps, severity, resource attributes, trace correlation, and the event body when logs from gateways and services share one pipeline. Adopting the model does not remove the need for redaction or retention rules. ## Configure a Bounded APISIX HTTP Log Pipeline The following APISIX 3.18 configuration excerpt defines a global allowlisted log format and a maximum pending-entry backlog for `http-logger`. Plugin metadata is global: it affects every Route and Service that uses this plugin, so review existing consumers before changing it. ``` curl "http://127.0.0.1:9180/apisix/admin/plugin_metadata/http-logger" \ -X PUT \ -H "X-API-KEY: ${admin_key}" \ -d '{ "log_format": { "timestamp": "$time_iso8601", "request_id": "$apisix_request_id", "route_id": "$route_id", "service_id": "$service_id", "method": "$request_method", "status": "$status", "request_time": "$request_time", "upstream_response_time": "$upstream_response_time", "bytes_sent": "$bytes_sent" }, "max_pending_entries": 8192 }' ``` Next, enable delivery to an internal HTTPS collector. Certificate verification is explicit because the current `http-logger` default is disabled. The example intentionally omits an authentication value; use your platform's secret-management mechanism if the collector requires one rather than embedding a real secret in a Route. ``` curl "http://127.0.0.1:9180/apisix/admin/routes/orders-read" \ -X PUT \ -H "X-API-KEY: ${admin_key}" \ -d '{ "uri": "/orders/*", "methods": ["GET"], "plugins": { "request-id": {}, "http-logger": { "uri": "https://logs.internal.example/v1/events", "ssl_verify": true, "include_req_body": false, "include_resp_body": false } }, "upstream": { "type": "roundrobin", "nodes": { "orders.internal:8080": 1 } } }' ``` APISIX 3.15 and later exposes `$apisix_request_id` as the gateway's current request ID. With `request-id` enabled, this is the value the plugin generated or accepted and returned in the configured response header. Because the plugin can accept a non-empty client-provided ID, do not treat the value as security evidence by itself: validate its format, keep authenticated identity in separate fields, and generate a server-controlled correlation identifier if your trust model requires one. According to the current APISIX documentation, `http-logger` batches records and its metadata defaults `max_pending_entries` to 8,192. When the backlog exceeds the configured bound, new entries are discarded so an unreachable logger cannot grow worker memory without limit. That is a deliberate availability trade-off, not durable delivery. If losing an access event is unacceptable, put a locally reachable durable transport in front of the central store and verify its disk and backpressure behavior. A gateway logger plugin alone should not be described as an immutable audit system. ## Separate Operational Logs from Audit Records Operational access logs answer questions such as “which route is slow?” or “which upstream returned 503?” Security audit records answer questions such as “who changed this policy?” and “who accessed protected data?” They can share some infrastructure, but they differ in provenance and controls. An audit design may require: - authenticated actor and administrative action details; - append-only or tamper-evident storage; - separation of duties for readers and administrators; - documented evidence preservation and deletion procedures; - alerts for access, export, or policy changes to the audit store. Gateway request logs cannot prove an administrative event they never observed. Combine data-plane access events with control-plane configuration and identity-provider audit events where the investigation requires both. ## Set Retention and Cost by Data Class There is no safe universal retention period. Choose it by working backward from incident response, customer commitments, regulation, litigation holds, and deletion obligations. A simple tiering model is: - **Hot:** recent, indexed data for interactive search and alert investigation. - **Warm:** older data with slower or less expensive query capacity. - **Archive:** compressed objects kept for a defined recovery or evidence need. - **Expired:** cryptographically or physically deleted according to policy, including derived copies where required. Estimate daily volume before selecting a store: ``` daily ingest bytes = requests per day × average encoded event size ``` Use that ingestion baseline to model each stage separately. A broker estimate applies its retention window, compression ratio, and replication factor. A hot-search estimate applies indexed size, shard replicas, retention, and index overhead. Archive and backup estimates apply their own compression, duplication, and retention rules. Add network transfer and query compute once at the stage that incurs them. This prevents the same replica cost from being counted twice. Sampling can reduce routine success-event cost, but do not sample error, security, billing, or compliance events unless the owning requirement explicitly allows it. Reducing event size through a narrow schema is often safer than collecting sensitive fields and relying on sampling. ## Test the Failure Path, Not Only the Happy Path Verify the pipeline with controlled experiments: 1. Send known requests and confirm the event schema, timestamps, and correlations. 2. Put test credentials and personal-looking values in disallowed headers and bodies; confirm they never arrive. 3. Make the collector slow, unreachable, and certificate-invalid; measure API latency and gateway memory. 4. Fill the configured backlog and confirm the expected discard behavior and alert. 5. Restore the collector and measure recovery time and ingestion lag. 6. Test retention expiration, archive retrieval, access approval, and audit trails. APISIX's [`prometheus` plugin](https://apisix.apache.org/docs/apisix/plugins/prometheus/) exposes `apisix_batch_process_entries`, a gauge for entries remaining in a logger batch. Pair it with destination-side delivery errors, broker lag, index rejection, storage saturation, and the age of the newest searchable event; add an independent discard signal where your delivery design requires one. ## API Gateway Log Storage Checklist - Is each event tied to a documented operational, security, or compliance purpose? - Are fields allowlisted and sensitive values excluded before transport? - Is transport encrypted and is the destination identity verified? - Is buffering bounded, with an explicit loss or backpressure policy? - Can the pipeline survive collector, broker, index, and network failures? - Are operational and audit data access controls appropriately separated? - Does every storage tier have an owner, cost estimate, retention rule, and deletion test? - Are logging-pipeline health and ingestion lag visible without depending on the same broken pipeline? ## FAQ ### Should an API gateway write directly to Elasticsearch? It can, but direct delivery couples gateway instances to the search platform's availability, credentials, and schema. A collector or broker is usually preferable when volume, multiple consumers, or failure isolation matter. ### How long should API gateway logs be retained? There is no one-size-fits-all duration. Retain each data class only as long as its incident, business, contractual, legal, or regulatory purpose requires, and test deletion when that period ends. ### Are API access logs the same as audit logs? No. Access logs describe request processing. Audit logs are evidence about security-relevant actions and often need stronger provenance, integrity, access, and retention controls. ## Next Steps Use [API gateway logging and monitoring best practices](https://api7.ai/learning-center/api-gateway-guide/api-gateway-logging-monitoring-best-practices) to connect the storage pipeline with metrics and traces. For centrally managed Apache APISIX deployments that need enterprise operations and governance, explore [API7 Enterprise](https://api7.ai/enterprise). [Previous article**How to Improve API Gateway QoS: Traffic Classes, Capacity Protection, and SLOs**](https://api7.ai/learning-center/api-gateway-guide/api-gateway-quality-of-service)[Next article**API Gateway Timeouts and Retries: Safe Policies and APISIX Configuration**](https://api7.ai/learning-center/api-gateway-guide/api-gateway-timeouts-retries) On this page - [Key Takeaways](#key-takeaways) - [The Gateway Is a Producer, Not the Archive](#the-gateway-is-a-producer-not-the-archive) - [Choose the Delivery Pattern Deliberately](#choose-the-delivery-pattern-deliberately) - [Design a Minimum Useful Schema](#design-a-minimum-useful-schema) - [Configure a Bounded APISIX HTTP Log Pipeline](#configure-a-bounded-apisix-http-log-pipeline) - [Separate Operational Logs from Audit Records](#separate-operational-logs-from-audit-records) - [Set Retention and Cost by Data Class](#set-retention-and-cost-by-data-class) - [Test the Failure Path, Not Only the Happy Path](#test-the-failure-path-not-only-the-happy-path) - [API Gateway Log Storage Checklist](#api-gateway-log-storage-checklist) - [FAQ](#faq) - [Should an API gateway write directly to Elasticsearch?](#should-an-api-gateway-write-directly-to-elasticsearch) - [How long should API gateway logs be retained?](#how-long-should-api-gateway-logs-be-retained) - [Are API access logs the same as audit logs?](#are-api-access-logs-the-same-as-audit-logs) - [Next Steps](#next-steps)