---
title: "Monitoring API Gateways with Prometheus and Grafana: Metrics, Dashboards, and Alerts - API7.ai"
description: "Build a low-cardinality APISIX monitoring path with a private Prometheus scrape, service dashboards, SLO-oriented queries, and actionable alerts."
url: "https://api7.ai/learning-center/api-gateway-guide/monitor-api-gateways-prometheus-grafana"
---

[Skip to article](#column-content)

Column contentsAPI Gateway Guide

[API Gateway Guide](https://api7.ai/learning-center/api-gateway-guide)

Search this column

01 Gateway Fundamentals

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=fundamentals)[1. What Is an API Gateway? API Gateway Software and Use Cases](https://api7.ai/learning-center/api-gateway-guide/what-is-an-api-gateway)[2. Why Do Modern Architectures Need an API Gateway?](https://api7.ai/learning-center/api-gateway-guide/why-do-modern-architectures-need-api-gateway)[3. Reverse Proxy vs Load Balancer vs API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-vs-reverse-proxy-vs-load-balancer)[4. API Gateway and Microservices: Decoupling and Traffic Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-and-microservices)[5. Understanding Core API Gateway Features](https://api7.ai/learning-center/api-gateway-guide/core-api-gateway-features)[6. Open Source vs. Commercial API Gateways: How to Choose the Right One?](https://api7.ai/learning-center/api-gateway-guide/open-source-vs-commercial-api-gateway)[7. API Gateway vs Service Mesh: How to Choose?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-vs-service-mesh)[8. Envoy vs NGINX, Java, and Go for API Gateways](https://api7.ai/learning-center/api-gateway-guide/nginx-vs-envoy-vs-java-vs-go)[9. Core Concepts of API Gateway: Service, Route, Upstream, Plugin, and Consumer](https://api7.ai/learning-center/api-gateway-guide/core-concepts-of-api-gateway)[10. API Gateway Comparison: Apache APISIX vs. Kong vs. Traefik vs. KrakenD vs. Tyk](https://api7.ai/learning-center/api-gateway-guide/api-gateway-comparison-apisix-kong-traefik-krakend-tyk)

02 Gateway Architecture and Design

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=architecture)[11. Deep Dive into API Gateway Architecture: Core Components and Design Principles](https://api7.ai/learning-center/api-gateway-guide/api-gateway-architecture)[12. Choosing the Right API Gateway for Your Architecture](https://api7.ai/learning-center/api-gateway-guide/api-gateway-monolithic-vs-microservices)[13. How to Architect an API Gateway for High Availability (HA)?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-high-availability)[14. API Gateway Plugin Architecture: How to Extend and Customize Your API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-plugin-system)[15. How Event-Driven Architecture (EDA) Works with API Gateway?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-event-driven-architecture)[16. API Gateway Request Lifecycle Explained](https://api7.ai/learning-center/api-gateway-guide/api-gateway-request-lifecycle-explained)[17. Cloud-Native API Gateway on Kubernetes](https://api7.ai/learning-center/api-gateway-guide/cloud-native-api-gateway-kubernetes)[18. API Gateway Control Plane vs. Data Plane](https://api7.ai/learning-center/api-gateway-guide/api-gateway-control-plane-vs-data-plane)[19. How API Gateways Work with GitOps: Automating API Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-gitops-integration)[20. How to Perform A/B Testing and Canary Releases with an API Gateway](https://api7.ai/learning-center/api-gateway-guide/ab-testing-canary-release-api-gateway)

03 Gateway Setup and Traffic Handling

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=traffic-protocols)[21. How to Deploy an API Gateway on Kubernetes: Step-by-Step Guide for Engineers](https://api7.ai/learning-center/api-gateway-guide/deploy-api-gateway-on-kubernetes)[22. Hands-on Setup with Apache APISIX API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-getting-started-guide)[23. API Gateway Auth: OAuth 2.0, JWT & OIDC with APISIX](https://api7.ai/learning-center/api-gateway-guide/api-gateway-authentication-apache-apisix-oauth-jwt-oidc)[24. Mastering Traffic Control: Efficiently Managing API Requests with Gateways](https://api7.ai/learning-center/api-gateway-guide/api-gateway-traffic-control-strategies)[25. Building Reliable API Gateways with Logging and Monitoring](https://api7.ai/learning-center/api-gateway-guide/api-gateway-logging-monitoring-best-practices)[26. Dynamic Routing in API Gateways: Best Practices for Flexible Traffic Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-dynamic-routing-management)[27. How API Gateways Support WebSocket](https://api7.ai/learning-center/api-gateway-guide/how-api-gateways-support-websocket)[28. API Gateway for GraphQL: Security, Cost, and Caching](https://api7.ai/learning-center/api-gateway-guide/api-gateway-handle-graphql)[29. How API Gateways Handle HTTP/3 QUIC Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-http3-quic)[30. How API Gateways Handle MQTT Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-mqtt)

04 Gateway Integrations, AI, and MCP

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=ai-mcp)[31. How API Gateways Handle gRPC Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-grpc-support)[32. How API Gateways Implement Custom Plugins](https://api7.ai/learning-center/api-gateway-guide/api-gateway-custom-plugins)[33. How API Gateways Integrate with Kafka](https://api7.ai/learning-center/api-gateway-guide/api-gateway-kafka-integration)[34. How to Implement API Aggregation in API Gateways](https://api7.ai/learning-center/api-gateway-guide/api-gateway-api-aggregation)[35. How API Gateways Enable Serverless Architecture](https://api7.ai/learning-center/api-gateway-guide/api-gateway-serverless-architecture)[36. How API Gateways Proxy LLM Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-proxy-llm-requests)[37. How API Gateways Enhance MCP Servers](https://api7.ai/learning-center/api-gateway-guide/api-gateway-enhance-mcp-server)[38. MCP Gateway vs MCP Proxy vs AI Gateway vs API Gateway](https://api7.ai/learning-center/api-gateway-guide/ai-gateway-vs-mcp-gateway-vs-api-gateway)[39. What Is an AI Gateway? Architecture and Benefits](https://api7.ai/learning-center/api-gateway-guide/what-is-an-ai-gateway)[40. What Is an MCP Gateway? Architecture, Use Cases & How It Works (2026 Guide)](https://api7.ai/learning-center/api-gateway-guide/what-is-mcp-gateway)

05 Gateway Performance

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=performance)[41. How API Gateway Reduces Latency: Performance Optimization Strategies](https://api7.ai/learning-center/api-gateway-guide/how-api-gateway-reduces-latency)[42. API Gateway Caching Strategy: Comprehensive Analysis and Implementation Guide](https://api7.ai/learning-center/api-gateway-guide/api-gateway-caching-strategy-analysis)[43. How to Optimize API Gateway Load Balancing: Advanced Strategies and Best Practices](https://api7.ai/learning-center/api-gateway-guide/how-to-optimize-api-gateway-load-balancing)[44. How API Gateway Efficiently Handles Large-Scale Traffic: Architecture and Strategies](https://api7.ai/learning-center/api-gateway-guide/how-api-gateway-handles-large-scale-traffic)[45. How to Improve API Gateway QoS: Traffic Classes, Capacity Protection, and SLOs](https://api7.ai/learning-center/api-gateway-guide/api-gateway-quality-of-service)[46. API Gateway Log Storage: Pipelines, Retention, Security, and Cost](https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-storage)[47. API Gateway Timeouts and Retries: Safe Policies and APISIX Configuration](https://api7.ai/learning-center/api-gateway-guide/api-gateway-timeouts-retries)[48. API Gateway Concurrency Control: Capacity Budgets, Backpressure, and Load Shedding](https://api7.ai/learning-center/api-gateway-guide/api-gateway-concurrency-control)[49. Asynchronous API Gateway Patterns: Queues, Callbacks, and Performance Boundaries](https://api7.ai/learning-center/api-gateway-guide/asynchronous-api-gateway-patterns)[50. API Gateway TLS Performance: Handshakes, Session Reuse, and Measurement](https://api7.ai/learning-center/api-gateway-guide/api-gateway-tls-performance)

06 Gateway Security

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=security)[51. DDoS Defense at the API Gateway: Layered Controls and Failure Planning](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ddos-defense)[52. Distributed API Gateway Rate Limiting: Local vs Redis Counters and Accuracy Trade-offs](https://api7.ai/learning-center/api-gateway-guide/distributed-api-gateway-rate-limiting)[53. API Gateway IP Allowlist and Denylist Management: Trust, Proxies, and Operations](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ip-allowlist-denylist-management)[54. API Gateway Security Scanning: What to Test and How to Operationalize Findings](https://api7.ai/learning-center/api-gateway-guide/api-gateway-security-scanning)[55. SQL Injection and XSS at the API Gateway: Detection Limits and Defense in Depth](https://api7.ai/learning-center/api-gateway-guide/sql-injection-xss-api-gateway)[56. API Access Log Auditing: Evidence, Integrity, Retention, and Review](https://api7.ai/learning-center/api-gateway-guide/api-access-log-auditing)[57. Preventing API Scraping and Abuse: Identity, Behavioral Signals, and Rate Controls](https://api7.ai/learning-center/api-gateway-guide/prevent-api-scraping-abuse)[58. API Gateway mTLS: Client Identity, Certificate Rotation, and APISIX Configuration](https://api7.ai/learning-center/api-gateway-guide/api-gateway-mtls)[59. Fine-Grained API Gateway Authorization: RBAC, ABAC, and External Policy Decisions](https://api7.ai/learning-center/api-gateway-guide/fine-grained-api-gateway-authorization)[60. API Gateway and WAF Reference Architecture: Ownership, Header Trust, and Failure Modes](https://api7.ai/learning-center/api-gateway-guide/api-gateway-waf-reference-architecture)

07 Gateway Ecosystem and Comparisons

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=ecosystem)[61. Apache APISIX vs Kong: Migration, Operations, and Team Fit](https://api7.ai/learning-center/api-gateway-guide/apache-apisix-vs-kong)[62. Traefik vs Kong vs APISIX: A Reproducible API Gateway Benchmark Method](https://api7.ai/learning-center/api-gateway-guide/traefik-kong-apisix-benchmark-method)[63. Amazon API Gateway Explained: Deployment Models, Trade-offs, and Use Cases](https://api7.ai/learning-center/api-gateway-guide/amazon-api-gateway-explained)[64. API Gateway and Istio: Clear North-South and East-West Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-istio)[65. API Gateway with NGINX: Layering, Ownership, and Migration Patterns](https://api7.ai/learning-center/api-gateway-guide/api-gateway-nginx)[66. API Gateway with Envoy: Deployment Patterns and Policy Ownership](https://api7.ai/learning-center/api-gateway-guide/api-gateway-envoy)[67. API Gateway with Cloudflare API Shield: Edge and Origin Security Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-cloudflare-api-shield)[68. Apigee vs MuleSoft vs Amazon API Gateway: A Decision Framework](https://api7.ai/learning-center/api-gateway-guide/apigee-mulesoft-amazon-api-gateway)[69. Should an API Gateway Connect Directly to a Database? Patterns and Risks](https://api7.ai/learning-center/api-gateway-guide/api-gateway-database-access-patterns)[70. API Gateway Architecture for Large-Scale SaaS Platforms](https://api7.ai/learning-center/api-gateway-guide/api-gateway-large-scale-saas-architecture)

08 Industry Applications and Integrations

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=applications)[71. API Gateway CI/CD: Validation, Promotion, and Safe Rollback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ci-cd)[72. API Gateways for AI and ML Inference: Traffic, Security, and Observability](https://api7.ai/learning-center/api-gateway-guide/api-gateways-ai-ml-inference)[73. API Gateway Architecture for Financial Services: Controls and Failure Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-financial-services)[74. API Gateway Architecture for E-Commerce: Checkout Reliability and Traffic Peaks](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ecommerce)[75. API Gateways for Mobile Applications: Authentication, Payloads, and Versioning](https://api7.ai/learning-center/api-gateway-guide/api-gateways-mobile-applications)[76. API Gateways for Web3 Applications: RPC Traffic, Identity, and Abuse Controls](https://api7.ai/learning-center/api-gateway-guide/api-gateways-web3-applications)[77. OpenAPI-Driven API Gateway Operations: Import, Validation, and Drift Control](https://api7.ai/learning-center/api-gateway-guide/openapi-driven-api-gateway-operations)[78. gRPC-Web and gRPC Transcoding Through an API Gateway](https://api7.ai/learning-center/api-gateway-guide/grpc-web-grpc-transcoding-api-gateway)[79. API Gateway Pricing and Cost Optimization: A Workload-Based Model](https://api7.ai/learning-center/api-gateway-guide/api-gateway-pricing-cost-optimization)[80. Evaluating API Gateway Directions: Evidence and Architectural Impact](https://api7.ai/learning-center/api-gateway-guide/api-gateway-trends)

09 Advanced Gateway Applications

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=operations)[81. API Gateways in Zero Trust Architecture: Identity, Policy, and Limits](https://api7.ai/learning-center/api-gateway-guide/api-gateways-zero-trust-architecture)[82. API Gateway Log Analytics: Correlation, Queries, and Operational Signals](https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-analytics)[83. From API Sprawl to Governed Delivery: The Gateway's Role in Modernization](https://api7.ai/learning-center/api-gateway-guide/api-gateway-modernization)[84. API Monetization at the Gateway: Plans, Metering, Entitlements, and Billing Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-monetization-gateway)[85. Safe AI-Assisted API Gateway Optimization: Recommendations, Guardrails, and Approval](https://api7.ai/learning-center/api-gateway-guide/ai-assisted-api-gateway-optimization)[86. API Gateways for Low-Code and No-Code Integration: Governance and Safety](https://api7.ai/learning-center/api-gateway-guide/api-gateways-low-code-no-code-integration)[87. API Gateway and CIAM Integration: Token Validation, Consent, and Identity Context](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ciam-integration)[88. Elastic API Gateway Scaling: Signals, Capacity, and Safe Scale-In](https://api7.ai/learning-center/api-gateway-guide/elastic-api-gateway-scaling)[89. Multi-Tenant SaaS API Gateways: Isolation, Quotas, and Tenant-Aware Policy](https://api7.ai/learning-center/api-gateway-guide/multi-tenant-saas-api-gateway-policy)[90. API Gateway DevSecOps: Policy as Code, Testing, and Runtime Feedback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-devsecops)

10 Platform Engineering and Planning

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=platform-governance)[91. API Discovery and Documentation at the Gateway: Inventory, Ownership, and Drift](https://api7.ai/learning-center/api-gateway-guide/api-discovery-documentation-gateway)[92. WebAssembly Plugins for API Gateways: Portability, Isolation, and APISIX Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-webassembly-plugins)[93. Monitoring API Gateways with Prometheus and Grafana: Metrics, Dashboards, and Alerts](https://api7.ai/learning-center/api-gateway-guide/monitor-api-gateways-prometheus-grafana)[94. API Gateway Reliability Engineering: SLOs, Failure Isolation, and Incident Response](https://api7.ai/learning-center/api-gateway-guide/api-gateway-reliability-engineering)[95. API Gateways for Edge AI Inference: Routing, Constraints, and Observability](https://api7.ai/learning-center/api-gateway-guide/api-gateways-edge-ai-inference)[96. Reducing Cloud Cost with API Gateway Controls: Measurement Before Optimization](https://api7.ai/learning-center/api-gateway-guide/reducing-cloud-cost-api-gateway-controls)[97. Improving API Developer Experience with Gateway Standards and Self-Service](https://api7.ai/learning-center/api-gateway-guide/improving-api-developer-experience-gateway-self-service)[98. API Gateway Change Management: Compatibility, Progressive Delivery, and Rollback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-change-management-rollback)[99. Operating Serverless APIs: Cold Starts, Idempotency, and Asynchronous Handoffs](https://api7.ai/learning-center/api-gateway-guide/operating-serverless-apis-gateway)[100. Planning an API Gateway Roadmap for the Next Five Years](https://api7.ai/learning-center/api-gateway-guide/planning-api-gateway-roadmap-next-five-years)

[API Gateway Guide](https://api7.ai/learning-center/api-gateway-guide)[Platform Engineering and Planning](https://api7.ai/learning-center/api-gateway-guide?topic=platform-governance)

On this page

On this page

- [Key Takeaways](#key-takeaways)

- [Design the Monitoring Path](#design-the-monitoring-path)

- [Configure the APISIX 3.19 Exporter](#configure-the-apisix-319-exporter)

- [Scrape from Prometheus](#scrape-from-prometheus)

- [Choose Metrics That Answer Operating Questions](#choose-metrics-that-answer-operating-questions)

- [Control Cardinality Before It Controls You](#control-cardinality-before-it-controls-you)

- [Build Recording Queries First](#build-recording-queries-first)

- [Design Grafana Dashboards for Questions](#design-grafana-dashboards-for-questions)

- [Alert on User Pain and Required Action](#alert-on-user-pain-and-required-action)

- [Validate the Failure Paths](#validate-the-failure-paths)

- [Monitoring Checklist](#monitoring-checklist)

- [FAQ](#faq)

  - [Should the APISIX metrics endpoint be public?](#should-the-apisix-metrics-endpoint-be-public)
  - [Should dashboards group by Route name or Route ID?](#should-dashboards-group-by-route-name-or-route-id)
  - [Can Prometheus metrics replace gateway logs?](#can-prometheus-metrics-replace-gateway-logs)

- [Next Steps](#next-steps)

Article 93

# Monitoring API Gateways with Prometheus and Grafana: Metrics, Dashboards, and Alerts

API7.ai

October 29, 2026

Prometheus and Grafana can turn an API gateway into a useful observation point for request rate, client-visible errors, latency, saturation, and upstream health. A dependable setup does more than expose `/metrics`: it keeps the scrape path private, controls label cardinality, separates gateway-generated responses from upstream responses, and connects dashboards and alerts to user-visible service objectives.

Metrics show aggregate behavior. They do not prove that an application returned the correct result, identify a caller unless authentication already did so, or replace logs and traces during an investigation.

## Key Takeaways

- Scrape a dedicated, private metrics listener instead of publishing operational data on the public API listener by default.
- Start with rate, errors, duration, saturation, and upstream health; add labels only for an operational question.
- Use stable Route and Service identifiers, and treat names as mutable display data.
- Separate client-visible status, upstream status, and the component that generated the response.
- Build Grafana dashboards for investigation and alerts for action; dashboard variables are not available when Grafana evaluates alert rules.
- Test missing metrics, scraper failure, restarts, low traffic, and partial fleet loss before trusting an alert.

## Design the Monitoring Path

```
flowchart LR
    C[Clients] --> G[APISIX data plane]
    G --> U[Upstream services]
    P[Prometheus on operator network] -->|scrape private endpoint| E[APISIX metrics exporter]
    E -. local gateway metrics .-> G
    P --> R[Recording and alert rules]
    P --> D[Grafana data source]
    D --> B[Dashboards]
    R --> A[Alertmanager or Grafana Alerting]
    A --> O[On-call owner and runbook]
```

The scrape path is an operational control plane. It can reveal Route IDs, Service IDs, upstream addresses, Consumers, hosts, traffic volumes, and failure patterns. Put it on a restricted network, authenticate or tunnel where the environment requires it, and limit who can query Prometheus and Grafana.

Do not infer end-user identity from a `consumer`, address, or model label alone. The label describes what APISIX associated with a request; the authentication method and trusted proxy configuration determine whether that association is authoritative.

## Configure the APISIX 3.19 Exporter

Apache APISIX 3.19.0's [`prometheus` plugin](https://apisix.apache.org/docs/apisix/3.19/plugins/prometheus/) exports a text-format metrics endpoint. Its default static configuration enables a dedicated exporter on `127.0.0.1:9091` at `/apisix/prometheus/metrics`.

This `config.yaml` excerpt makes that choice explicit and binds the exporter to a private address placeholder:

```
plugin_attr:
  prometheus:
    export_uri: /apisix/prometheus/metrics
    metric_prefix: apisix_
    enable_export_server: true
    export_addr:
      ip: 10.0.8.21
      port: 9091
```

Replace the address with a real private interface and enforce network policy before reloading APISIX. With `enable_export_server: true`—the default—the dedicated exporter listens at `export_addr`. With `false`, that exporter does not listen. APISIX documents an alternative using `public-api` on the normal request listener, but that places metrics on the public API path and adds request-path exposure and overhead; use it only with an explicit access design.

Enable the plugin on the intended Route, Service, or Global Rule according to the required scope. This Route fragment is illustrative:

```
{
  "id": "orders-read",
  "uri": "/orders/*",
  "plugins": {
    "prometheus": {
      "prefer_name": false
    }
  },
  "upstream_id": "orders-read"
}
```

`prefer_name` defaults to `false`: when the relevant Route or Service exists, APISIX selects its ID for that label. With `true`, APISIX selects its configured name, so provide names before depending on this path; do not assume that a missing name automatically falls back to an ID. These labels also have absence paths. An unmatched request has an empty Route label, and for the built-in HTTP status, latency, and bandwidth metrics, APISIX uses the configured Route host as the Service label when no Service exists. The example above has an `upstream_id` but no Service, so its Service label is not a Service ID or name. IDs are usually safer for joins and history because display names can change; names can improve readability only when the organization governs their uniqueness and lifecycle. Neither Boolean branch changes authentication or makes a label trusted identity.

## Scrape from Prometheus

The following Prometheus fragment is runnable after replacing the target and making the private address reachable from Prometheus:

```
scrape_configs:
  - job_name: apisix-gateway
    metrics_path: /apisix/prometheus/metrics
    static_configs:
      - targets:
          - 10.0.8.21:9091
        labels:
          environment: production
          gateway_cluster: edge-eu-1
```

Use service discovery rather than a static target for an elastic fleet, but keep `environment` and `gateway_cluster` values bounded. Verify `up` for every expected target and alert on a missing fraction of the fleet; one healthy exporter does not prove that all gateway instances are being scraped.

## Choose Metrics That Answer Operating Questions

APISIX 3.19 provides, among other metrics:

| Question                                     | APISIX metric or label                                                        |
| -------------------------------------------- | ----------------------------------------------------------------------------- |
| What status did the client receive?          | `apisix_http_status` and `code`                                               |
| Who generated the response?                  | `response_source`: `apisix`, `nginx`, or `upstream`                           |
| How long did the request take?               | `apisix_http_latency{type="request"}`                                         |
| How much time was upstream waiting?          | `apisix_http_latency{type="upstream"}`                                        |
| What was gateway overhead?                   | `apisix_http_latency{type="apisix"}`                                          |
| Are active connections approaching capacity? | `apisix_nginx_http_current_connections`                                       |
| Is the configuration store reachable?        | `apisix_etcd_reachable`                                                       |
| Which checked upstream nodes are healthy?    | `apisix_upstream_status`                                                      |
| Is metrics storage running out of space?     | `apisix_shared_dict_capacity_bytes` and `apisix_shared_dict_free_space_bytes` |

Client-visible status can differ from upstream status after plugins and proxy error handling. A gateway-generated `429`, an NGINX `502`, and an upstream `503` require different owners even if all appear as unsuccessful requests.

## Control Cardinality Before It Controls You

[Prometheus instrumentation guidance](https://prometheus.io/docs/practices/instrumentation/) warns that every label set creates another time series with memory, CPU, disk, and network cost. Never use request IDs, raw URLs, user IDs, email addresses, tokens, or other unbounded values as metric labels.

Before adding a label, document:

1. the operational question it answers;
2. the maximum number of values per environment;
3. who owns stale values;
4. the expected series multiplication;
5. whether logs or traces are a better tool.

APISIX 3.19 Plugin Metadata can collapse selected built-in labels to an empty string while preserving the metric schema. Use that control when per-node, per-consumer, or per-model detail is not required. Do not add `upstream_addr` merely because it is available; elastic node churn can create long-lived series.

## Build Recording Queries First

These PromQL expressions are query examples, not universal alert thresholds. Validate them against the labels present in your deployment.

Client-visible 5xx ratio over five minutes:

```
sum(rate(apisix_http_status{code=~"5.."}[5m]))
/
sum(rate(apisix_http_status[5m]))
```

Per-Route p95 request latency in milliseconds:

```
histogram_quantile(
  0.95,
  sum by (le, route) (
    rate(apisix_http_latency_bucket{type="request"}[5m])
  )
)
```

Fraction of free space in APISIX's `prometheus-metrics` shared dictionary:

```
apisix_shared_dict_free_space_bytes{name="prometheus-metrics"}
/
apisix_shared_dict_capacity_bytes{name="prometheus-metrics"}
```

Record expensive or frequently reused expressions. Keep units in panel titles and alert annotations: APISIX HTTP latency histograms are in milliseconds, while many application and Prometheus ecosystem duration metrics use seconds.

## Design Grafana Dashboards for Questions

Grafana's [Prometheus data source](https://grafana.com/docs/grafana/latest/datasources/prometheus/) supports PromQL, alerting, recording rules, and exemplars. A useful gateway dashboard has layers:

- **Service overview:** request rate, success/error ratio, p50/p95/p99 latency, rejected traffic, and SLO budget consumption.
- **Gateway diagnosis:** response source, APISIX versus upstream latency, active connections, shared-dictionary capacity, config-store reachability, and fleet scrape coverage.
- **Upstream diagnosis:** health-check status, per-Service latency/error change, and node distribution where cardinality is bounded.
- **Change context:** deployment, configuration revision, and incident annotations from a separate trusted source.

Store dashboard and data-source provisioning in version control. Grafana permits UI edits to provisioned dashboards, but those edits are not automatically written back to the provisioning source. Decide whether Git or the UI is authoritative to avoid silent drift.

## Alert on User Pain and Required Action

[Prometheus alerting guidance](https://prometheus.io/docs/practices/alerting/) recommends symptom-based alerts and avoiding pages with no action. Page on sustained user-visible error or latency budget burn; route capacity and component signals to investigation or ticket workflows unless they predict imminent user harm.

Each alert should include:

- the affected service, environment, and gateway cluster;
- the user-visible symptom and observation window;
- the SLO or capacity threshold being consumed;
- a dashboard link that does not depend on unresolved variables;
- a runbook and owning team;
- suppression and recovery expectations.

Grafana evaluates alert rules on the backend without dashboard variable context. A panel query using `$route` or `$cluster` must become a rule with explicit label matchers or a recording rule. Test the rule independently from the dashboard.

## Validate the Failure Paths

Before production paging, test:

1. Prometheus cannot reach one gateway exporter;
2. the entire scrape job disappears;
3. APISIX restarts and counters reset;
4. one Route has no traffic and therefore no series yet;
5. the upstream returns errors while APISIX still answers other Routes;
6. APISIX generates a rejection before proxying;
7. label cardinality grows during an autoscaling or model rollout;
8. Grafana is unavailable while Prometheus alert evaluation continues, or vice versa.

Use `absent()` and fleet coverage carefully: an absent per-Route series can mean no traffic, disabled instrumentation, configuration drift, or a failed scrape. Corroborate before paging.

## Monitoring Checklist

- Is the metrics endpoint reachable only from the intended operator network?
- Is `enable_export_server` explicitly understood for both `true` and `false`?
- Are Route/Service IDs or names chosen deliberately through `prefer_name`?
- Are unbounded and unnecessary labels prohibited or collapsed?
- Can dashboards separate APISIX, NGINX, and upstream response sources?
- Are latency units and histogram buckets appropriate for the SLO?
- Are alerts evaluated without dashboard-only variables?
- Does every page identify user impact, owner, runbook, and action?
- Are scrape loss, missing series, counter reset, and partial fleet failure tested?
- Are logs and traces available for questions metrics cannot answer?

## FAQ

### Should the APISIX metrics endpoint be public?

Usually no. It exposes operational detail and can add work to the public request path. Prefer the dedicated exporter on a private operator network.

### Should dashboards group by Route name or Route ID?

IDs are the APISIX default and are normally more stable. Use names only when their uniqueness and rename lifecycle are governed. Keep a separate inventory to map IDs to human ownership.

### Can Prometheus metrics replace gateway logs?

No. Metrics reveal aggregate trends. Logs and traces provide request-level context, while application checks are needed for semantic correctness.

## Next Steps

Use these signals in [API gateway reliability engineering](https://api7.ai/learning-center/api-gateway-guide/api-gateway-reliability-engineering), correlate incidents through [gateway log analytics](https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-analytics), and keep long-term records within the [log storage architecture](https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-storage).

[Previous article**WebAssembly Plugins for API Gateways: Portability, Isolation, and APISIX Boundaries**](https://api7.ai/learning-center/api-gateway-guide/api-gateway-webassembly-plugins)[Next article**API Gateway Reliability Engineering: SLOs, Failure Isolation, and Incident Response**](https://api7.ai/learning-center/api-gateway-guide/api-gateway-reliability-engineering)

On this page

- [Key Takeaways](#key-takeaways)

- [Design the Monitoring Path](#design-the-monitoring-path)

- [Configure the APISIX 3.19 Exporter](#configure-the-apisix-319-exporter)

- [Scrape from Prometheus](#scrape-from-prometheus)

- [Choose Metrics That Answer Operating Questions](#choose-metrics-that-answer-operating-questions)

- [Control Cardinality Before It Controls You](#control-cardinality-before-it-controls-you)

- [Build Recording Queries First](#build-recording-queries-first)

- [Design Grafana Dashboards for Questions](#design-grafana-dashboards-for-questions)

- [Alert on User Pain and Required Action](#alert-on-user-pain-and-required-action)

- [Validate the Failure Paths](#validate-the-failure-paths)

- [Monitoring Checklist](#monitoring-checklist)

- [FAQ](#faq)

  - [Should the APISIX metrics endpoint be public?](#should-the-apisix-metrics-endpoint-be-public)
  - [Should dashboards group by Route name or Route ID?](#should-dashboards-group-by-route-name-or-route-id)
  - [Can Prometheus metrics replace gateway logs?](#can-prometheus-metrics-replace-gateway-logs)

- [Next Steps](#next-steps)
