--- title: "MCP Observability: Monitor Every AI Agent Tool Call - API7.ai" description: "Learn how AISIX usage events and Prometheus metrics reveal MCP tool volume, latency, failures, limits, and guardrail blocks without storing sensitive payloads." url: "https://api7.ai/blog/mcp-observability-ai-agent-tool-calls" --- [Blog](https://api7.ai/blog) [Products](https://api7.ai/category/products)[Technology](https://api7.ai/category/technology)[Ecosystem](https://api7.ai/category/ecosystem)[Case Study](https://api7.ai/category/usercase)[Team & Community](https://api7.ai/category/community) Search ⌘ + K [Blog](https://api7.ai/blog) Technology # MCP Observability: Monitor Every AI Agent Tool Call [![Yilia Lin](https://github.com/Yilialinn.png)](https://github.com/Yilialinn) [Yilia Lin](https://github.com/Yilialinn) August 14, 2026 [Technology](https://api7.ai/category/technology) ## Key Takeaways - MCP observability must identify the caller, registered server, tool, outcome, and latency—not just the HTTP endpoint used by every request. - AISIX AI Gateway emits one usage event for every `tools/call` attempt, including calls rejected by rate limits, guardrails, and applicable AISIX Cloud budgets. - Usage events include MCP-specific server and tool fields but intentionally exclude tool arguments and results. - MCP calls carry no model tokens, so token and cost fields remain zero; tool volume should be measured by call events rather than token dashboards. - Prometheus labels distinguish MCP traffic from model traffic and show in-flight requests and usage-event emission. - The current MCP path makes one upstream attempt, so latency data must not be interpreted as evidence of retries or failover. An agent reports that “the tool failed.” That statement is not enough to operate an MCP platform. Platform teams need to know which caller made the attempt, which registered server and tool were selected, whether the gateway rejected the call or the upstream failed, how long the caller waited, and whether a content policy intervened. Security teams need those answers without automatically copying sensitive tool arguments and results into every analytics record. Traditional HTTP monitoring is necessary but insufficient because MCP tool calls share the same endpoint and method. Many requests arrive as `POST /mcp`; the meaningful operation is inside the JSON-RPC body. An MCP-aware gateway must turn that protocol context into structured telemetry. [AISIX AI Gateway](https://api7.ai/ai-gateway) sends MCP tool calls through the same telemetry pipeline as model traffic while tagging them with protocol-, server-, and tool-specific fields. Usage events provide per-attempt detail for analysis and audit. Prometheus metrics provide a live view of gateway activity and event-pipeline health. Together, they support incident response without making payload retention the default. The [AI Gateway Guide](https://api7.ai/learning-center/ai-gateway-guide) places this MCP telemetry model within the wider production architecture for models, agents, routing, security, rate limits, cost, and governance. ## What Platform Teams Need to Measure Useful MCP monitoring begins with questions rather than dashboards. A production platform should be able to answer: - How many tool calls are being attempted, and how does that change over time? - Which caller API keys generate the most traffic? - Which registered MCP servers and tools are used most frequently? - Which calls succeed, fail upstream, exceed a limit, or trigger a guardrail? - How many MCP requests are active right now? - How long does an upstream tool call take, and how long does the caller wait? - Are usage events still reaching the configured sink? These questions span three operational perspectives. **Platform reliability** focuses on volume, concurrency, latency, failure rate, and telemetry delivery. It helps teams size the gateway and upstream services, find saturation, and detect regressions. **Security operations** focuses on caller identity, unauthorized or policy-blocked attempts, unusual tool selection, and sudden guardrail activity. It helps distinguish a broken integration from a compromised or poorly constrained agent. **Product and capacity planning** focuses on which tools are actually used. A tool catalog may contain hundreds of entries, but usage events show which capabilities deliver value and which expensive dependencies need tighter limits. Do not collapse these views into one global request counter. A spike in `runbooks__search` is different from a spike in `payments__create_refund`, even if both travel through the same `/mcp` endpoint. ``` flowchart LR Agent["MCP client or agent"] --> AISIX["AISIX AI Gateway"] AISIX --> MCP["MCP server or OpenAPI tool"] AISIX --> Events["Usage event pipeline"] AISIX --> Metrics["Prometheus /metrics"] Events --> Analytics["Per-caller, server, and tool analysis"] Events --> Audit["Incident and policy review"] Metrics --> Alerts["Live health alerts"] Metrics --> Ops["Gateway operations dashboard"] ``` The separation is deliberate. Usage events carry the detailed dimensions needed for tool analysis. Prometheus metrics expose bounded operational series suitable for scraping and alerting. ## How AISIX Represents MCP Usage Every MCP `tools/call` attempt emits one AISIX usage event. The event is produced for a successful upstream call and for calls rejected before upstream routing by request limits, guardrails, or a covering AISIX Cloud budget. This ensures that policy enforcement does not disappear from analytics simply because no backend request was made. The MCP-specific fields include: | Field | Operational meaning | | ----------------------- | ---------------------------------------------------- | | `inbound_protocol` | Set to `mcp`, allowing separation from model traffic | | `mcp_server_name` | Registered server that owns the tool | | `mcp_tool_name` | Upstream tool name that was called | | `api_key_id` | Caller API key responsible for the attempt | | `status_code` | Outcome status recorded for the call | | `upstream_latency_ms` | Time spent on the upstream tool call | | `downstream_latency_ms` | Total time observed by the caller | | `guardrail_blocked` | `true` when input or output was blocked | | `request_id` | Correlation identifier for investigation | | `occurred_at` | Event timestamp | An illustrative event projected into a JSON-capable sink could look like this: ``` { "inbound_protocol": "mcp", "mcp_server_name": "runbooks", "mcp_tool_name": "search", "api_key_id": "operations-agent-key-id", "status_code": 200, "upstream_latency_ms": 84, "downstream_latency_ms": 84, "guardrail_blocked": false, "request_id": "req-7c8d2f", "occurred_at": "2026-08-14T08:30:00Z" } ``` The event contains identity, routing, outcome, and timing metadata. It does not contain the tool arguments or tool result. That boundary reduces the risk of copying customer data, credentials, search queries, or internal records into a general analytics pipeline. AISIX can inspect content in flight with guardrails without persisting that content in the usage event. MCP also differs from model traffic in its unit of consumption. A tool call has no prompt or completion tokens, so token and cost fields remain zero. A dashboard that filters only for token spend will miss MCP activity. Use `mcp_server_name`, `mcp_tool_name`, and event counts for volume and attribution. The latency fields require another careful interpretation. The current AISIX MCP implementation makes a single upstream attempt that spans the request, so `upstream_latency_ms` and `downstream_latency_ms` report the same duration. Their separate names keep the event shape consistent with other gateway traffic, where retries or additional processing can make the values differ. Do not infer retry overhead, failover, or gateway queue time from a difference that the current MCP path does not produce. Usage events follow the same configured export paths as model usage. In AISIX Cloud, they also reach the control plane's usage sink. This shared pipeline lets teams correlate model and tool activity at the caller-key boundary without pretending that token and tool-call economics are identical. ## Build MCP Dashboards and Alerts with Prometheus AISIX exposes metrics through the gateway's dedicated Prometheus listener, normally at `GET /metrics`. MCP samples are labeled so operators can isolate them from other protocols. Two documented metric families are the starting point: | Goal | Metric and filter | | -------------------------------- | ------------------------------------------------------------------------ | | Track active MCP requests | `aisix_proxy_in_flight_requests{endpoint="/mcp",inbound_protocol="mcp"}` | | Confirm MCP usage-event emission | `aisix_usage_events_emitted_total{handler="mcp",inbound_protocol="mcp"}` | Metric families appear after the first observation. A new gateway with no MCP tool call may not show an MCP-labeled series yet. Generate a controlled call before treating an absent series as a failure. This PromQL expression shows the total number of MCP requests currently in flight across scraped gateway instances: ``` sum(aisix_proxy_in_flight_requests{endpoint="/mcp",inbound_protocol="mcp"}) ``` This expression shows the rate at which MCP usage events are emitted over five minutes: ``` sum( rate( aisix_usage_events_emitted_total{ handler="mcp", inbound_protocol="mcp" }[5m] ) ) ``` Use Prometheus for live gateway and exporter health, then build the richer tool dashboard from usage events. A practical dashboard has two layers. **Gateway health panels:** - current MCP requests in flight; - usage-event emission rate; - the same metrics split by the infrastructure labels added by the deployment, such as gateway instance or environment; - scrape health and exporter errors from the surrounding monitoring stack. **Usage-event panels:** - total tool-call attempts over time; - top tools and servers by event count; - success and non-success outcomes by server; - latency by server and tool; - guardrail-blocked calls by caller key; - high-volume callers and sudden changes from their baseline. Do not invent server or tool labels on a Prometheus series that does not document them. The AISIX usage event is the supported source for `mcp_server_name` and `mcp_tool_name`. Keeping high-cardinality names in the event pipeline also avoids turning every tool and caller combination into an unbounded metrics series. Alerts should describe an actionable failure mode rather than a vague traffic change. Useful examples include: - sustained in-flight requests above the capacity tested for a gateway deployment; - a drop to zero usage-event emissions during a controlled health call; - a usage-event query showing a sharp increase in non-success statuses for one server; - repeated `guardrail_blocked=true` events from one caller key; - a new caller becoming the dominant source of calls to a sensitive tool; - latency for a normally fast tool crossing its operational objective. Thresholds should come from measured baseline traffic and upstream capacity. A universal “more than ten calls” alert will be noisy for a search tool and dangerously high for a destructive administrative tool. ## Investigate an MCP Incident Without Collecting Payloads Consider an operations agent whose runbook searches begin failing. The investigation should move from the broad signal to the narrow request without requiring the search text itself. ``` sequenceDiagram participant Alert as Monitoring Alert participant Metrics as Prometheus participant Events as Usage Event Store participant Logs as Gateway and Upstream Logs participant Owner as Platform or Security Owner Alert->>Metrics: Confirm active MCP traffic and event emission Metrics-->>Owner: Gateway pipeline is healthy Owner->>Events: Filter by server, tool, status, and time Events-->>Owner: Identify caller key and request_id Owner->>Logs: Correlate request_id without querying payload content Logs-->>Owner: Policy rejection or upstream failure context Owner->>Owner: Adjust policy, limit, credential, or upstream service ``` Use the following workflow: 1. **Confirm the data path.** Check that the gateway exposes MCP metrics and the usage-event counter is increasing when a controlled call is made. This separates an observability outage from a tool outage. 2. **Narrow the event set.** Filter usage events by `occurred_at`, `mcp_server_name`, and `mcp_tool_name`. Compare successful and non-success statuses. 3. **Identify the caller.** Use `api_key_id` to determine whether the issue affects one application, one team, or every caller of the server. 4. **Check policy evidence.** `guardrail_blocked=true` directly identifies a content-policy decision. For rate or budget failures, correlate the status and request ID with gateway logs and the caller's configured limits or budget state. 5. **Inspect timing.** Use the recorded latency to detect a slow upstream. Remember that the current MCP path reports one upstream attempt; do not diagnose an undocumented retry loop. 6. **Correlate safely.** Use `request_id` across the gateway and upstream logs. Escalate to content capture only through an explicitly approved, access-controlled process when metadata is insufficient. 7. **Apply the narrow fix.** Rotate one upstream credential, adjust one server limit, tune one guardrail, or repair one backend. Avoid disabling the entire MCP control layer to resolve a local problem. This workflow also helps distinguish a policy rollout from an upstream incident. A sudden cluster of guardrail blocks immediately after a rule change suggests tuning. A status shift isolated to one server, with unchanged policy signals, suggests an upstream or credential problem. A sharp volume increase from one key suggests a looping agent or compromised caller. ## MCP Observability FAQ ### Does AISIX store tool arguments or results in usage events? No. Usage events contain caller, routing, outcome, timing, and policy metadata, but not tool arguments or results. Guardrails may inspect content in flight without adding that content to the event. ### Why are token and cost values zero for MCP calls? MCP tools do not consume model tokens at the gateway. Count `tools/call` usage events to measure tool volume; do not use a token-cost dashboard as a tool-call ledger. ### How do I isolate a failure in one tool? Filter events by time, `mcp_server_name`, and `mcp_tool_name`. Then use `api_key_id` to measure caller scope and `request_id` to correlate gateway and upstream logs. Prometheus confirms live gateway and event-pipeline health; the high-cardinality tool diagnosis belongs in usage events. ## Operate MCP as a First-Class Traffic Type MCP observability should preserve the semantics that HTTP access logs lose: caller, server, tool, policy outcome, and latency. AISIX usage events provide those dimensions for every attempted tool call, including pre-upstream rejections. Prometheus metrics show whether MCP requests are active and whether the usage pipeline is emitting events. The privacy boundary is equally important. Usage telemetry records what operational teams need to manage the platform, but it does not copy tool arguments and results into each event. Guardrail inspection and content storage are different decisions and should remain separately governed. Start by sending one controlled MCP tool call, verify the labeled metrics, locate its usage event by request ID, and build a dashboard from the documented fields. Then add alerts based on measured traffic and the risk of each tool category. See the [AISIX MCP Observability](https://docs.api7.ai/ai-gateway/mcp-gateway/observability) guide and [Metrics Reference](https://docs.api7.ai/ai-gateway/reference/metrics) for the current telemetry contract. To complete the series, review how to [turn REST APIs into MCP tools](https://api7.ai/blog/openapi-to-mcp-secure-ai-tools), enforce [least-privilege MCP access](https://api7.ai/blog/mcp-access-control-least-privilege-ai-agents), and [secure MCP calls with guardrails and rate limits](https://api7.ai/blog/secure-mcp-tool-calls-guardrails-rate-limits). Tags: [MCP](https://api7.ai/tags/mcp) [Observability](https://api7.ai/tags/observability) [AI Agents](https://api7.ai/tags/ai-agents) [Prometheus](https://api7.ai/tags/prometheus) Share article link Share Related Posts [![What Is an MCP Registry? Discovery and Gateway Control](https://static.api7.ai/uploads/2026/09/15/PL8CU1Q1_what-is-mcp-registry.webp)](https://api7.ai/blog/what-is-mcp-registry) ## [What Is an MCP Registry? Discovery and Gateway Control](https://api7.ai/blog/what-is-mcp-registry) [![MCP Access Control: Least Privilege for AI Agents](https://static.api7.ai/uploads/2026/08/14/yhTEMszT_mcp-access-control-least-privilege-cover.svg)](https://api7.ai/blog/mcp-access-control-least-privilege-ai-agents) ## [MCP Access Control: Least Privilege for AI Agents](https://api7.ai/blog/mcp-access-control-least-privilege-ai-agents) Table of Contents - [Key Takeaways](#key-takeaways) - [What Platform Teams Need to Measure](#what-platform-teams-need-to-measure) - [How AISIX Represents MCP Usage](#how-aisix-represents-mcp-usage) - [Build MCP Dashboards and Alerts with Prometheus](#build-mcp-dashboards-and-alerts-with-prometheus) - [Investigate an MCP Incident Without Collecting Payloads](#investigate-an-mcp-incident-without-collecting-payloads) - [MCP Observability FAQ](#mcp-observability-faq) - [Does AISIX store tool arguments or results in usage events?](#does-aisix-store-tool-arguments-or-results-in-usage-events) - [Why are token and cost values zero for MCP calls?](#why-are-token-and-cost-values-zero-for-mcp-calls) - [How do I isolate a failure in one tool?](#how-do-i-isolate-a-failure-in-one-tool) - [Operate MCP as a First-Class Traffic Type](#operate-mcp-as-a-first-class-traffic-type) ## Related Posts [![LLM Monitoring with Prometheus: SLOs for AI Traffic](https://static.api7.ai/uploads/2026/10/13/vJoXpfa4_llm-monitoring-prometheus.webp)](https://api7.ai/blog/llm-monitoring-prometheus) Technology October 13, 2026 #### [LLM Monitoring with Prometheus: SLOs for AI Traffic](https://api7.ai/blog/llm-monitoring-prometheus) Build LLM monitoring with Prometheus using terminal request outcomes, streaming latency, TTFT, fallback, and policy signals from AISIX 1.6.0. [AI Gateway](https://api7.ai/tags/ai-gateway) [AISIX](https://api7.ai/tags/aisix) [Prometheus](https://api7.ai/tags/prometheus) [![MCP Security Best Practices: Auth, Tool Access, and Egress](https://static.api7.ai/uploads/2026/10/08/Nd9pj31A_mcp-security-best-practices.webp)](https://api7.ai/blog/mcp-security-best-practices) Technology October 8, 2026 #### [MCP Security Best Practices: Auth, Tool Access, and Egress](https://api7.ai/blog/mcp-security-best-practices) Secure MCP with resource-bound authentication, least-privilege tools, separate credentials, correctly timed guardrails, and egress controls. [MCP](https://api7.ai/tags/mcp) [AI Gateway](https://api7.ai/tags/ai-gateway) [AISIX](https://api7.ai/tags/aisix) [![Native Histograms for API Gateway SLOs: A Safe Migration Guide](https://static.api7.ai/uploads/2026/09/15/EUX6SgZk_native-histograms-api-gateway-slos.webp)](https://api7.ai/blog/native-histograms-api-gateway-slos) Technology September 15, 2026 #### [Native Histograms for API Gateway SLOs: A Safe Migration Guide](https://api7.ai/blog/native-histograms-api-gateway-slos) Learn how Prometheus native histograms improve API latency visibility and how to migrate gateway dashboards, alerts, and SLOs safely. [API Gateway](https://api7.ai/tags/api-gateway) [Observability](https://api7.ai/tags/observability) [Prometheus](https://api7.ai/tags/prometheus)