MCP Observability: Monitor Every AI Agent Tool Call

Key Takeaways

• MCP observability must identify the caller, registered server, tool, outcome, and latency—not just the HTTP endpoint used by every request.
• AISIX AI Gateway emits one usage event for every tools/call attempt, including calls rejected by rate limits, guardrails, and applicable AISIX Cloud budgets.
• Usage events include MCP-specific server and tool fields but intentionally exclude tool arguments and results.
• MCP calls carry no model tokens, so token and cost fields remain zero; tool volume should be measured by call events rather than token dashboards.
• Prometheus labels distinguish MCP traffic from model traffic and show in-flight requests and usage-event emission.
• The current MCP path makes one upstream attempt, so latency data must not be interpreted as evidence of retries or failover.

An agent reports that “the tool failed.” That statement is not enough to operate an MCP platform.

Platform teams need to know which caller made the attempt, which registered server and tool were selected, whether the gateway rejected the call or the upstream failed, how long the caller waited, and whether a content policy intervened. Security teams need those answers without automatically copying sensitive tool arguments and results into every analytics record.

Traditional HTTP monitoring is necessary but insufficient because MCP tool calls share the same endpoint and method. Many requests arrive as POST /mcp; the meaningful operation is inside the JSON-RPC body. An MCP-aware gateway must turn that protocol context into structured telemetry.

AISIX AI Gateway sends MCP tool calls through the same telemetry pipeline as model traffic while tagging them with protocol-, server-, and tool-specific fields. Usage events provide per-attempt detail for analysis and audit. Prometheus metrics provide a live view of gateway activity and event-pipeline health. Together, they support incident response without making payload retention the default.

What Platform Teams Need to Measure

Useful MCP monitoring begins with questions rather than dashboards. A production platform should be able to answer:

• How many tool calls are being attempted, and how does that change over time?
• Which caller API keys generate the most traffic?
• Which registered MCP servers and tools are used most frequently?
• Which calls succeed, fail upstream, exceed a limit, or trigger a guardrail?
• How many MCP requests are active right now?
• How long does an upstream tool call take, and how long does the caller wait?
• Are usage events still reaching the configured sink?

These questions span three operational perspectives.

Platform reliability focuses on volume, concurrency, latency, failure rate, and telemetry delivery. It helps teams size the gateway and upstream services, find saturation, and detect regressions.

Security operations focuses on caller identity, unauthorized or policy-blocked attempts, unusual tool selection, and sudden guardrail activity. It helps distinguish a broken integration from a compromised or poorly constrained agent.

Product and capacity planning focuses on which tools are actually used. A tool catalog may contain hundreds of entries, but usage events show which capabilities deliver value and which expensive dependencies need tighter limits.

Do not collapse these views into one global request counter. A spike in runbooks__search is different from a spike in payments__create_refund, even if both travel through the same /mcp endpoint.

flowchart LR
    Agent["MCP client or agent"] --> AISIX["AISIX AI Gateway"]
    AISIX --> MCP["MCP server or OpenAPI tool"]
    AISIX --> Events["Usage event pipeline"]
    AISIX --> Metrics["Prometheus /metrics"]
    Events --> Analytics["Per-caller, server, and tool analysis"]
    Events --> Audit["Incident and policy review"]
    Metrics --> Alerts["Live health alerts"]
    Metrics --> Ops["Gateway operations dashboard"]

The separation is deliberate. Usage events carry the detailed dimensions needed for tool analysis. Prometheus metrics expose bounded operational series suitable for scraping and alerting.

How AISIX Represents MCP Usage

Every MCP tools/call attempt emits one AISIX usage event. The event is produced for a successful upstream call and for calls rejected before upstream routing by request limits, guardrails, or a covering AISIX Cloud budget. This ensures that policy enforcement does not disappear from analytics simply because no backend request was made.

The MCP-specific fields include:

An illustrative event projected into a JSON-capable sink could look like this:

{
  "inbound_protocol": "mcp",
  "mcp_server_name": "runbooks",
  "mcp_tool_name": "search",
  "api_key_id": "operations-agent-key-id",
  "status_code": 200,
  "upstream_latency_ms": 84,
  "downstream_latency_ms": 84,
  "guardrail_blocked": false,
  "request_id": "req-7c8d2f",
  "occurred_at": "2026-08-14T08:30:00Z"
}

The event contains identity, routing, outcome, and timing metadata. It does not contain the tool arguments or tool result. That boundary reduces the risk of copying customer data, credentials, search queries, or internal records into a general analytics pipeline. AISIX can inspect content in flight with guardrails without persisting that content in the usage event.

MCP also differs from model traffic in its unit of consumption. A tool call has no prompt or completion tokens, so token and cost fields remain zero. A dashboard that filters only for token spend will miss MCP activity. Use mcp_server_name, mcp_tool_name, and event counts for volume and attribution.

The latency fields require another careful interpretation. The current AISIX MCP implementation makes a single upstream attempt that spans the request, so upstream_latency_ms and downstream_latency_ms report the same duration. Their separate names keep the event shape consistent with other gateway traffic, where retries or additional processing can make the values differ. Do not infer retry overhead, failover, or gateway queue time from a difference that the current MCP path does not produce.

Usage events follow the same configured export paths as model usage. In AISIX Cloud, they also reach the control plane's usage sink. This shared pipeline lets teams correlate model and tool activity at the caller-key boundary without pretending that token and tool-call economics are identical.

Build MCP Dashboards and Alerts with Prometheus

AISIX exposes metrics through the gateway's dedicated Prometheus listener, normally at GET /metrics. MCP samples are labeled so operators can isolate them from other protocols.

Two documented metric families are the starting point:

Metric families appear after the first observation. A new gateway with no MCP tool call may not show an MCP-labeled series yet. Generate a controlled call before treating an absent series as a failure.

This PromQL expression shows the total number of MCP requests currently in flight across scraped gateway instances:

sum(aisix_proxy_in_flight_requests{endpoint="/mcp",inbound_protocol="mcp"})

This expression shows the rate at which MCP usage events are emitted over five minutes:

sum(
  rate(
    aisix_usage_events_emitted_total{
      handler="mcp",
      inbound_protocol="mcp"
    }[5m]
  )
)

Use Prometheus for live gateway and exporter health, then build the richer tool dashboard from usage events. A practical dashboard has two layers.

Gateway health panels:

• current MCP requests in flight;
• usage-event emission rate;
• the same metrics split by the infrastructure labels added by the deployment, such as gateway instance or environment;
• scrape health and exporter errors from the surrounding monitoring stack.

Usage-event panels:

• total tool-call attempts over time;
• top tools and servers by event count;
• success and non-success outcomes by server;
• latency by server and tool;
• guardrail-blocked calls by caller key;
• high-volume callers and sudden changes from their baseline.

Do not invent server or tool labels on a Prometheus series that does not document them. The AISIX usage event is the supported source for mcp_server_name and mcp_tool_name. Keeping high-cardinality names in the event pipeline also avoids turning every tool and caller combination into an unbounded metrics series.

Alerts should describe an actionable failure mode rather than a vague traffic change. Useful examples include:

• sustained in-flight requests above the capacity tested for a gateway deployment;
• a drop to zero usage-event emissions during a controlled health call;
• a usage-event query showing a sharp increase in non-success statuses for one server;
• repeated guardrail_blocked=true events from one caller key;
• a new caller becoming the dominant source of calls to a sensitive tool;
• latency for a normally fast tool crossing its operational objective.

Thresholds should come from measured baseline traffic and upstream capacity. A universal “more than ten calls” alert will be noisy for a search tool and dangerously high for a destructive administrative tool.

Investigate an MCP Incident Without Collecting Payloads

Consider an operations agent whose runbook searches begin failing. The investigation should move from the broad signal to the narrow request without requiring the search text itself.

sequenceDiagram
    participant Alert as Monitoring Alert
    participant Metrics as Prometheus
    participant Events as Usage Event Store
    participant Logs as Gateway and Upstream Logs
    participant Owner as Platform or Security Owner

    Alert->>Metrics: Confirm active MCP traffic and event emission
    Metrics-->>Owner: Gateway pipeline is healthy
    Owner->>Events: Filter by server, tool, status, and time
    Events-->>Owner: Identify caller key and request_id
    Owner->>Logs: Correlate request_id without querying payload content
    Logs-->>Owner: Policy rejection or upstream failure context
    Owner->>Owner: Adjust policy, limit, credential, or upstream service

Use the following workflow:

1. Confirm the data path. Check that the gateway exposes MCP metrics and the usage-event counter is increasing when a controlled call is made. This separates an observability outage from a tool outage.
2. Narrow the event set. Filter usage events by occurred_at, mcp_server_name, and mcp_tool_name. Compare successful and non-success statuses.
3. Identify the caller. Use api_key_id to determine whether the issue affects one application, one team, or every caller of the server.
4. Check policy evidence. guardrail_blocked=true directly identifies a content-policy decision. For rate or budget failures, correlate the status and request ID with gateway logs and the caller's configured limits or budget state.
5. Inspect timing. Use the recorded latency to detect a slow upstream. Remember that the current MCP path reports one upstream attempt; do not diagnose an undocumented retry loop.
6. Correlate safely. Use request_id across the gateway and upstream logs. Escalate to content capture only through an explicitly approved, access-controlled process when metadata is insufficient.
7. Apply the narrow fix. Rotate one upstream credential, adjust one server limit, tune one guardrail, or repair one backend. Avoid disabling the entire MCP control layer to resolve a local problem.

This workflow also helps distinguish a policy rollout from an upstream incident. A sudden cluster of guardrail blocks immediately after a rule change suggests tuning. A status shift isolated to one server, with unchanged policy signals, suggests an upstream or credential problem. A sharp volume increase from one key suggests a looping agent or compromised caller.

MCP Observability FAQ

Does AISIX store tool arguments or results in usage events?

No. Usage events contain caller, routing, outcome, timing, and policy metadata, but not tool arguments or results. Guardrails may inspect content in flight without adding that content to the event.

Why are token and cost values zero for MCP calls?

MCP tools do not consume model tokens at the gateway. Count tools/call usage events to measure tool volume; do not use a token-cost dashboard as a tool-call ledger.

How do I isolate a failure in one tool?

Filter events by time, mcp_server_name, and mcp_tool_name. Then use api_key_id to measure caller scope and request_id to correlate gateway and upstream logs. Prometheus confirms live gateway and event-pipeline health; the high-cardinality tool diagnosis belongs in usage events.

Operate MCP as a First-Class Traffic Type

MCP observability should preserve the semantics that HTTP access logs lose: caller, server, tool, policy outcome, and latency. AISIX usage events provide those dimensions for every attempted tool call, including pre-upstream rejections. Prometheus metrics show whether MCP requests are active and whether the usage pipeline is emitting events.

The privacy boundary is equally important. Usage telemetry records what operational teams need to manage the platform, but it does not copy tool arguments and results into each event. Guardrail inspection and content storage are different decisions and should remain separately governed.

Start by sending one controlled MCP tool call, verify the labeled metrics, locate its usage event by request ID, and build a dashboard from the documented fields. Then add alerts based on measured traffic and the risk of each tool category.

See the AISIX MCP Observability guide and Metrics Reference for the current telemetry contract. To complete the series, review how to turn REST APIs into MCP tools, enforce least-privilege MCP access, and secure MCP calls with guardrails and rate limits.