Grounded AI Agents: Verify Evidence, Verdicts, and Failure States

A grounded AI agent should never turn “the grounding check did not finish” into “the answer is supported.” Use a three-state delivery contract: supported, unsupported, or check unavailable. Pass the same authorized evidence to answer generation and evaluation, validate the evaluator's response, and expose the failure state to the application instead of collapsing it into a green badge.

This article shows the application and gateway boundaries for that contract. It does not claim that a gateway can prove a source document is true, that a grounding score is universal across evaluators, or that a content guardrail replaces document authorization.

Before implementing the contract, prepare an authorized retriever, a pinned evaluator and result schema, sanitized supported and unsupported fixtures, and a delivery policy for evaluator outages. Python 3.10 or later is required only for the standalone validator example; the architecture is language-independent.

What Grounding Can and Cannot Prove

Grounding evaluation asks whether an answer is supported by supplied evidence. It can help detect contradictions, unsupported additions, or missing support. Its result remains conditional on four inputs:

1. the documents retrieved for this request;
2. the exact answer being checked;
3. the evaluator and its version or configuration;
4. the threshold and decision logic used by the application.

Even a correct grounding verdict does not establish that the source document is current, authentic, authorized for the user, or factually correct. Those are separate provenance, access-control, and content-governance questions.

The practical boundary is:

authorized evidence -> answer generation -> grounding evaluation -> delivery decision
         |                    |                     |                    |
 document ACLs        model may add text      structured verdict     UI/API state

An AI gateway can authenticate the application, protect provider credentials, route the model request, apply configured content controls, and record gateway-visible usage. The application still owns retrieval authorization, evidence identifiers, the grounding verdict contract, and what the user sees.

Define a Three-State Contract

Use an explicit response shape rather than a Boolean that defaults to false or true when data is missing:

{
  "answer": "Employees may claim up to $500 for a home-office chair.",
  "grounding": {
    "state": "supported",
    "evaluator": "bedrock-grounding-policy",
    "score": 0.98,
    "threshold": 0.70,
    "evidence_ids": ["hr-policy-2026-04#home-office"],
    "checked_at": "2026-11-02T01:15:00Z"
  }
}

Allowed states:

unavailable is not a weaker form of supported. It means the application has no grounding decision.

Give Generation and Evaluation the Same Authorized Evidence

Build the authorized evidence set once, but do not send the entire authorization record to both external processors. Keep an application-side envelope for identity and audit data:

request_id: req-7f84
principal: employee-123
authorization_decision: allow
authorized_evidence_ids: hr-policy-2026-04#home-office

Derive a minimal generation payload from that envelope:

{
  "query": "How much can I claim for a home-office chair?",
  "documents": [
    {
      "id": "hr-policy-2026-04#home-office",
      "title": "Home Office Policy",
      "revision": "2026-04",
      "text": "A chair may be reimbursed up to USD 500 with manager approval."
    }
  ]
}

After generation, send the evaluator the answer plus the same approved query and document contents or revisions. “Same evidence” means the same authorized evidence set, not the same identity metadata. Keep the principal, entitlements, and authorization decision in the application unless a processor contract requires them and a privacy review approves that disclosure.

Do not let the evaluator silently see more or less evidence than the generator. If the answer was generated from documents A and B but evaluated against B and C, the verdict no longer describes the answer's actual evidence path.

Keep document IDs, revisions, and authorization decisions beside the request. A URL alone may change after the answer is produced. Avoid sending full documents to logs merely to preserve provenance; store stable identifiers and the minimum approved audit fields.

Keep Retrieval Authorization Before Model Access

The user must be allowed to retrieve a document before its content reaches the model or grounding evaluator. Gateway authentication for the model request does not replace document-level authorization.

Test these cases:

• the user can retrieve every document in the evidence bundle;
• one otherwise relevant document is forbidden and therefore absent;
• the user's entitlement changes between requests;
• a cached retrieval result was produced under another principal;
• the evaluator is a separate data processor with its own region and retention rules.

If an unauthorized document is retrieved, a perfectly grounded answer can still be a data leak.

Validate the Evaluator Response

Treat evaluator output as untrusted structured input. Check schema, types, score range, threshold presence, and the evaluator-specific meaning of negative findings.

The following Python 3.10+ function is intentionally provider-neutral. Adapt the upstream result into this small contract, then validate it before the UI sees it:

from dataclasses import dataclass
from typing import Any, Literal

State = Literal["supported", "unsupported", "unavailable"]

@dataclass(frozen=True)
class GroundingDecision:
    state: State
    score: float | None
    threshold: float | None
    reason: str

def decide_grounding(result: dict[str, Any] | None) -> GroundingDecision:
    if not result:
        return GroundingDecision("unavailable", None, None, "missing result")

    score = result.get("score")
    threshold = result.get("threshold")
    detected = result.get("unsupported_detected")

    if isinstance(score, bool) or not isinstance(score, (int, float)) or not 0 <= score <= 1:
        return GroundingDecision("unavailable", None, None, "invalid score")
    if isinstance(threshold, bool) or not isinstance(threshold, (int, float)) or not 0 <= threshold <= 1:
        return GroundingDecision("unavailable", None, None, "invalid threshold")
    if not isinstance(detected, bool):
        return GroundingDecision("unavailable", None, None, "invalid finding")

    if detected or score < threshold:
        return GroundingDecision("unsupported", float(score), float(threshold), "support rule failed")
    return GroundingDecision("supported", float(score), float(threshold), "support rule passed")

The function does not choose the right evaluator or threshold. It proves that missing and malformed results cannot become a successful verdict by accident.

Expected checks:

assert decide_grounding(None).state == "unavailable"
assert decide_grounding({"score": 0.9}).state == "unavailable"
assert decide_grounding({
    "score": 0.92, "threshold": 0.7, "unsupported_detected": False
}).state == "supported"
assert decide_grounding({
    "score": 0.62, "threshold": 0.7, "unsupported_detected": False
}).state == "unsupported"
assert decide_grounding({
    "score": 0.95, "threshold": 0.7, "unsupported_detected": True
}).state == "unsupported"

Run the adapter against saved, sanitized provider responses and pin those fixtures to the evaluator version. Do not describe fixture tests as a live model or gateway evaluation.

Decide Failure Behavior Before Production

For each failure, choose a delivery action:

Fail-open and fail-closed are product decisions, not universal labels. Failing open preserves availability but may deliver unchecked content. Failing closed preserves the grounding requirement but reduces availability. State which outcome the user sees and which event operators receive.

Separate Grounding from Content Guardrails

Grounding and content policy answer different questions:

• Grounding: Is this answer supported by the supplied evidence?
• PII policy: Does this content contain data that must be redacted or blocked?
• Prompt-injection policy: Does the input contain a pattern or classification that violates policy?
• Tool authorization: May this principal call this tool with this action?

A supported answer can still contain sensitive data. A policy-compliant answer can still be unsupported. Run the checks required by the application and keep their verdicts separate.

AISIX guardrails can inspect supported model traffic at configured input or output hooks. Their documented behavior, scope, streaming implications, and remote failure policy should be tested for the deployed release. They do not replace a RAG application's evidence-level grounding contract unless the configured guardrail explicitly performs that evaluation and returns a result the application validates.

For rollout mechanics, use AI gateway guardrails rather than assuming every guardrail blocks in the same phase.

Build the Acceptance Set

Use examples from your own domain, reviewed by the document owner:

1. Supported: every material answer claim appears in the evidence.
2. Contradicted: the answer conflicts with the evidence.
3. Incomplete: the answer adds one unsupported detail.
4. Irrelevant retrieval: documents are real but do not answer the question.
5. Conflicting sources: two authorized documents disagree.
6. Forbidden source: the best document exists but the principal cannot access it.
7. Evaluator timeout: no result before the delivery deadline.
8. Malformed result: missing score, threshold, or finding.
9. Changed revision: the evidence ID points to a newer policy version.

For each case, record the evidence IDs, answer, evaluator configuration, raw sanitized result, normalized state, and delivery action. Measure false-supported and false-unsupported outcomes separately; a single average score hides the more dangerous error.

Troubleshoot by Boundary

Definition of Done

A grounded agent is ready when:

• retrieval authorization is tested independently from model access;
• generation and evaluation use the same versioned evidence bundle;
• evaluator responses are schema-validated;
• supported, unsupported, and unavailable states remain distinct through the API and UI;
• content-policy and tool-authorization verdicts are not confused with grounding;
• timeout, missing-result, and malformed-result paths have explicit delivery behavior;
• the acceptance set includes contradictions, incomplete answers, authorization failures, and source changes;
• logs preserve request, evidence, evaluator, and decision identifiers without storing secrets or unnecessary document text.

LiteLLM's recent grounded-agent guide usefully emphasizes the same three-way distinction and notes that a grounding assessment does not verify the source itself. The implementation in your stack may differ, but that boundary should not.

Use AISIX AI Gateway for governed model access and gateway-visible policy, then keep evidence authorization and the final grounding decision explicit in the application that delivers the answer.