Decision Models Through an AI Gateway: Test the API Contract

Before routing application decisions through an AI gateway, test the contract you will actually consume: the request shape, answer order and types, refusal handling, usage, and transport failures. A 200 response means the HTTP request succeeded; it does not prove the answers are usable or correct.

This tutorial targets AISIX 1.7.0, a direct TypeSafe decision model, and the client-facing POST /v1/decisions format. The gateway examples follow the official decision-model guide: AISIX converts that request to the upstream's System One format. The application-side response checker was run against synthetic valid and invalid responses. It checks the application profile and refusal handling; it does not measure a live model, gateway throughput, or provider accuracy.

AISIX 1.7.0 does not run model or caller-key guardrails on either decision endpoint. Put required input validation, sensitive-data handling, and application authorization outside this call. Do not assume an existing chat guardrail also screens decision requests.

Start With a Concrete Decision Task

Suppose a support application needs two judgments:

1. Which queue should receive a ticket: billing or engineering?
2. How likely is the ticket to describe a service outage?

The application can use the first answer to suggest a queue and the second to request human escalation. Neither answer grants access, issues a refund, or overrides an authorization rule.

Our existing Decisions API architecture article explains that policy boundary. This tutorial adds the implementation acceptance path.

Recent LiteLLM decision-model reranking coverage shows interest in using structured judgments inside applications. That is a discovery signal, not evidence that AISIX implements JevRank or converts its rerank interface. The integration procedure below follows AISIX's documented decision endpoint.

Pin the Endpoint, Provider, and Model

Record the gateway image version/digest, deployment mode, provider, upstream model revision where available, model alias, and question-set version. A moving alias such as jev-latest is not a fixed model snapshot; record that limitation if the provider does not expose a revision.

AISIX exposes two distinct client contracts:

The gateway chooses the upstream format from the model's provider key. In 1.7.0 the decision providers are TypeSafe, Microsoft Foundry through azure, OpenRouter, and OpenAI. Both endpoints are non-streaming. A chat model on an arbitrary provider is not automatically eligible.

For this tutorial, prepare an existing AISIX Enterprise/Cloud 1.7.0 environment:

1. Create a Provider Key with provider typesafe, a valid provider credential, and access to the test environment.
2. Create a direct model with alias decision-check, that provider key, and an upstream model ID your TypeSafe account supports.
3. Grant decision-check to a dedicated caller key. Do not give the application the provider key or an admin token.
4. Configure the matching provider/model price if you need recorded spend or managed budget tests. Token usage alone does not establish priced spend.
5. Keep chat-based background health checks off for this decision-only model. Use the endpoint-specific probes below.

Use the AISIX decision-model setup guide for the deployment-specific API or OSS resources equivalent. OSS uses the same proxy endpoints, but managed budgets and organization-level cost reporting require Enterprise or Cloud.

Do not use the chat-only dashboard playground as proof that a decision model works.

Send a Small, Versioned Request

Save this as decision-request.json. It adapts the official guide’s Decisions-format choice example, adds a documented predicate question, and uses the application alias configured above. The text is synthetic and contains no customer data.

{
  "model": "decision-check",
  "input": "The API returns 500 on every call; engineering should look.",
  "questions": [
    {
      "type": "choice",
      "name": "team",
      "instructions": "Which team should handle this ticket?",
      "choices": [
        {"value": "billing", "description": "Charges and invoices"},
        {"value": "engineering", "description": "Bugs and outages"}
      ]
    },
    {
      "type": "predicate",
      "name": "is_outage",
      "instructions": "Does this describe a service outage?"
    }
  ]
}

Use unique, stable question names and keep their order in the application's versioned profile. AISIX has rules for absent or repeated names during conversion, but the application need not depend on that ambiguity.

Set AISIX_PROXY to the gateway origin without /v1 or a trailing slash, and inject AISIX_API_KEY through your normal credential mechanism. This example passes the bearer header through curl's standard input rather than writing its value in the command arguments:

umask 077
status=$(
  printf 'header = "Authorization: Bearer %s"\n' "${AISIX_API_KEY}" |
    curl -sS --config - \
      -o decision-response.json -w '%{http_code}' \
      -H "Content-Type: application/json" \
      --data-binary @decision-request.json \
      "${AISIX_PROXY}/v1/decisions"
)
test "$status" = "200" || {
  echo "Decision call failed with HTTP $status; inspect a sanitized error."
  exit 1
}

Keep TLS verification enabled. Store response captures under restricted permissions and sanitize them before sharing; even a decision response can reveal sensitive input-derived information.

Do not compare model probabilities for exact equality between direct and gateway calls. Start by comparing schema, identity, question mapping, refusal semantics, and usage presence. Use a separate representative labeled dataset to evaluate answer quality.

Check the Application Contract

The following synthetic response fixture illustrates a valid answer. Its probabilities and token counts are not measured results:

{
  "model": "decision-check",
  "answers": [
    {
      "type": "choice",
      "name": "team",
      "choice": "engineering",
      "confidence": 0.97,
      "probabilities": [
        {"value": "billing", "probability": 0.03},
        {"value": "engineering", "probability": 0.97}
      ]
    },
    {"type": "predicate", "name": "is_outage", "probability": 0.92}
  ],
  "usage": {"input_tokens": 381, "output_tokens": 62, "total_tokens": 443}
}

Save it as decision-fixture.json. Save this checker as decision_check.py:

import json
import math
import sys

def number(value):
    return (isinstance(value, (int, float))
            and not isinstance(value, bool) and math.isfinite(value))

def probability(value):
    return number(value) and 0 <= value <= 1

def check(body):
    if not isinstance(body, dict) or body.get("model") != "decision-check":
        raise ValueError("Unexpected response model or envelope")
    answers = body.get("answers")
    if not isinstance(answers, list) or len(answers) != 2:
        raise ValueError("Expected two ordered answers")
    review = False
    for answer, name, kind in zip(
        answers, ("team", "is_outage"), ("choice", "predicate")
    ):
        if not isinstance(answer, dict) or answer.get("name") != name:
            raise ValueError("Question identity or order changed")
        if answer.get("type") == "refusal":
            review = True
            continue
        if answer.get("type") != kind:
            raise ValueError("Unexpected answer type")
        if kind == "predicate":
            if not probability(answer.get("probability")):
                raise ValueError("Invalid outage probability")
        else:
            allowed = {"billing", "engineering"}
            if (not isinstance(answer.get("choice"), str)
                    or answer["choice"] not in allowed):
                raise ValueError("Unknown team")
            if not probability(answer.get("confidence")):
                raise ValueError("Invalid confidence")
            values = answer.get("probabilities")
            if not isinstance(values, list) or len(values) != 2:
                raise ValueError("Expected both team probabilities")
            seen = set()
            total = 0.0
            for item in values:
                if not isinstance(item, dict):
                    raise ValueError("Invalid probability entry")
                value = item.get("value")
                if not isinstance(value, str) or value not in allowed:
                    raise ValueError("Unknown probability choice")
                if value in seen or not probability(item.get("probability")):
                    raise ValueError("Repeated choice or invalid probability")
                seen.add(value)
                total += item["probability"]
            if abs(total - 1.0) > 0.001:
                raise ValueError("Team probabilities do not sum to one")
    usage = body.get("usage")
    if not isinstance(usage, dict):
        raise ValueError("Missing usage")
    for field in ("input_tokens", "output_tokens", "total_tokens"):
        value = usage.get(field)
        if type(value) is not int or value < 0:
            raise ValueError("Missing or invalid usage counter")
    return "REVIEW" if review else "PASS"

if __name__ == "__main__":
    try:
        with open(sys.argv[1], encoding="utf-8") as source:
            outcome = check(json.load(source))
    except (ValueError, OSError, IndexError):
        print("FAIL: response violates the application contract")
        raise SystemExit(1)
    print(outcome)
    raise SystemExit(2 if outcome == "REVIEW" else 0)

Run the fixture first:

python3 decision_check.py decision-fixture.json
# Expected: PASS, exit 0

Then run the captured gateway response through the same checker. Exit 0 means it satisfies this application's structural profile. Exit 2 means a refusal requires the application's review path. Exit 1 means the response cannot be consumed under this profile.

This is deliberately narrower than the full API. It accepts exactly two uniquely named questions with these types and requires the displayed usage fields. Its team probabilities must sum to one within 0.001. Change the profile and its fixtures together if your application has different needs. It does not certify answer correctness, probability calibration, or completeness of every provider's metadata.

Exercise Refusals and Failures Deliberately

A refusal is a valid decision outcome, not a probability of zero. For a converted response, AISIX represents an absent answer or an answer of the wrong question type as a refusal on /v1/decisions. On /v1/systemone, which has no refusal type, the corresponding answer is omitted. Application code must handle both shapes if it uses both endpoints.

Change the fixture's first answer to {"type":"refusal","name":"team"} and rerun: expect REVIEW and exit 2. Removing an answer, swapping its position, using an unknown team, or setting a probability above one should produce FAIL.

Add these deployment-level tests using a controlled mock upstream or a provider-approved sandbox:

The official malformed-answer documentation explicitly distinguishes versions: AISIX 1.7.0 or earlier fails a routing model over to its next target on a malformed converted answer; later behavior does not retry that already-served answer. Do not apply the later no-retry behavior to a 1.7.0 rollout. Start with the direct model described here, then test attempt counts and cost in your deployment before adding routing.

When the upstream and client formats match, AISIX passes through fields it does not know, apart from model-name rewriting and configured body overrides. That path is not a universal schema-validation guarantee. Keep the application checker even if one converted response was validated by the gateway.

Record Usage Without Calling It a Quality Result

For decision traffic, usage-event operation identifies the client endpoint: decisions or systemone. Check the resolved provider/model, token counts, retries, and recorded cost alongside the application outcome.

Configure a pricing override where the selected provider/model has no catalog price. AI cost attribution and budgets are useful after the usage is priced, but they cannot tell you whether the chosen support queue was correct.

Keep three separate acceptance results:

1. Transport and policy: the intended caller can access the intended model, with known failure handling.
2. Contract: the response has the fields and fallback states the application consumes.
3. Task quality: representative labeled tickets meet your false-routing and escalation requirements.

The local fixture verifies the checker, not a live provider or gateway deployment. Run the live capture and failure matrix in your environment, record actual versions and observations, and only then approve that integration. No production accuracy, cost reduction, or latency improvement is claimed by this tutorial.

Start with decision-request.json, the fixture, and the checker. Link the resulting contract version to your application deployment so a provider or gateway change triggers the same acceptance tests again.