OpenAI-Compatible AI Gateway: Test the Contract

An OpenAI-compatible AI gateway can let applications keep a familiar SDK while the platform owns caller identity, model aliases, provider credentials, routing, and policy. That convenience is valuable, but a successful text response proves very little.

“Compatible” may describe one endpoint, one request shape, or one provider adapter. It does not automatically cover model discovery, streaming order, tool-call fragments, structured output, usage fields, embeddings, audio, batch jobs, or the Responses API.

The safe approach is to treat compatibility as a versioned contract. Define the behavior each application requires, test it through the real gateway path, and publish a capability profile for every model alias.

Key Takeaways

OpenAI compatibility describes a supported API surface, not universal feature equivalence.
The client, gateway endpoint, provider adapter, and upstream model each own part of the contract.
Chat Completions success does not validate Responses, embeddings, media, batch, or provider-native features.
Tests should cover identity, discovery, fields, streams, tools, usage, errors, timeouts, and correlation.
Every primary and fallback target behind an alias must satisfy the alias's published capabilities.

Compatibility Has Four Layers

Teams often use “OpenAI-compatible” for different promises. A client may mean it can set a custom base URL. A model server may mean it accepts POST /v1/chat/completions. A gateway may expose OpenAI-shaped endpoints while translating requests to another provider.

These claims are related but not interchangeable.

Layer	Contract owner	What to verify
Client	SDK or application	Base URL, key, model, timeout, retry, parser behavior
Gateway	Caller-facing endpoint	Routes, fields, stream events, errors, usage, authorization
Adapter	Gateway-to-provider translation	Preserved fields and unsupported semantics
Upstream	Provider or model server	Modality, context, tool, output, and reliability behavior

The OpenAI API reference is a useful source contract, but model support still varies. A compatible service can implement a deliberate subset. The operational risk comes from treating an undocumented subset as a complete standard.

This is also why a private LLM gateway should be tested against the exact engine, route, and model version rather than trusted because it exposes an OpenAI-shaped URL.

Choose the Caller-Facing API Family

Start with the application contract, not the provider list.

Chat Completions

Chat Completions remains a common integration surface. Many clients can switch to a gateway by changing three values:

base_url = https://gateway.example.com/v1
api_key  = gateway-issued caller key
model    = support-chat-prod

The model value should be a gateway-facing alias, not an upstream credential or secret model identifier. The AISIX Chat Completions guide documents the caller-facing path and how it maps into the gateway.

“Broad client support” is not “all features.” A text-only alias should reject image content clearly. A model without tool support should not be placed behind an alias promised to agent workflows.

Responses

The Responses API has a different input and output model and may expose agent-oriented features. Some providers accept it natively; others require translation.

A translated text response may work while provider-specific state or hosted tools have no equivalent. The OpenAI Responses API with AISIX explains these boundaries. Test Responses separately even when the same model already passed Chat Completions.

Native and Passthrough APIs

Sometimes translation is the wrong abstraction. An application may depend on provider-native content blocks, token counting, batch semantics, or Realtime events.

The AISIX supported endpoints are listed by API family. When a capability has no safe common representation, use an explicit native route or separately authorized provider passthrough instead of hiding it behind a misleading shared alias.

Build a Compatibility Test Matrix

A useful matrix tests behavior, not marketing language.

Area	Minimum test	Failure it catches
Connection	base URL, TLS, key, timeout	wrong path, leaked provider key, unreachable gateway
Discovery	`GET /v1/models` and direct alias access	visible but unusable or unauthorized models
Request	required fields and supported options	accepted but silently dropped fields
Streaming	event order, finish, usage, cancellation	buffered, hanging, or malformed streams
Tools	call, arguments, result, follow-up	corrupted tool loop
Operations	errors, request IDs, retries	retry storms and untraceable incidents

Prove the Client Calls the Gateway

Use a gateway-issued caller key and confirm that the gateway records the expected identity and alias. The upstream provider credential must not appear in application configuration, logs, or error messages.

Test the exact base URL. Some clients append /chat/completions to a /v1 root; supplying the complete endpoint can create duplicated paths. Record the value expected by each SDK or platform.

Model discovery needs both positive and negative tests. A caller should see only intended aliases, but the model list is not an authorization boundary. A direct request for an unauthorized alias must still fail.

Assert Behavior, Not Status 200

Build fixtures from fields the application actually sends. Cover messages, output limits, sampling controls, stop behavior, and expected response fields. Add structured output, image input, or other features only when required.

Some services accept an unknown field and ignore it. The request returns 200 while the application contract is broken. Assertions should examine the returned shape and behavior, not only the status code.

The AISIX provider compatibility matrix is endpoint-specific. That is healthier than one global compatibility claim because Chat Completions may cover more adapters than Files, Batch, Fine-tuning, Realtime, or media endpoints.

Test Streaming Through the Real Path

Streaming failures often appear only after deployment. Verify:

first-event latency and event ordering;
text-delta assembly;
tool-call indexes and argument fragments;
finish reason and expected terminator;
usage placement when requested;
client cancellation, upstream disconnect, and timeout behavior.

A proxy can forward bytes yet still break the client by buffering, altering event boundaries, or dropping indexes. Run tests through the production load balancer and ingress path, not only against localhost.

Complete the Tool Loop

Tool calling spans multiple requests. The model proposes a tool and arguments. The application validates and executes it, then returns the result with the matching call identifier.

Test the full loop in streaming and non-streaming modes. Verify parallel calls only when the application permits them. The gateway governs model traffic; the application remains responsible for tool authorization, argument validation, and execution safety.

Test Errors and Retries as a Contract

Error shape affects client behavior. Exercise invalid authentication, unauthorized aliases, bad fields, rate limits, budget rejection, upstream timeout, provider error, and gateway overload.

For each case, define:

HTTP status and response body;
whether the client may retry;
backoff and attempt limit;
request or trace identifier;
usage or cost accounting behavior.

Do not retry every 429 or 5xx blindly. A budget denial, invalid request, or unsupported capability is not repaired by repetition. For retryable failures, cap attempts and preserve enough identity to detect duplicate work.

Usage deserves its own assertions. Confirm token fields and streaming usage placement for every adapter used in cost reporting. A translated response can be syntactically valid while its accounting fields are absent or mapped differently.

Publish Capability Profiles for Aliases

A stable alias reduces application coupling only when its promise is explicit.

alias: support-chat-prod
api_family: chat_completions
streaming: required
tools: required
structured_output: json_schema
image_input: unsupported
context_window: organization-defined minimum
max_output: organization-defined minimum

All routing targets should satisfy the profile. If a fallback cannot call tools, it should not silently serve a tool-dependent alias. Use a narrower alias or make degraded behavior explicit.

Re-run the contract suite when the client SDK, gateway, adapter, model, inference runtime, prompt template, quantization, or routing target changes. OpenAI's backward-compatibility guidance also distinguishes API stability from model-output consistency. Conformance and quality need separate evaluations.

What the Gateway Should Normalize

Normalize cross-cutting controls that applications should not rebuild:

caller authentication and model authorization;
provider credential isolation;
stable aliases and controlled routing;
rate limits, budgets, guardrails, and audit events;
metrics, usage, and request correlation.

Keep protocol translation narrower. Translate documented fields when semantics are known. Reject unsupported behavior clearly. Do not fabricate provider behavior merely to return 200.

AISIX AI Gateway uses explicit caller-facing endpoint families and provider adapters. That boundary lets platform teams change providers behind tested aliases while developers keep a documented entry point and security teams avoid distributing upstream keys.

Production Checklist

Define the API family and features each application uses.
Issue a dedicated caller identity rather than reusing a provider key.
Publish a capability profile for every alias.
Test discovery and direct unauthorized-alias rejection.
Assert request and response behavior, not only HTTP status.
Test streaming, cancellation, tools, usage, and errors.
Run the suite against every primary and fallback target.
Record gateway, adapter, SDK, and model versions with results.
Canary changes and preserve a rollback target.

An OpenAI-compatible AI gateway reduces application coupling without making providers identical. Treat compatibility as evidence, not a badge: start with one alias and one tested API family, then expand the contract deliberately.