Private LLM Gateway: Secure vLLM, SGLang, and Ollama

Running a model with vLLM, SGLang, or Ollama gives you an inference API. It does not automatically give every application a safe production platform.

A private LLM gateway adds the missing runtime boundary. Applications authenticate to the gateway and request a stable model alias. The gateway applies access and traffic policy, selects an approved private endpoint, records the outcome, and forwards the request without exposing infrastructure addresses or upstream credentials to every client.

The inference server still loads models, schedules GPU work, manages batching and KV cache, and generates tokens. The gateway does not replace that work. It makes access to the work governable.

Key Takeaways

• vLLM, SGLang, and Ollama are inference servers; an LLM gateway is the identity, policy, routing, and telemetry layer in front of them.
• OpenAI compatibility is an API contract for supported routes, not proof that every endpoint or field behaves identically.
• Keep inference ports private. A placeholder API key required by a configuration schema is not authentication.
• Stable model aliases decouple applications from server addresses, model IDs, runtime versions, and pool migrations.
• Rate limits should cover requests, tokens, and concurrency so retries do not amplify GPU overload.
• Private-model economics include hardware time, utilization, queueing, operations, and power—not only a token price.
• AISIX AI Gateway can connect private OpenAI-compatible endpoints to the same caller, routing, policy, and observability model used for public providers.

A Private LLM Server Is Not Yet a Governed Platform

Modern inference servers make open-weight models easy to expose over HTTP. vLLM's OpenAI-compatible server supports common OpenAI-style APIs. The SGLang quickstart starts a server and calls it with an OpenAI client. Ollama's compatibility API lets local tools use familiar endpoints and client libraries.

Those capabilities solve the serving problem: turn a model and runtime configuration into an API that can produce a response.

Production platforms face a broader problem:

• Which application or team may use each model?
• Should a developer receive the inference server's address or credential?
• How do clients survive a server, model name, or runtime migration?
• What stops a retry loop from filling the GPU queue?
• Where can operators compare latency, tokens, failures, and capacity by caller?
• How do private endpoints coexist with Bedrock, Azure OpenAI, Gemini, or another public provider?

An inference server can implement some of these controls. A shared gateway prevents each engine, application, and team from implementing them differently.

Divide Responsibilities Explicitly

The inference server loads weights, schedules GPU work, manages batching and KV cache, and generates tokens. The gateway authenticates applications, maps callers to model aliases, applies traffic policy, chooses an eligible endpoint, and records the request outcome. Keep that boundary visible: a gateway cannot repair an overloaded scheduler, and an inference engine should not become every team's policy database.

Treat OpenAI Compatibility as a Tested Contract

OpenAI-compatible APIs let applications reuse familiar clients, but the label is not a certification. Start with the exact route you need—often POST /v1/chat/completions—and verify the base URL, served model name, streaming termination, usage fields, error shapes, authentication, tokenizer, and chat template.

Do not infer that embeddings, Responses, audio, rerank, or administrative endpoints behave the same because chat completions works. The AISIX provider compatibility matrix documents endpoint boundaries. Maintain contract tests for every approved runtime and rerun them after changes to the model, engine, quantization, adapter, or prompt template.

Keep engine-specific extensions out of the shared alias unless every client is meant to depend on them. A native operations path should be named and authorized separately.

Build a Private Request Path

Applications should present a gateway-facing credential for an application, team, tenant, or environment. The gateway separately holds the credential expected by the inference endpoint and maps the caller to approved model aliases. A placeholder value required by a configuration schema is not authentication and must never justify exposing an unauthenticated engine publicly.

Use an alias such as support-chat-prod, not a node-specific model ID. The alias can represent a tested context limit, data region, quality tier, and endpoint contract. The platform can then replace a node or runtime after validation without editing every application.

Keep engine ports on loopback, a private container network, a Kubernetes Service protected by NetworkPolicy, or a private VPC path. Administrative and health routes deserve their own access rules. NIST SP 800-207 reinforces the principle that network location alone is not trust.

Protect Finite Inference Capacity

Requests per minute is not enough. A short prompt and a long-context generation consume different work, while concurrent generations directly affect memory, batching, and queue time. Combine caller-level request limits with token windows, concurrency caps, maximum context and output lengths, and deliberate timeouts. The AI Gateway rate-limiting guide explains the provider, tenant, and token dimensions.

Retry carefully. When a client times out while the original generation continues, an automatic retry can double work during saturation. Propagate cancellation where supported, back off, and route only to targets that satisfy the same model and data-policy contract. A reachable server running an incompatible model is not a valid fallback.

Guardrails also consume time and may buffer streams or call another service. Attach them by risk, measure their failure mode, and benchmark the final policy set rather than a policy-free demo.

Join Gateway and Engine Telemetry

Gateway metrics explain who requested a platform contract; engine metrics explain how the serving layer executed it. Record caller, alias, resolved endpoint, status, routing outcome, time to first token, total latency, input and output tokens, and cancellation. At the engine, observe queue depth, active sequences, batch size, cache utilization, accelerator use, memory, and model-loading state. Correlate the layers with a request identifier. The OpenTelemetry GenAI conventions provide a vocabulary, but prompts and customer data do not belong in metric labels.

Private inference is not free because it lacks a provider invoice. Allocate GPU or CPU time, idle headroom, storage, power, operations, and failed work alongside token usage. The AI Gateway cost-control guide covers the broader budget loop.

Benchmark three paths separately: direct-to-engine, gateway with a mock upstream, and the production-like path with real inference, streaming, authentication, policy, and concurrency. AISIX's performance and sizing baseline isolates gateway overhead with a mock upstream; it is not an end-to-end LLM workload promise.

Connect One Private Endpoint to AISIX

The AISIX Bring Your Own Endpoint guide covers OpenAI-compatible private servers such as vLLM, SGLang, and Ollama. The caller-facing contract remains stable across both deployments, but management differs: AISIX Cloud manages and projects endpoint and provider-key resources through its control plane, while open-source AISIX loads operator-managed entries from resources.yaml or etcd and resolves secrets from the gateway environment or configured store. In either case, the application calls AISIX with a caller key and stable alias rather than an upstream credential.

Start with one engine and one known route. Verify the request directly from the gateway network, register the endpoint, create a stable alias, grant it to one caller, and test success, denial, streaming, cancellation, and error behavior through the gateway. Add traffic limits and correlated telemetry before introducing a second pool or public-provider fallback.

Use separate routes for provider-native extensions and check the compatibility matrix before enabling other APIs. A private pool and public provider may share a platform, but data residency, model behavior, endpoint support, and cost remain eligibility constraints. The multi-cloud AI Gateway guide explains how to preserve those boundaries.

Private LLM Gateway Checklist

Before production, confirm that the inference port is private; every exposed route is authenticated; caller and upstream credentials are separate; aliases match tested model contracts; request, token, concurrency, timeout, and retry policies protect the pool; gateway and engine metrics correlate; policy-enabled benchmarks use representative traffic; and endpoint, credential, runtime, and model changes all have a rollback target.

Conclusion

vLLM, SGLang, and Ollama make private inference practical. A production platform must also make that inference accountable, bounded, observable, and replaceable.

A private LLM gateway provides the contract between application teams and model infrastructure. It authenticates callers, exposes stable aliases, keeps endpoint details private, applies traffic policy, and records which workload used which pool. The inference server continues to do what it does best: schedule hardware and generate tokens.

Start with one OpenAI-compatible chat endpoint and test its real behavior. Keep the engine on a private network, grant one caller the minimum model access, add capacity-aware limits, and correlate gateway telemetry with engine metrics. Then expand carefully to more models, routes, and providers.

Explore AISIX AI Gateway and follow the Bring Your Own Endpoint guide to put governed access in front of your first private inference server.