Self-Hosted LLM Gateway: Why and How to Choose (2026)

What an LLM gateway does

An LLM gateway (also called an AI gateway or LLM proxy) sits between your applications and model providers. In practice, the layer covers six jobs: credential separation, rate and token limits, routing and failover, caching, observability, and guardrails.

• Credential separation. Provider API keys live at the gateway. Applications and teams get gateway-issued caller keys instead, so a leaked app key never exposes your OpenAI or Anthropic account, and revoking one consumer doesn’t rotate keys for everyone.
• Rate and token limits. Requests per minute/day and tokens per minute/day, per caller key, model, or team — plus concurrency caps so one runaway agent can’t starve everyone else.
• Routing and failover. One model alias that maps to real deployments behind it, with retries, fallback targets, and load balancing across providers or API keys.
• Caching. Serving repeated requests from the gateway instead of paying the provider again.
• Observability. Per-request logs with token counts and cost attribution, metrics for dashboards, and traces you can send to whatever backend you already run.
• Guardrails. Input and output content policies — keyword blocks, PII detection and redaction, or calls out to moderation services — enforced in one place instead of in every app.

Increasingly there is a seventh job: agent traffic. Agents call tools over MCP (Model Context Protocol) and other agents over A2A, and that traffic needs the same keys, limits, and logs as chat completions. Some gateways now govern all three through one control point.

Why self-host an LLM gateway?

Four reasons come up consistently with platform teams: credentials stay in your infrastructure, no per-request third-party dependency, compliance, and latency.

• Credentials stay in your infrastructure. With a hosted gateway, your provider keys — and every prompt and completion — transit a third party. Self-hosting keeps both inside your network boundary.
• No per-request third-party dependency. A hosted gateway is on the critical path of every LLM call. Its outages are your outages, and its rate limits are your rate limits. A gateway you run has exactly the availability you engineer for it.
• Compliance. Prompts routinely carry customer data. If you have data-residency or regulatory constraints, routing that data through a gateway vendor is a contract negotiation; routing it through your own VPC usually isn’t.
• Latency. A self-hosted gateway adds one hop inside your network before the call goes to the provider. A hosted gateway adds a round-trip to someone else’s cloud first.

The honest trade-off: you now operate it. Upgrades, TLS, and high availability are your problem — the FAQ below covers the standard patterns, and they are well-trodden.

What to evaluate in a self-hosted LLM gateway

Six dimensions separate the options in practice: deployment weight, wire compatibility, governance depth in the open-source tier, agent-protocol support, observability exports, and license stability.

Six self-hostable LLM gateways

Disclosure: this guide is published by API7.ai, the company behind AISIX; facts about the other five projects are drawn from their official READMEs, docs, and LICENSE files. Star counts were fetched from the GitHub API on 2026-08-21 — treat them as order-of-magnitude.

What a single-container deployment looks like

Deployment weight is easiest to judge with a concrete example. This is AISIX’s quickstart, verbatim from its README: “One container. No control plane, no database, no configuration store — the gateway reads every dynamic resource from one declarative resources.yaml.”

# config.yaml
resources_file: /etc/aisix/resources.yaml
proxy:
  addr: "0.0.0.0:3000"
admin:
  enabled: false          # a declarative gateway needs no admin listener
observability:
  metrics:
    prometheus:
      enabled: true
      addr: "0.0.0.0:9090"

# resources.yaml
_format_version: "1"

provider_keys:
  - display_name: openai-main
    provider: openai
    api_key: ${OPENAI_API_KEY}        # interpolated from the environment

models:
  - display_name: my-model
    provider: openai
    model_name: gpt-4o-mini
    provider_key: openai-main

api_keys:
  - display_name: local-dev
    key_env: CALLER_API_KEY           # hashed at load; the plaintext is never stored
    allowed_models: ["my-model"]

export OPENAI_API_KEY="YOUR_PROVIDER_KEY"
export CALLER_API_KEY="YOUR_CALLER_KEY"

docker run -d --name aisix \
  --platform linux/amd64 \
  -v "$(pwd)/config.yaml:/etc/aisix/config.yaml:ro" \
  -v "$(pwd)/resources.yaml:/etc/aisix/resources.yaml:ro" \
  -e OPENAI_API_KEY -e CALLER_API_KEY \
  -p 3000:3000 -p 127.0.0.1:9090:9090 \
  ghcr.io/api7/aisix:latest        # proxy → :3000, metrics + status → :9090
#                                  ^ the metrics/status listener is unauthenticated;
#                                    keep it on loopback or a private network

curl http://localhost:3000/v1/chat/completions \
  -H "Authorization: Bearer $CALLER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"my-model","messages":[{"role":"user","content":"hello"}]}'

Edit resources.yaml and send SIGHUP (docker kill -s HUP aisix) to apply changes with no restart — an invalid file is rejected whole and the last good configuration keeps serving. For a multi-replica cluster, the same binary points at etcd instead of a file. Full walkthrough in the gateway quickstart.

How to choose

There is no single winner here — the projects genuinely optimize for different constraints. A reasonable evaluation is an afternoon each with your two shortlisted options: deploy the minimal shape, point one real service at it, kill a provider key mid-stream, and read the logs it produced.

The bottom line

Self-hosting the LLM middle layer keeps credentials, prompts, and availability under your control; the cost is operating one more service, and the patterns for TLS, HA, and upgrades are well-trodden.

Frequently asked questions