7 Open-Source LiteLLM Alternatives (Self-Hosted, 2026)
By API7.ai Team
Last updated: August 2026
LiteLLM offers a unified OpenAI-compatible interface to 100+ providers, a proxy with virtual keys, spend tracking, guardrails, and load balancing, its own MCP gateway, and roughly 57k GitHub stars as of August 2026. This page covers seven open-source LLM gateways you can self-host instead, with the language, license, and standout capabilities of each — sourced from each project's own repository and documentation, and including our own AISIX, evaluated against the same evidence standard.
All seven alternatives are open source and self-hostable — pick by constraint, not ranking. AISIX governs LLM, MCP, and A2A traffic with one policy layer; Bifrost majors on multi-key failover; Portkey on config-driven routing with guardrails; Kong and Higress extend gateways you already run; GoModel keeps the proxy minimal. (TensorZero appears for completeness: its repository was archived in June 2026.)
- Provider breadth and Python extensibility matter most: Stay with LiteLLM
- Governing agents — LLM, MCP, and A2A under one policy layer: AISIX
- Uptime through redundancy across many API keys: Bifrost
- Routing and guardrails in config, or edge deployment: Portkey
- You already run an API gateway: Kong AI Gateway or Higress
- Smallest possible operational surface: AISIX or GoModel
Why look beyond LiteLLM?
None of the common triggers imply LiteLLM did something wrong — teams comparison-shop because they outgrow a specific shape, not because the incumbent failed them.
- Runtime shape — a compiled Go or Rust binary can simplify deployment and operations compared with running a Python service, depending on your platform team's preferences.
- Scope of governance — agent workloads add MCP tool calls and agent-to-agent traffic alongside LLM calls; some gateways govern all three surfaces with one policy model, others focus on LLM calls.
- Consolidation — if Kong, Envoy, or another gateway already fronts your APIs, adding AI capability there may beat running a second proxy.
- License packaging — some teams standardize on repositories entirely under one OSI-approved license; LiteLLM is MIT-licensed with the enterprise/ directory carved out under a separate license, a packaging detail worth knowing before you fork or embed it.
- Beyond proxying — if your real problem is evaluating and improving model behavior, an LLMOps platform with a gateway inside it may fit better than a pure proxy.
How we evaluated
This guide is published by API7.ai, the company behind AISIX and the original creators of Apache APISIX. We include AISIX alongside six competitors, state where each project stands out, and say plainly when staying with LiteLLM is the right call. Facts about every project — including star counts, fetched on 2026-08-21 — come from each project's own repository and documentation.
- Language, runtime shape, and operational footprint
- License and how the repository is packaged
- Self-hosting path and deployment options
- MCP and agent-traffic support, and how tool access is governed
- Standout capabilities per official repositories and docs
1. AISIX
A Rust single binary under Apache-2.0 that governs LLM, MCP, and A2A traffic with one set of caller keys, rate and token limits, guardrails, and usage logs.
Disclosure: AISIX is built by API7.ai, which publishes this page. It is an open-source AI gateway from the original creators of Apache APISIX. OpenAI-compatible and Anthropic Messages (/v1/messages) are both first-class client protocols: it translates requests and responses, streaming included, both ways across OpenAI, Anthropic, Gemini (Vertex AI), AWS Bedrock, Azure OpenAI, and any OpenAI-compatible endpoint such as vLLM, Ollama, or DeepSeek.
Language
Rust
License
Apache-2.0 — whole repository, one license
Form factor
Single static binary
Best for
Governing LLM, MCP, and A2A traffic together
Pros
- MCP gateway: register upstream MCP servers behind one governed Streamable HTTP /mcp endpoint — per-tool access control by caller API key, guardrails on tool arguments and results, rate limits, and usage logs
- Also an A2A agent gateway: the same caller keys, limits, guardrails, and logs apply across LLM, MCP, and A2A traffic
- Governance in the open-source core: caller API keys; rate and token limits (RPM, RPD, TPM, TPD, concurrency); exact and semantic caching
- Routing intelligence in the OSS core: semantic routing dispatches each request by its meaning, and ensemble models fan one request out to a panel with a judge model synthesizing a single answer
- Guardrails in OSS: keyword filtering, PII detection and redaction, plus AWS Bedrock Guardrails, Azure Content Safety, Aliyun moderation, Lakera, Presidio, and OpenAI Moderation
- Observability: Prometheus /metrics plus OTLP GenAI spans (Langfuse, Honeycomb, Grafana, any OTLP backend), with Datadog and Aliyun SLS exporters
- Small footprint: single-container declarative mode runs from one static binary; one declarative resources.yaml holds every dynamic resource — no database, configuration store, or control plane required, with etcd optional for multi-replica clusters
Cons
- Providers are covered through five native adapter families plus any OpenAI-compatible endpoint, rather than per-vendor integrations for 100+ providers
- Multi-tenant dashboard, teams, spend management, and audit sit in the commercial AISIX Cloud control plane, not in the OSS gateway
When to pick it: You want LLM, MCP, and A2A traffic under one governance layer rather than separate tools, and you prefer a compiled single binary with a small deployment footprint. Read the full AISIX vs LiteLLM head-to-head, or start from the AISIX AI Gateway overview.
2. Bifrost
A Go gateway from Maxim built around multi-key load balancing: requests spread across multiple API keys with automatic fallbacks between providers and models.
Bifrost is an AI gateway from Maxim that provides unified, OpenAI-compatible access to 23+ providers, including OpenAI, Anthropic, AWS Bedrock, and Google Vertex. It starts via npx or Docker, or embeds directly as a Go SDK.
Language
Go
License
Apache-2.0
Form factor
Binary / Docker / Go SDK
GitHub stars
Roughly 7.5k as of August 2026
Best for
Resilience across many API keys
Pros
- Load balancing distributes requests across multiple API keys, with automatic fallbacks between providers and models (adaptive load balancing sits in its enterprise tier)
- Text, image, audio, and streaming behind a common interface
- Semantic caching, Model Context Protocol (MCP) support for external tools, and a plugin middleware architecture for custom logic
- Hierarchical cost control with virtual keys, teams, and customer budgets
- Prometheus metrics and distributed tracing
Cons
- Adaptive load balancing, clustering, guardrails, the MCP gateway, and OIDC user provisioning are enterprise capabilities, per its README
- Provider catalog (23+) is far smaller than LiteLLM's 100+
- Best suited to teams comfortable in the Go ecosystem
When to pick it: Your priority is resilience — spreading traffic across many keys with automatic provider failover — and your team is comfortable in the Go ecosystem, or wants a built-in UI and plugin layer.
3. Portkey AI Gateway
A TypeScript gateway routing to 1,600+ models, with guardrails and routing rules expressed together in configuration — and deployable at the edge on Cloudflare Workers.
Portkey's open-source gateway routes to 1,600+ language, vision, audio, and image models through one API; Portkey also offers a hosted gateway and enterprise deployments. A Gateway 2.0 pre-release is underway that, per the project's README, merges Portkey's core enterprise gateway into the open-source repository.
Language
TypeScript
License
MIT
Form factor
Node.js service; runs on Cloudflare Workers
GitHub stars
Roughly 12.8k as of August 2026
Best for
Config-driven routing with integrated guardrails
Pros
- 40+ pre-built guardrails verify inputs and outputs against your specified checks, directly in the gateway — and you can bring your own
- Automatic retries, fallbacks, load balancing, conditional routing, and request timeouts expressed as configuration
- Simple caching in the OSS gateway (semantic caching is hosted/enterprise); multimodal support; the broader Portkey product offers a hosted MCP gateway
- Deploys on Docker, Kubernetes, AWS, Azure, GCP — and notably Cloudflare Workers, which makes edge deployment straightforward
Cons
- TypeScript/Node.js service rather than a compiled single binary
- Gateway 2.0 — the merge of the core enterprise gateway into the open-source repository — is still a pre-release
- The hosted gateway and enterprise deployments are separate commercial offerings
When to pick it: You want routing rules and guardrails expressed together in config, your team lives in TypeScript, or you want to run the gateway at the edge on Cloudflare Workers.
4. Kong AI Gateway
AI plugins on Kong Gateway that let LLM traffic inherit the API platform you already operate — authentication, rate limiting, ingress, and observability included.
Kong AI Gateway is a set of AI plugins on top of Kong Gateway, the widely deployed API gateway. Rather than a standalone LLM proxy, it extends existing API infrastructure with AI capability, inheriting Kong's authentication, rate limiting, Kubernetes ingress, and observability.
Language
Lua on OpenResty
License
Apache-2.0 (Kong Gateway repository)
Form factor
Plugins on Kong Gateway
GitHub stars
Roughly 44k (Kong Gateway) as of August 2026
Best for
Teams already running Kong
Pros
- AI Proxy plugin accepts requests in OpenAI format and translates them to providers including Azure OpenAI, Bedrock, Anthropic, Gemini, Mistral, and vLLM (Kong Gateway 3.6+)
- AI Gateway Enterprise guardrail plugins: Azure Content Safety, AWS Bedrock Guardrails, Lakera Guard, semantic guards, and PII sanitization
- AI Gateway Enterprise adds prompt-engineering plugins (templates, decorators, compression), semantic caching and routing, and automated RAG injection
- AI Gateway Enterprise plugins handle MCP and A2A traffic, on top of Kong's mature gateway machinery
Cons
- Not a standalone LLM proxy — you adopt (or already run) Kong Gateway to use it
- AI plugin availability varies — check each plugin's compatibility page for the editions and deployment modes it supports
When to pick it: You already run Kong, or you want AI traffic governed by the same platform as the rest of your APIs instead of introducing a separate proxy.
5. TensorZero
An LLMOps platform in Rust that wraps its gateway with observability, evaluation, experimentation, and feedback-driven optimization — the loop, not just the proxy.
TensorZero is an LLMOps platform rather than a pure gateway: it unifies an LLM gateway, observability, evaluation, optimization, and experimentation in one open-source system, with inferences and feedback stored in your own database. Note: the tensorzero/tensorzero repository was archived on GitHub in June 2026 and is read-only — the project is no longer maintained.
Language
Rust (with Python tooling)
License
Apache-2.0
Form factor
LLMOps platform with a gateway inside
GitHub stars
Roughly 11.7k as of August 2026
Best for
Feedback-driven model improvement
Pros
- Gateway access to every major LLM provider through a unified API, with A/B testing, routing, fallbacks, and retries
- Inferences and feedback stored in your own database — the data stays in your infrastructure
- Evaluations benchmark individual inferences or end-to-end workflows using heuristics or LLM judges
- Collected metrics and human feedback feed back into optimizing prompts, models, and inference strategies
Cons
- The GitHub repository was archived in June 2026 and is read-only — no further development, fixes, or security patches
- An LLMOps platform, not a pure proxy — broader scope than teams that only need routing
- An MCP gateway is not a documented headline feature as of August 2026
When to pick it: Your bottleneck is not proxying but improving model behavior over time — you want experiments, evaluations, and feedback-driven optimization as first-class features, with the data staying in your infrastructure.
6. Higress
A cloud-native API gateway on Envoy and Istio that handles conventional APIs and AI traffic in one ingress layer, with Wasm plugin extensibility.
Higress is a cloud-native API gateway built on Istio and Envoy, extensible with Wasm plugins written in Go, Rust, or JavaScript. It originated at Alibaba and is now developed as a vendor-neutral CNCF project (the repository moved from alibaba/higress to higress-group/higress).
Language
Go (Envoy/Istio core)
License
Apache-2.0
Form factor
Envoy/Istio-based gateway, Wasm plugins
GitHub stars
Roughly 9.2k as of August 2026
Best for
One ingress for APIs and AI on Kubernetes
Pros
- Supports mainstream model providers both international and China-based
- AI observability, multi-model load balancing, token rate limiting, and caching
- Can host MCP servers through its plugin mechanism, letting AI agents call tools and services through the gateway
- Ships an openapi-to-mcp tool that converts OpenAPI specifications into remote MCP servers
Cons
- Built on an Envoy/Istio core — a Kubernetes-shaped platform rather than a single small binary
- A general-purpose API gateway first; AI capability arrives through its plugin layer
When to pick it: You are standardized on Kubernetes and Envoy, want one ingress layer handling both conventional APIs and AI traffic, or need first-class coverage of China-region model providers.
7. GoModel
A lightweight Go proxy that speaks both the OpenAI and Anthropic API dialects, with cost tracking, hard spend limits, and its own MCP gateway built in.
GoModel (by Enterpilot) is a lightweight AI gateway that explicitly positions itself as a LiteLLM and Portkey alternative, exposing both OpenAI-compatible and Anthropic-compatible APIs. It installs as a single binary via script or as a Docker image.
Language
Go
License
MIT
Form factor
Single Go binary or Docker image
GitHub stars
Roughly 1.1k as of August 2026
Best for
Minimal dual-dialect proxy with cost controls
Pros
- Unified access to OpenAI, Anthropic, Cohere, Google Gemini, Vertex AI, DeepSeek, Groq, Fireworks AI, and others
- Intelligent routing with automatic rerouting to backup providers
- Exact and semantic response caching; request and response guardrail policies enforced at the gateway
- Cost tracking with spending breakdowns and hard spend limits
- Prometheus metrics, audit logs, live request streaming, and a documented MCP gateway that aggregates your MCP servers behind one authenticated endpoint
Cons
- Roughly 1.1k GitHub stars as of August 2026
- A minimal proxy by design — a smaller feature surface than the platform-scale gateways here
When to pick it: You want a minimal Go proxy that speaks both the OpenAI and Anthropic API dialects, with cost controls built in, and you are comfortable adopting an early-stage project.
Comparison at a glance
Every project below is open source and self-hostable; they differ in language, license packaging, form factor, MCP support, and the problem each one majors on.
| Feature | AISIX | Bifrost | Portkey | Kong | TensorZero | Higress | GoModel |
|---|---|---|---|---|---|---|---|
| Language | Rust | Go | TypeScript | Lua (Kong plugins) | Rust (+ Python tooling) | Go (Envoy/Istio core) | Go |
| License | Apache-2.0 | Apache-2.0 | MIT | Apache-2.0 (Kong Gateway repo) | Apache-2.0 | Apache-2.0 | MIT |
| Form factor | Single static binary | Binary / Docker / Go SDK | Node.js service; runs on Cloudflare Workers | Plugins on Kong Gateway | LLMOps platform with a gateway inside | Envoy/Istio gateway, Wasm plugins | Single Go binary |
| MCP gateway | ✓ Governed /mcp, per-tool ACL | ✓ Tool support (MCP gateway: enterprise) | Hosted-product MCP gateway | ✓ MCP + A2A traffic (AI license) | — (archived) | ✓ Hosts MCP servers | ✓ MCP gateway |
| Standout focus | Semantic routing + ensemble models in OSS; LLM + MCP + A2A in one gateway | Multi-key load balancing + failover | Config-driven routing + integrated guardrails | AI traffic on an established API gateway | Observe/evaluate/optimize loop (archived June 2026) | Cloud-native API + AI gateway on Envoy | Lightweight dual OpenAI/Anthropic proxy |
| Best for | Dual OpenAI/Anthropic protocols; agent-traffic governance | Resilience across many API keys | TypeScript teams, edge deployment | Teams already on Kong | Archived — evaluate successors before adopting | Kubernetes/Envoy shops, China-region providers | Minimal proxy with cost controls |
“—” means the capability is not a documented headline feature as of August 2026; check the project's current docs. Star counts and version details throughout were fetched on 2026-08-21 and will drift. LiteLLM itself (the baseline): Python, MIT with a separately licensed enterprise/ directory, self-hostable, with an MCP gateway of its own.
How to choose
Match the project to the constraint that pushed you to look — provider breadth, agent governance, redundancy, consolidation, evaluation, footprint, or license packaging.
Choose LiteLLM if you…
- Value the broadest provider catalog (100+) behind one OpenAI-compatible API
- Want a Python codebase your team can extend itself
- Want the largest community on this page — roughly 57k GitHub stars as of August 2026
Choose AISIX if you…
- Are governing agents, not just chat completions — one set of keys, limits, guardrails, and logs across LLM, MCP, and A2A traffic
- Want a small operational surface: one container plus one YAML file
- Standardize on repositories under a single Apache-2.0 license
Choose Bifrost if you…
- Chase uptime through redundancy — many API keys with automatic provider failover
- Are comfortable in the Go ecosystem or want its plugin middleware layer
Choose Portkey if you…
- Want routing rules and guardrails expressed together in config
- Live in TypeScript, or want to run the gateway at the edge on Cloudflare Workers
Choose Kong AI Gateway if you…
- Already run Kong and want AI traffic on the same platform as your APIs
- Want mature gateway machinery — auth, rate limiting, ingress, observability — applied to LLM traffic
Choose Higress if you…
- Are standardized on Kubernetes and Envoy and want one ingress for APIs and AI
- Need first-class coverage of China-region model providers
Choose GoModel if you…
- Want a minimal Go proxy speaking both OpenAI and Anthropic dialects
- Want cost tracking and hard spend limits built in, and accept an early-stage project
If license packaging matters to your legal team: the AISIX, Bifrost, TensorZero, Higress, and Kong Gateway repositories are Apache-2.0; Portkey and GoModel are MIT; LiteLLM is MIT with a separately licensed enterprise/ directory.
The bottom line
There is no ranked winner here — each of the seven earns its slot for a different constraint, and staying with LiteLLM is a legitimate outcome too.
If provider breadth and Python extensibility matter most, stay with LiteLLM — its catalog and community are genuinely the largest here. The alternatives win on shape: compiled single binaries (AISIX, GoModel), redundancy-first load balancing (Bifrost), config-driven guardrails and edge deployment (Portkey), consolidation onto an existing gateway (Kong, Higress), or a full evaluation loop (TensorZero).
Choose AISIX for semantic routing and ensemble models in the OSS core, dual OpenAI/Anthropic client protocols, and one governance layer across LLM, MCP, and A2A traffic in an Apache-2.0 Rust binary — the code, quickstart, and issue tracker are at github.com/api7/aisix, the documentation lives at docs.api7.ai/ai-gateway, and the detailed head-to-head is AISIX vs LiteLLM.
Frequently asked questions
Related comparisons
AISIX vs LiteLLM · Self-hosted LLM gateway guide · Portkey vs LiteLLM · AISIX AI Gateway · All AI gateway comparisons
Ready to get started?
For more information about full API lifecycle management, please contact us to Meet with our API Experts.
