---
title: "How API Gateways Proxy LLM Requests - API7.ai"
description: "Discover how API gateways efficiently proxy large language model (LLM) requests."
url: "https://api7.ai/learning-center/api-gateway-guide/api-gateway-proxy-llm-requests"
---

[Skip to article](#column-content)

Column contentsAPI Gateway Guide

[API Gateway Guide](https://api7.ai/learning-center/api-gateway-guide)

Search this column

01 Gateway Fundamentals

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=fundamentals)[1. What Is an API Gateway? API Gateway Software and Use Cases](https://api7.ai/learning-center/api-gateway-guide/what-is-an-api-gateway)[2. Why Do Modern Architectures Need an API Gateway?](https://api7.ai/learning-center/api-gateway-guide/why-do-modern-architectures-need-api-gateway)[3. Reverse Proxy vs Load Balancer vs API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-vs-reverse-proxy-vs-load-balancer)[4. API Gateway and Microservices: Decoupling and Traffic Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-and-microservices)[5. Understanding Core API Gateway Features](https://api7.ai/learning-center/api-gateway-guide/core-api-gateway-features)[6. Open Source vs. Commercial API Gateways: How to Choose the Right One?](https://api7.ai/learning-center/api-gateway-guide/open-source-vs-commercial-api-gateway)[7. API Gateway vs Service Mesh: How to Choose?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-vs-service-mesh)[8. Envoy vs NGINX, Java, and Go for API Gateways](https://api7.ai/learning-center/api-gateway-guide/nginx-vs-envoy-vs-java-vs-go)[9. Core Concepts of API Gateway: Service, Route, Upstream, Plugin, and Consumer](https://api7.ai/learning-center/api-gateway-guide/core-concepts-of-api-gateway)[10. API Gateway Comparison: Apache APISIX vs. Kong vs. Traefik vs. KrakenD vs. Tyk](https://api7.ai/learning-center/api-gateway-guide/api-gateway-comparison-apisix-kong-traefik-krakend-tyk)

02 Gateway Architecture and Design

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=architecture)[11. Deep Dive into API Gateway Architecture: Core Components and Design Principles](https://api7.ai/learning-center/api-gateway-guide/api-gateway-architecture)[12. Choosing the Right API Gateway for Your Architecture](https://api7.ai/learning-center/api-gateway-guide/api-gateway-monolithic-vs-microservices)[13. How to Architect an API Gateway for High Availability (HA)?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-high-availability)[14. API Gateway Plugin Architecture: How to Extend and Customize Your API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-plugin-system)[15. How Event-Driven Architecture (EDA) Works with API Gateway?](https://api7.ai/learning-center/api-gateway-guide/api-gateway-event-driven-architecture)[16. API Gateway Request Lifecycle Explained](https://api7.ai/learning-center/api-gateway-guide/api-gateway-request-lifecycle-explained)[17. Cloud-Native API Gateway on Kubernetes](https://api7.ai/learning-center/api-gateway-guide/cloud-native-api-gateway-kubernetes)[18. API Gateway Control Plane vs. Data Plane](https://api7.ai/learning-center/api-gateway-guide/api-gateway-control-plane-vs-data-plane)[19. How API Gateways Work with GitOps: Automating API Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-gitops-integration)[20. How to Perform A/B Testing and Canary Releases with an API Gateway](https://api7.ai/learning-center/api-gateway-guide/ab-testing-canary-release-api-gateway)

03 Gateway Setup and Traffic Handling

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=traffic-protocols)[21. How to Deploy an API Gateway on Kubernetes: Step-by-Step Guide for Engineers](https://api7.ai/learning-center/api-gateway-guide/deploy-api-gateway-on-kubernetes)[22. Hands-on Setup with Apache APISIX API Gateway](https://api7.ai/learning-center/api-gateway-guide/api-gateway-getting-started-guide)[23. API Gateway Auth: OAuth 2.0, JWT & OIDC with APISIX](https://api7.ai/learning-center/api-gateway-guide/api-gateway-authentication-apache-apisix-oauth-jwt-oidc)[24. Mastering Traffic Control: Efficiently Managing API Requests with Gateways](https://api7.ai/learning-center/api-gateway-guide/api-gateway-traffic-control-strategies)[25. Building Reliable API Gateways with Logging and Monitoring](https://api7.ai/learning-center/api-gateway-guide/api-gateway-logging-monitoring-best-practices)[26. Dynamic Routing in API Gateways: Best Practices for Flexible Traffic Management](https://api7.ai/learning-center/api-gateway-guide/api-gateway-dynamic-routing-management)[27. How API Gateways Support WebSocket](https://api7.ai/learning-center/api-gateway-guide/how-api-gateways-support-websocket)[28. API Gateway for GraphQL: Security, Cost, and Caching](https://api7.ai/learning-center/api-gateway-guide/api-gateway-handle-graphql)[29. How API Gateways Handle HTTP/3 QUIC Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-http3-quic)[30. How API Gateways Handle MQTT Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-mqtt)

04 Gateway Integrations, AI, and MCP

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=ai-mcp)[31. How API Gateways Handle gRPC Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-grpc-support)[32. How API Gateways Implement Custom Plugins](https://api7.ai/learning-center/api-gateway-guide/api-gateway-custom-plugins)[33. How API Gateways Integrate with Kafka](https://api7.ai/learning-center/api-gateway-guide/api-gateway-kafka-integration)[34. How to Implement API Aggregation in API Gateways](https://api7.ai/learning-center/api-gateway-guide/api-gateway-api-aggregation)[35. How API Gateways Enable Serverless Architecture](https://api7.ai/learning-center/api-gateway-guide/api-gateway-serverless-architecture)[36. How API Gateways Proxy LLM Requests](https://api7.ai/learning-center/api-gateway-guide/api-gateway-proxy-llm-requests)[37. How API Gateways Enhance MCP Servers](https://api7.ai/learning-center/api-gateway-guide/api-gateway-enhance-mcp-server)[38. MCP Gateway vs MCP Proxy vs AI Gateway vs API Gateway](https://api7.ai/learning-center/api-gateway-guide/ai-gateway-vs-mcp-gateway-vs-api-gateway)[39. What Is an AI Gateway? Architecture and Benefits](https://api7.ai/learning-center/api-gateway-guide/what-is-an-ai-gateway)[40. What Is an MCP Gateway? Architecture, Use Cases & How It Works (2026 Guide)](https://api7.ai/learning-center/api-gateway-guide/what-is-mcp-gateway)

05 Gateway Performance

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=performance)[41. How API Gateway Reduces Latency: Performance Optimization Strategies](https://api7.ai/learning-center/api-gateway-guide/how-api-gateway-reduces-latency)[42. API Gateway Caching Strategy: Comprehensive Analysis and Implementation Guide](https://api7.ai/learning-center/api-gateway-guide/api-gateway-caching-strategy-analysis)[43. How to Optimize API Gateway Load Balancing: Advanced Strategies and Best Practices](https://api7.ai/learning-center/api-gateway-guide/how-to-optimize-api-gateway-load-balancing)[44. How API Gateway Efficiently Handles Large-Scale Traffic: Architecture and Strategies](https://api7.ai/learning-center/api-gateway-guide/how-api-gateway-handles-large-scale-traffic)[45. How to Improve API Gateway QoS: Traffic Classes, Capacity Protection, and SLOs](https://api7.ai/learning-center/api-gateway-guide/api-gateway-quality-of-service)[46. API Gateway Log Storage: Pipelines, Retention, Security, and Cost](https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-storage)[47. API Gateway Timeouts and Retries: Safe Policies and APISIX Configuration](https://api7.ai/learning-center/api-gateway-guide/api-gateway-timeouts-retries)[48. API Gateway Concurrency Control: Capacity Budgets, Backpressure, and Load Shedding](https://api7.ai/learning-center/api-gateway-guide/api-gateway-concurrency-control)[49. Asynchronous API Gateway Patterns: Queues, Callbacks, and Performance Boundaries](https://api7.ai/learning-center/api-gateway-guide/asynchronous-api-gateway-patterns)[50. API Gateway TLS Performance: Handshakes, Session Reuse, and Measurement](https://api7.ai/learning-center/api-gateway-guide/api-gateway-tls-performance)

06 Gateway Security

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=security)[51. DDoS Defense at the API Gateway: Layered Controls and Failure Planning](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ddos-defense)[52. Distributed API Gateway Rate Limiting: Local vs Redis Counters and Accuracy Trade-offs](https://api7.ai/learning-center/api-gateway-guide/distributed-api-gateway-rate-limiting)[53. API Gateway IP Allowlist and Denylist Management: Trust, Proxies, and Operations](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ip-allowlist-denylist-management)[54. API Gateway Security Scanning: What to Test and How to Operationalize Findings](https://api7.ai/learning-center/api-gateway-guide/api-gateway-security-scanning)[55. SQL Injection and XSS at the API Gateway: Detection Limits and Defense in Depth](https://api7.ai/learning-center/api-gateway-guide/sql-injection-xss-api-gateway)[56. API Access Log Auditing: Evidence, Integrity, Retention, and Review](https://api7.ai/learning-center/api-gateway-guide/api-access-log-auditing)[57. Preventing API Scraping and Abuse: Identity, Behavioral Signals, and Rate Controls](https://api7.ai/learning-center/api-gateway-guide/prevent-api-scraping-abuse)[58. API Gateway mTLS: Client Identity, Certificate Rotation, and APISIX Configuration](https://api7.ai/learning-center/api-gateway-guide/api-gateway-mtls)[59. Fine-Grained API Gateway Authorization: RBAC, ABAC, and External Policy Decisions](https://api7.ai/learning-center/api-gateway-guide/fine-grained-api-gateway-authorization)[60. API Gateway and WAF Reference Architecture: Ownership, Header Trust, and Failure Modes](https://api7.ai/learning-center/api-gateway-guide/api-gateway-waf-reference-architecture)

07 Gateway Ecosystem and Comparisons

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=ecosystem)[61. Apache APISIX vs Kong: Migration, Operations, and Team Fit](https://api7.ai/learning-center/api-gateway-guide/apache-apisix-vs-kong)[62. Traefik vs Kong vs APISIX: A Reproducible API Gateway Benchmark Method](https://api7.ai/learning-center/api-gateway-guide/traefik-kong-apisix-benchmark-method)[63. Amazon API Gateway Explained: Deployment Models, Trade-offs, and Use Cases](https://api7.ai/learning-center/api-gateway-guide/amazon-api-gateway-explained)[64. API Gateway and Istio: Clear North-South and East-West Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-istio)[65. API Gateway with NGINX: Layering, Ownership, and Migration Patterns](https://api7.ai/learning-center/api-gateway-guide/api-gateway-nginx)[66. API Gateway with Envoy: Deployment Patterns and Policy Ownership](https://api7.ai/learning-center/api-gateway-guide/api-gateway-envoy)[67. API Gateway with Cloudflare API Shield: Edge and Origin Security Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-cloudflare-api-shield)[68. Apigee vs MuleSoft vs Amazon API Gateway: A Decision Framework](https://api7.ai/learning-center/api-gateway-guide/apigee-mulesoft-amazon-api-gateway)[69. Should an API Gateway Connect Directly to a Database? Patterns and Risks](https://api7.ai/learning-center/api-gateway-guide/api-gateway-database-access-patterns)[70. API Gateway Architecture for Large-Scale SaaS Platforms](https://api7.ai/learning-center/api-gateway-guide/api-gateway-large-scale-saas-architecture)

08 Industry Applications and Integrations

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=applications)[71. API Gateway CI/CD: Validation, Promotion, and Safe Rollback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ci-cd)[72. API Gateways for AI and ML Inference: Traffic, Security, and Observability](https://api7.ai/learning-center/api-gateway-guide/api-gateways-ai-ml-inference)[73. API Gateway Architecture for Financial Services: Controls and Failure Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-financial-services)[74. API Gateway Architecture for E-Commerce: Checkout Reliability and Traffic Peaks](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ecommerce)[75. API Gateways for Mobile Applications: Authentication, Payloads, and Versioning](https://api7.ai/learning-center/api-gateway-guide/api-gateways-mobile-applications)[76. API Gateways for Web3 Applications: RPC Traffic, Identity, and Abuse Controls](https://api7.ai/learning-center/api-gateway-guide/api-gateways-web3-applications)[77. OpenAPI-Driven API Gateway Operations: Import, Validation, and Drift Control](https://api7.ai/learning-center/api-gateway-guide/openapi-driven-api-gateway-operations)[78. gRPC-Web and gRPC Transcoding Through an API Gateway](https://api7.ai/learning-center/api-gateway-guide/grpc-web-grpc-transcoding-api-gateway)[79. API Gateway Pricing and Cost Optimization: A Workload-Based Model](https://api7.ai/learning-center/api-gateway-guide/api-gateway-pricing-cost-optimization)[80. Evaluating API Gateway Directions: Evidence and Architectural Impact](https://api7.ai/learning-center/api-gateway-guide/api-gateway-trends)

09 Advanced Gateway Applications

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=operations)[81. API Gateways in Zero Trust Architecture: Identity, Policy, and Limits](https://api7.ai/learning-center/api-gateway-guide/api-gateways-zero-trust-architecture)[82. API Gateway Log Analytics: Correlation, Queries, and Operational Signals](https://api7.ai/learning-center/api-gateway-guide/api-gateway-log-analytics)[83. From API Sprawl to Governed Delivery: The Gateway's Role in Modernization](https://api7.ai/learning-center/api-gateway-guide/api-gateway-modernization)[84. API Monetization at the Gateway: Plans, Metering, Entitlements, and Billing Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-monetization-gateway)[85. Safe AI-Assisted API Gateway Optimization: Recommendations, Guardrails, and Approval](https://api7.ai/learning-center/api-gateway-guide/ai-assisted-api-gateway-optimization)[86. API Gateways for Low-Code and No-Code Integration: Governance and Safety](https://api7.ai/learning-center/api-gateway-guide/api-gateways-low-code-no-code-integration)[87. API Gateway and CIAM Integration: Token Validation, Consent, and Identity Context](https://api7.ai/learning-center/api-gateway-guide/api-gateway-ciam-integration)[88. Elastic API Gateway Scaling: Signals, Capacity, and Safe Scale-In](https://api7.ai/learning-center/api-gateway-guide/elastic-api-gateway-scaling)[89. Multi-Tenant SaaS API Gateways: Isolation, Quotas, and Tenant-Aware Policy](https://api7.ai/learning-center/api-gateway-guide/multi-tenant-saas-api-gateway-policy)[90. API Gateway DevSecOps: Policy as Code, Testing, and Runtime Feedback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-devsecops)

10 Platform Engineering and Planning

[View this topic · 10](https://api7.ai/learning-center/api-gateway-guide?topic=platform-governance)[91. API Discovery and Documentation at the Gateway: Inventory, Ownership, and Drift](https://api7.ai/learning-center/api-gateway-guide/api-discovery-documentation-gateway)[92. WebAssembly Plugins for API Gateways: Portability, Isolation, and APISIX Boundaries](https://api7.ai/learning-center/api-gateway-guide/api-gateway-webassembly-plugins)[93. Monitoring API Gateways with Prometheus and Grafana: Metrics, Dashboards, and Alerts](https://api7.ai/learning-center/api-gateway-guide/monitor-api-gateways-prometheus-grafana)[94. API Gateway Reliability Engineering: SLOs, Failure Isolation, and Incident Response](https://api7.ai/learning-center/api-gateway-guide/api-gateway-reliability-engineering)[95. API Gateways for Edge AI Inference: Routing, Constraints, and Observability](https://api7.ai/learning-center/api-gateway-guide/api-gateways-edge-ai-inference)[96. Reducing Cloud Cost with API Gateway Controls: Measurement Before Optimization](https://api7.ai/learning-center/api-gateway-guide/reducing-cloud-cost-api-gateway-controls)[97. Improving API Developer Experience with Gateway Standards and Self-Service](https://api7.ai/learning-center/api-gateway-guide/improving-api-developer-experience-gateway-self-service)[98. API Gateway Change Management: Compatibility, Progressive Delivery, and Rollback](https://api7.ai/learning-center/api-gateway-guide/api-gateway-change-management-rollback)[99. Operating Serverless APIs: Cold Starts, Idempotency, and Asynchronous Handoffs](https://api7.ai/learning-center/api-gateway-guide/operating-serverless-apis-gateway)[100. Planning an API Gateway Roadmap for the Next Five Years](https://api7.ai/learning-center/api-gateway-guide/planning-api-gateway-roadmap-next-five-years)

[API Gateway Guide](https://api7.ai/learning-center/api-gateway-guide)[Gateway Integrations, AI, and MCP](https://api7.ai/learning-center/api-gateway-guide?topic=ai-mcp)

On this page

On this page

- [Introduction](#introduction)

- [Quick Answer](#quick-answer)

- [Core Concepts and Challenges of Proxying LLM APIs](#core-concepts-and-challenges-of-proxying-llm-apis)

  - [LLM-Specific API Characteristics](#llm-specific-api-characteristics)
  - [Challenges in Proxying LLM Requests](#challenges-in-proxying-llm-requests)

- [Architectural Overview: API Gateway as LLM Proxy](#architectural-overview-api-gateway-as-llm-proxy)

  - [Key Responsibilities of API Gateway for LLM](#key-responsibilities-of-api-gateway-for-llm)
  - [Request Lifecycle Through API Gateway](#request-lifecycle-through-api-gateway)

- [Using Apache APISIX to Proxy LLM Requests](#using-apache-apisix-to-proxy-llm-requests)

  - [Key Plugins](#key-plugins)
  - [Example: Proxy to OpenAI](#example-proxy-to-openai)
  - [Failover Between OpenAI and DeepSeek](#failover-between-openai-and-deepseek)

- [Load Balance between LLM Instances](#load-balance-between-llm-instances)

- [Best Practices for Proxying LLM APIs](#best-practices-for-proxying-llm-apis)

  - [Use Token-Aware Rate Limiting](#use-token-aware-rate-limiting)
  - [Enable Retry with Backoff](#enable-retry-with-backoff)
  - [Preserve SSE/Streaming](#preserve-ssestreaming)
  - [Differentiate Providers](#differentiate-providers)
  - [Use Observability Plugins](#use-observability-plugins)

- [Real-World Use Case: LLM Gateway for Multi-Agent System](#real-world-use-case-llm-gateway-for-multi-agent-system)

- [Conclusion](#conclusion)

- [Next Steps](#next-steps)

Article 36

# How API Gateways Proxy LLM Requests

API7.ai

June 11, 2025

## Introduction

Large Language Models (LLMs), such as OpenAI's GPT-4 and DeepSeek's models, are increasingly used in applications ranging from AI assistants to code generation platforms. However, LLM APIs introduce unique challenges: they are rate-limited, compute-intensive, and occasionally unstable. API gateways can help software teams manage, secure, and optimize LLM requests in production systems.

This article explains how API gateways proxy LLM requests, including architectural design, plugin integration, flow control mechanisms, and real-world use cases. We will also explore how Apache APISIX can be configured to handle LLM workloads effectively.

## Quick Answer

Proxying LLM requests through an API gateway gives platform teams one place to enforce provider routing, token limits, API keys, retries, fallback, logging, and tenant policies across OpenAI-compatible and non-OpenAI-compatible model APIs.

The [AI Gateway Guide](https://api7.ai/learning-center/ai-gateway-guide) places these proxy patterns in a broader production architecture. Teams evaluating a dedicated implementation can also review [AISIX AI Gateway](https://api7.ai/ai-gateway), while [AI Gateway vs MCP Gateway vs API Gateway](https://api7.ai/learning-center/api-gateway-guide/ai-gateway-vs-mcp-gateway-vs-api-gateway) clarifies the protocol boundaries.

## Core Concepts and Challenges of Proxying LLM APIs

### LLM-Specific API Characteristics

- **High latency**: LLM responses may take seconds to return.
- **Token-based billing**: Most LLM providers charge based on token usage.
- **Rate limits**: Strict per-minute or per-second rate limits.
- **Retry strategies**: Occasional transient errors due to overuse.
- **Streaming responses**: SSE (Server-Sent Events) or chunked responses.

### Challenges in Proxying LLM Requests

- Managing retries with exponential backoff.
- Enforcing request/response timeout thresholds.
- Handling SSE and preserving response streaming.
- Differentiating between upstream APIs (OpenAI vs. DeepSeek).

## Architectural Overview: API Gateway as LLM Proxy

### Key Responsibilities of API Gateway for LLM

- **Traffic routing** based on model or provider.
- **Authentication** management (e.g., API keys for OpenAI, DeepSeek).
- **Rate limiting** per client or tenant.
- **Caching** for non-dynamic prompts.
- **Observability**: Logging, tracing, metrics.
- **Failover and fallback** mechanisms.

### Request Lifecycle Through API Gateway

```
sequenceDiagram
  participant Client
  participant API Gateway
  participant OpenAI
  Client->>API Gateway: POST /v1/chat/completions
  API Gateway->>API Gateway: Validate key, apply rate limiting
  API Gateway->>OpenAI: Forward request with upstream credentials
  OpenAI-->>API Gateway: LLM response (possibly streaming)
  API Gateway-->>Client: Stream response
```

## Using Apache APISIX to Proxy LLM Requests

[Apache APISIX](https://api7.ai/apisix) provides rich plugin capabilities suitable for LLM workloads.

### Key Plugins

- [ai-proxy](https://docs.api7.ai/hub/ai-proxy): The `ai-proxy` plugin simplifies access to LLM and embedding models by transforming plugin configurations into the designated request format. It supports the integration with OpenAI, DeepSeek, and other OpenAI-compatible APIs.
- [ai-rate-limiting](https://docs.api7.ai/hub/ai-rate-limiting): The `ai-rate-limiting` plugin enforces token-based rate limiting for requests sent to LLM services. It helps manage API usage by controlling the number of tokens consumed within a specified time frame, ensuring fair resource allocation and preventing excessive load on the service. It is often used with [ai-proxy-multi](https://docs.api7.ai/hub/ai-proxy-multi) plugin.
- [ai-request-rewrite](https://docs.api7.ai/hub/ai-request-rewrite): The `ai-request-rewrite` plugin processes client requests by forwarding them to LLM services for transformation before relaying them to upstream services. This enables LLM-powered modifications such as data redaction, content enrichment, or reformatting. The plugin supports the integration with OpenAI, DeepSeek, and other OpenAI-compatible APIs.
- [ai-aws-content-moderation](https://docs.api7.ai/hub/ai-aws-content-moderation): The `ai-aws-content-moderation` plugin supports the integration with AWS Comprehend to check request bodies for toxicity when proxying to LLMs, such as profanity, hate speech, insult, harassment, violence, and more, rejecting requests if the evaluated outcome exceeds the configured threshold.

### Example: Proxy to OpenAI

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "ai-proxy-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy": {
        "provider": "openai",
        "auth": {
          "header": {
            "Authorization": "Bearer '"$OPENAI_API_KEY"'"
          }
        },
        "options":{
          "model": "gpt-4"
        }
      }
    }
  }'
```

### Failover Between OpenAI and DeepSeek

```
sequenceDiagram
  participant Client
  participant API Gateway
  participant OpenAI
  participant DeepSeek
  Client->>API Gateway: POST /v1/chat/completions
  API Gateway->>OpenAI: Primary upstream call
  OpenAI-->>API Gateway: Error (e.g. 429)
  API Gateway->>DeepSeek: Retry as fallback
  DeepSeek-->>API Gateway: Success
  API Gateway-->>Client: Return DeepSeek response
```

## Load Balance between LLM Instances

The following example demonstrates how you can configure two models for load balancing, forwarding 80% of the traffic to one instance and 20% to the other.

For demonstration and easier differentiation, you will be configuring one OpenAI instance and one DeepSeek instance as the upstream LLM services.

Create a route as such and update with your LLM providers, models, API keys, and endpoints if applicable:

```
curl "http://127.0.0.1:9180/apisix/admin/routes" -X PUT \
  -H "X-API-KEY: ${ADMIN_API_KEY}" \
  -d '{
    "id": "ai-proxy-multi-route",
    "uri": "/anything",
    "methods": ["POST"],
    "plugins": {
      "ai-proxy-multi": {
        "instances": [
          {
            "name": "openai-instance",
            "provider": "openai",
            "weight": 8,
            "auth": {
              "header": {
                "Authorization": "Bearer '"$OPENAI_API_KEY"'"
              }
            },
            "options": {
              "model": "gpt-4"
            }
          },
          {
            "name": "deepseek-instance",
            "provider": "deepseek",
            "weight": 2,
            "auth": {
              "header": {
                "Authorization": "Bearer '"$DEEPSEEK_API_KEY"'"
              }
            },
            "options": {
              "model": "deepseek-chat"
            }
          }
        ]
      }
    }
  }'
```

## Best Practices for Proxying LLM APIs

### Use Token-Aware Rate Limiting

- Avoid flat per-request rate limits.
- Use per-user or per-token limits.

### Enable Retry with Backoff

- Use [fallback\_strategy](https://docs.api7.ai/hub/ai-proxy-multi/configuration) in ai-proxy-multi plugin.
- Detect status codes (429, 5xx) properly.

### Preserve SSE/Streaming

- Avoid plugins that buffer the full response.
- Confirm `Transfer-Encoding`: chunked is preserved.

### Differentiate Providers

- Route requests to different upstreams by model name, header, or request pattern.

### Use Observability Plugins

- Enable prometheus plugin for tracing.
- Log response time and token usage for cost control.

## Real-World Use Case: LLM Gateway for Multi-Agent System

**Scenario:**

- AI platform using OpenAI GPT-4, fallback to DeepSeek.
- Custom token quota per user.
- Streaming response to frontend.

**Benefits from API Gateway:**

- Reduce load on primary upstream.
- Improve user experience by retrying automatically.
- Add analytics based on token consumption.

## Conclusion

As LLMs become critical to production applications, API gateways play a pivotal role in managing LLM API traffic reliably, efficiently, and securely. With tools like Apache APISIX and its [LLM-compatible plugins](https://docs.api7.ai/hub#ai), engineers can implement token-based rate limiting, intelligent retries, streaming proxy, and failover across multiple providers such as OpenAI and DeepSeek.

By architecting the API gateway layer with these capabilities, teams can build resilient AI-powered systems with controlled cost, robust error handling, and excellent developer experience.

## Next Steps

Stay tuned for our upcoming column on the API gateway Guide, where you'll find the latest updates and insights!

Eager to deepen your knowledge about API gateways? Follow [our Linkedin](https://sg.linkedin.com/company/api7-ai) for valuable insights delivered straight to your inbox!

If you have any questions or need further assistance, feel free to contact [API7 Experts](https://api7.ai/contact).

[Previous article**How API Gateways Enable Serverless Architecture**](https://api7.ai/learning-center/api-gateway-guide/api-gateway-serverless-architecture)[Next article**How API Gateways Enhance MCP Servers**](https://api7.ai/learning-center/api-gateway-guide/api-gateway-enhance-mcp-server)

On this page

- [Introduction](#introduction)

- [Quick Answer](#quick-answer)

- [Core Concepts and Challenges of Proxying LLM APIs](#core-concepts-and-challenges-of-proxying-llm-apis)

  - [LLM-Specific API Characteristics](#llm-specific-api-characteristics)
  - [Challenges in Proxying LLM Requests](#challenges-in-proxying-llm-requests)

- [Architectural Overview: API Gateway as LLM Proxy](#architectural-overview-api-gateway-as-llm-proxy)

  - [Key Responsibilities of API Gateway for LLM](#key-responsibilities-of-api-gateway-for-llm)
  - [Request Lifecycle Through API Gateway](#request-lifecycle-through-api-gateway)

- [Using Apache APISIX to Proxy LLM Requests](#using-apache-apisix-to-proxy-llm-requests)

  - [Key Plugins](#key-plugins)
  - [Example: Proxy to OpenAI](#example-proxy-to-openai)
  - [Failover Between OpenAI and DeepSeek](#failover-between-openai-and-deepseek)

- [Load Balance between LLM Instances](#load-balance-between-llm-instances)

- [Best Practices for Proxying LLM APIs](#best-practices-for-proxying-llm-apis)

  - [Use Token-Aware Rate Limiting](#use-token-aware-rate-limiting)
  - [Enable Retry with Backoff](#enable-retry-with-backoff)
  - [Preserve SSE/Streaming](#preserve-ssestreaming)
  - [Differentiate Providers](#differentiate-providers)
  - [Use Observability Plugins](#use-observability-plugins)

- [Real-World Use Case: LLM Gateway for Multi-Agent System](#real-world-use-case-llm-gateway-for-multi-agent-system)

- [Conclusion](#conclusion)

- [Next Steps](#next-steps)
