Building AI Agents Resilience: Auto-Switch Between OpenAI & DeepSeek via API Gateway

Key Takeaways

• AI applications relying on OpenAI or DeepSeek must handle API failures gracefully to ensure uninterrupted service.
• An API gateway like Apache APISIX can act as a load balancer and failover mechanism, improving the resilience of AI-driven systems.
• DeepSeek's API is OpenAI-compatible, enabling seamless integration with Apache APISIX's OpenAI plugin.
• A practical configuration guide for setting up Apache APISIX to auto-switch between OpenAI and DeepSeek.

The Rise of DeepSeek and the Need for AI Resilience

DeepSeek has recently gained significant attention as a powerful LLM provider, competing with OpenAI in terms of performance and API compatibility. However, both OpenAI and DeepSeek have experienced service disruptions, causing issues for AI applications that depend on them.

For AI agents and applications, resilience is critical. Service interruptions can degrade user experience, disrupt business processes, and result in downtime. A robust failover strategy is essential to maintain continuous access to LLM services.

Using an API Gateway for Automatic Failover

An API gateway can serve as a centralized traffic management layer, directing requests to multiple upstream services. In the case of OpenAI and DeepSeek, the gateway can monitor availability and automatically switch between providers when one fails.

Apache APISIX, an open-source API gateway, allows for such dynamic upstream switching. With its OpenAI plugin and built-in load-balancing capabilities, it can intelligently route requests based on real-time health checks and failover strategies.

Benefits of Using an API Gateway for LLM Failover

• High Availability: Ensures AI agents remain operational even when one provider is down.
• Automatic Switching: Detects API failures and seamlessly redirects traffic.
• Load Balancing: Distributes requests efficiently across multiple upstreams.
• Security & Observability: Enhances API security and monitoring.

Configuring Apache APISIX for OpenAI & DeepSeek Failover

Since DeepSeek's API is fully compatible with OpenAI's, integrating both under Apache APISIX is straightforward. Below is a step-by-step guide to configuring Apache APISIX for automatic failover.

Step 1: Define LLM Service and Route

services:
  - name: LLM Service
    routes:
      - uris:
          - /v1/chat/completions
        name: llm-chat

Step 2: Set LLM API key by plugin

services:
  - name: LLM Service
    routes:
      - uris:
          - /v1/chat/completions
        name: llm-chat
        plugins:
          proxy-rewrite:
            headers:
              set:
                Authorization: LLM-API-KEY

Step 3: Configure Upstreams and Failover Logic

services:
  - name: LLM Service
    routes:
      - uris:
          - /v1/chat/completions
        name: llm-chat
        plugins:
          proxy-rewrite:
            headers:
              set:
                Authorization: LLM-API-KEY
    upstream:
      scheme: https
      type: roundrobin
      nodes:
        - host: api.openai.com
          port: 443
          weight: 100
        - host: api.deepseek.com
          port: 443
          weight: 1
          priority: -1
      checks:
        active:
          http_path: /
          healthy:
            successes: 1
          unhealthy:
            http_failures: 3
        passive:
          healthy:
            successes: 1
          unhealthy:
            http_failures: 3

With this setup, Apache APISIX will automatically redirect traffic to DeepSeek if OpenAI's API becomes unavailable, ensuring minimal disruption.

The magic occurs when the priority of DeepSeek is set to -1, which means that this service is for backup purposes and will automatically take effect when the OpenAI service is unavailable.

Conclusion

Implementing an API gateway-based failover strategy is a crucial step toward enhancing the resilience of AI-driven applications. Apache APISIX provides a flexible and efficient solution to automatically switch between OpenAI and DeepSeek, reducing downtime and improving reliability.

For developers looking to build robust AI systems, integrating Apache APISIX with OpenAI and DeepSeek is a practical and scalable approach. Try Apache APISIX today to ensure uninterrupted AI operations!