Multi-LLM Routing with Qwen and Cloud Models

Multi-LLM routing sends each AI request to a model endpoint selected by an explicit policy. The endpoint can be a hosted provider, a self-hosted open-weight model such as Qwen, or a fallback used when the preferred endpoint is unavailable or over quota.

The goal is not to guess a universally "best" model. It is to make model selection observable, testable, and reversible while applications use a stable gateway endpoint.

Key Takeaways

• Choose models with workload-specific evaluations, not provider benchmark headlines alone.
• Treat self-hosted inference as an operated service with hardware, availability, security, and staffing costs.
• Start with deterministic routing and fallback before adding semantic or cost-aware decisions.
• Keep routing policy separate from prompts and application business logic.
• Measure quality, latency, token usage, errors, and fallback behavior for every route.
• Do not claim savings until you compare fully loaded costs under representative traffic.

Why Qwen Makes Multi-LLM Routing Practical

The Qwen family illustrates why model routing has become an infrastructure concern. The official Qwen3.5 release introduced an open-weight multimodal model, while the Qwen organization publishes multiple model families, sizes, and deployment resources. Teams can deploy supported models through inference servers that expose an OpenAI-compatible API, or consume hosted model APIs.

That flexibility creates useful choices, but it does not make the choices interchangeable. Models can differ in:

• response quality for a specific task and language;
• context limits and multimodal support;
• latency and throughput on available hardware;
• tool-use and structured-output behavior;
• data residency and logging boundaries;
• hosted token prices or self-hosted operating costs.

A coding assistant, a low-latency classifier, and a regulated document workflow may therefore need different endpoints. Multi-LLM routing gives the platform team one place to express those decisions without hard-coding provider URLs and credentials into every application.

What an AI Gateway Should Decide

An AI gateway can select an endpoint, apply traffic policy, and record request-level signals. It should not silently invent quality or compliance decisions that the organization has not defined.

Cost and quality usually require information beyond a single request. A gateway can enforce a policy derived from offline evaluations, but it cannot prove that the cheapest endpoint is good enough without a task-specific quality signal.

Reference Architecture

flowchart LR
    A[AI application] --> G[AI gateway]
    G --> P[Routing policy]
    P --> Q[Self-hosted Qwen endpoint]
    P --> C[Cloud model endpoint]
    P --> F[Fallback endpoint]
    Q --> O[Metrics and logs]
    C --> O
    F --> O
    O --> E[Evaluation and policy review]
    E --> P

This architecture separates four responsibilities:

1. Applications send a supported request format to one gateway endpoint.
2. Routing policy selects an approved model instance from trusted request context or semantic classification.
3. Model endpoints remain responsible for inference behavior, capacity, and provider-specific limits.
4. Evaluation uses production telemetry and controlled test sets to decide whether the routing policy should change.

The gateway is on the request path, while evaluation is a separate feedback loop. Avoid running expensive quality comparisons synchronously for every production request unless the use case explicitly requires that cost and latency.

Build the Routing Policy in Stages

1. Define the Workload Contract

Start with the application requirements before choosing a model:

• accepted input and output formats;
• maximum latency and timeout behavior;
• minimum quality on a representative evaluation set;
• residency, retention, and logging restrictions;
• required tool calling, structured output, or multimodal behavior;
• expected volume, concurrency, and context size.

This contract turns "use Qwen for cheaper requests" into a testable rule such as "route English classification requests with no restricted data to the approved local endpoint when its p95 latency and evaluation score remain within the agreed thresholds."

2. Expose the Self-Hosted Model Safely

Deploy the selected Qwen model with a supported inference server and confirm the exact API, model identifier, context limit, and hardware requirements from its current model card. The official Qwen documentation lists deployment options and OpenAI-compatible serving paths for supported frameworks.

Treat the inference server as a production upstream:

• keep it on a trusted network;
• authenticate gateway-to-model traffic where the server supports it;
• set request, connection, and upstream timeouts;
• limit concurrent work to protect GPU memory;
• monitor queue depth, failures, token usage, and latency;
• define what happens when the endpoint is unavailable.

Running open weights does not make inference free. Include hardware acquisition or rental, idle capacity, electricity, storage, observability, upgrades, and on-call ownership in the cost model.

3. Configure Deterministic Routing First

Apache APISIX provides the ai-proxy-multi plugin for multiple LLM instances. It supports documented providers and OpenAI-compatible endpoints, along with weighted load balancing, consistent hashing, semantic routing, retries, fallback, configurable active health checks, and LLM request summaries.

A production configuration can register a self-hosted Qwen endpoint as openai-compatible and a hosted model as another instance. Keep credentials in the deployment secret system rather than committing them with the route configuration.

The following excerpt shows the shape of a local primary endpoint and a lower-priority hosted fallback. Replace the endpoint, model names, and secret references with values verified for your deployment:

plugins:
  ai-proxy-multi:
    fallback_strategy:
      - http_429
      - http_5xx
    max_retries: 1
    instances:
      - name: local-qwen
        provider: openai-compatible
        priority: 1
        weight: 0
        auth:
          header:
            Authorization: "Bearer ${LOCAL_MODEL_TOKEN}"
        options:
          model: your-approved-qwen-model
        override:
          endpoint: "http://qwen-inference:8000/v1/chat/completions"
      - name: hosted-fallback
        provider: openai-compatible
        priority: 0
        weight: 0
        auth:
          header:
            Authorization: "Bearer ${HOSTED_MODEL_TOKEN}"
        options:
          model: your-approved-hosted-model
        override:
          endpoint: "https://provider.example/v1/chat/completions"

This is a configuration excerpt, not a complete deployment. Validate it against the APISIX version you run and the exact API contract exposed by each model endpoint.

4. Choose Semantic Routing Only When Its Failure Semantics Fit

APISIX also supports semantic routing. Each model instance has example prompts, and an embedding model compares the incoming prompt with those examples. A configured threshold and fallback determine where ambiguous prompts go.

Semantic routing is a separate balancer mode, not an extension of the priority-based HTTP fallback shown above. When balancer.algorithm is semantic, APISIX does not use active health checks, fallback_strategy, or retries for an upstream failure; the selected instance's failure is returned to the client. semantic_opts.fallback applies only when no instance clears its similarity threshold or the embedding request fails. Do not expect the 429 and 5xx provider-error failover from the previous stage to remain active after switching to semantic routing.

Use semantic routing only when prompt intent is a meaningful model-selection signal and these failure semantics fit the workload. It introduces another model call and another policy to calibrate. Build a labeled test set, inspect false matches, and configure semantic_opts.fallback for unmatched or unscorable prompts. Debug scores can expose instance names, so use debug response headers only during controlled testing rather than in production.

5. Bound Fallback Behavior

Fallback improves resilience only when its semantics are understood. A backup model may differ in output format, tool support, safety behavior, latency, or cost.

Define:

• which failures permit a retry;
• the maximum number of additional attempts;
• an overall request deadline;
• whether streaming can be retried safely;
• which endpoints are contract-compatible;
• how clients learn that a fallback occurred, if that matters to them.

Avoid retrying unboundedly across every model. It increases latency and can multiply spend during an outage.

Measure Routing Outcomes

Track enough information to explain each decision without logging sensitive prompts by default.

Payload logging can expose prompts, credentials, personal data, or model output. Log summaries by default and enable payload capture only with an approved retention, access, and redaction policy.

Common Failure Modes

Routing Only by Model Price

Published token prices do not capture self-hosted infrastructure, retries, cache effects, long contexts, or human review caused by lower-quality output. Optimize against a workload-level cost and quality objective.

Trusting Provider Benchmarks as Acceptance Tests

Benchmark results describe a model under a particular evaluation setup. Build a representative internal test set and keep the prompts, expected behavior, and scoring method versioned.

Using Untrusted Client Headers for Sensitive Routing

A client-controlled header should not be allowed to bypass residency or security policy. Derive sensitive routing attributes from authenticated consumer identity or a trusted policy service.

Assuming Every Endpoint Is Interchangeable

OpenAI-compatible transport does not guarantee identical tool calls, reasoning fields, multimodal input, streaming, or error behavior. Test the exact features your application uses.

Hiding Fallback and Retry Costs

A successful final response can conceal repeated upstream failures. Record attempts and fallback decisions so availability does not mask latency and cost problems.

Multi-LLM Routing Checklist

• Define workload-specific quality, latency, security, and cost requirements.
• Verify each model and inference server against current official documentation.
• Use a stable gateway contract for applications.
• Start with deterministic routing and one bounded fallback.
• Protect credentials and strip headers that must not reach model providers.
• Test output, tool, streaming, and error compatibility.
• Measure model selection, tokens, latency, retries, and quality.
• Review routing policy against versioned evaluation results.
• Recalculate hosted and self-hosted costs from observed traffic.
• Maintain a rollback path when a model or policy changes.

Next Steps

For a broader implementation path, read the AI Gateway deployment and operations guide. For a separate AI-native gateway product with its own configuration model, explore AISIX AI Gateway. For the Apache APISIX plugin configuration used in this article, follow the current ai-proxy-multi documentation.