Migrating from LiteLLM to AISIX: Keep Model Profiles, Add Gateway Orchestration

A useful LiteLLM-to-AISIX migration has two stages: preserve the model profile an application calls, then put the model-specific rules and workflows you need into gateway configuration. The application keeps a stable model alias while operators manage the behavior behind it.

This practice gives the migration three concrete purposes:

• Centralize target-specific effort rules in model configuration, giving operators one place to maintain those translations.
• Manage model and access resources as a reviewable bundle, retaining the last valid snapshot when a malformed update is rejected.
• Put panel-and-judge orchestration behind one alias, so an application can make one request to a gateway-managed workflow.

A self-hosted gateway discussion and its configuration example supplied the starting problem: one Qwen model, several aliases, and sampling and reasoning settings bound to those aliases. We used that profile to validate the migration contract, then exercised the three AISIX capabilities above.

Validation scope. The examples use real gateways with an explicitly synthetic local upstream. They establish configuration and request behavior, not model quality, production availability, or a real Token bill.

Keep the application alias and move its model profile

Start with the calling contract the application already depends on. A LiteLLM alias can carry more than an upstream model name. The tested source profile included temperature, top_p, top_k, min_p, presence_penalty, repetition_penalty, reasoning_effort, and chat_template_kwargs.

Here is the LiteLLM input used for the low-effort alias. Other aliases, authentication, limits, and fallback targets are outside this excerpt:

{
  "model_list": [
    {
      "model_name": "qwen-thinking-low",
      "litellm_params": {
        "temperature": 1.0,
        "top_p": 0.95,
        "top_k": 20,
        "min_p": 0.0,
        "presence_penalty": 0.0,
        "repetition_penalty": 1.0,
        "reasoning_effort": "low",
        "chat_template_kwargs": {
          "enable_thinking": true,
          "preserve_thinking": true,
          "reasoning_effort": "low"
        },
        "model": "hosted_vllm/Qwen/Qwen3-0.6B",
        "api_base": "http://capture:18481/primary/v1",
        "api_key": "unused-local-key",
        "timeout": 60
      }
    }
  ]
}

AISIX separates the caller-facing model from its ProviderKey. The model selects an upstream ID and credential resource; the ProviderKey's request.default_body_fields supplies missing outbound fields. This loaded excerpt reproduced the alias-only defaults:

{
  "_format_version": "1",
  "provider_keys": [
    {
      "display_name": "local-low",
      "provider": "vllm",
      "adapter": "openai",
      "api_key": "unused-local-key",
      "api_base": "http://capture:18481/primary/v1",
      "request": {
        "default_body_fields": {
          "temperature": 1.0,
          "top_p": 0.95,
          "top_k": 20,
          "min_p": 0.0,
          "presence_penalty": 0.0,
          "repetition_penalty": 1.0,
          "reasoning_effort": "low",
          "chat_template_kwargs": {
            "enable_thinking": true,
            "preserve_thinking": true,
            "reasoning_effort": "low"
          }
        }
      }
    }
  ],
  "models": [
    {
      "display_name": "qwen-thinking-low",
      "provider": "vllm",
      "model_name": "Qwen/Qwen3-0.6B",
      "provider_key": "local-low",
      "timeout": 60000,
      "retries": 0
    }
  ]
}

These configuration excerpts omit the caller key and other lab targets, and use local lab addresses and dummy credentials. For another deployment, replace the endpoint, credential, and model with your own values.

This alias-only client body omits the lab's correlation field. The tested route was http://127.0.0.1:18402/v1/chat/completions, using the dummy lab caller key:

{
  "model": "qwen-thinking-low",
  "messages": [
    {
      "role": "user",
      "content": "What is 2 + 2? Answer briefly."
    }
  ],
  "max_tokens": 256,
  "stream": false
}

The ProviderKey schema defines this configuration, and the OpenAI adapter's send path applies it. In the alias-only test, AISIX resolved the alias to Qwen/Qwen3-0.6B and sent the configured sampling values and complete template object. Client scalar values overrode those defaults in the tested override request.

Defaults belong to the ProviderKey, so aliases sharing one resource share its request defaults. Our low- and extra-high-effort profiles used separate ProviderKey resources pointing at the same local upstream. Account for that organization when managing credential rotation and budget scopes.

Verify the calling contract before cutover

The local lab pinned LiteLLM v1.106.6 and AISIX v1.8.0. Gateways ran sequentially with host ports bound to loopback. The upstream returned SYNTHETIC FIXTURE and fixed usage; outbound requests, responses and logs were recorded. The LiteLLM starting configuration used single-target aliases. AISIX completed 21 migration cases and six additional feature cases.

A separate Qwen3-0.6B attempt downloaded the pinned weights but hit its 4 GiB container limit during loading and was stopped without raising memory. No inference completed. The original discussion's Qwen3.8-27B template therefore remains a source reference, not a model whose reasoning behavior was tested here.

The compatibility details fit into a short acceptance table:

The null row is a real difference in the captured requests; this setup does not preserve LiteLLM's omission behavior. Handle such inputs before cutover. The nested-object result also means these profiles do not recursively merge missing template members.

Both gateways enforced the tested 1 RPM limit and performed controlled fallback. With AISIX, injected 503, 429, and an eight-second delayed primary each led to one primary and one backup attempt; the clean timeout case used a five-second model timeout. The backup retained the primary's complete outbound body. The accepted timeout result came from a clean rerun after a capture-tool file race invalidated the initial trial. These are acceptance checks for the selected profiles.

Centralize target-specific reasoning rules

When client and target-model effort labels differ, keep their translation in the gateway's model configuration. AISIX's model-level effort_mapping applies the rule after resolving the final target. Operators can maintain that target-specific policy in one place. The model schema and request mapping code define missing-value, exact-match, wildcard, and removal behavior.

We tested this separate model using a ProviderKey with no body defaults:

{
  "models": [
    {
      "display_name": "qwen-effort-mapped",
      "provider": "vllm",
      "model_name": "Qwen/Qwen3-0.6B",
      "provider_key": "local-raw",
      "timeout": 60000,
      "retries": 0,
      "effort_mapping": {
        "": "low",
        "high": "xhigh",
        "*": "medium"
      }
    }
  ]
}

The wildcard in this example is an explicit operator policy: other non-empty string effort values become medium. Matching happens once, so the mapped result is not looked up again. It is useful when client vocabularies and target expectations differ; it is not transparent preservation of every caller's intent.

For example, with thinking enabled the pinned original Qwen template accepts low, medium, and xhigh, while rejecting high. A gateway mapping can express that vocabulary adjustment. Our test proves the outgoing value, not the template's execution or answer quality. Nested template effort remains a separate input and needs its own policy.

Review and reload model resources as one bundle

Once model profiles are in the gateway, operators need a way to maintain them. AISIX's file mode puts model profiles and access resources in one reviewable bundle. The gateway validates that file and reloads it on SIGHUP, giving teams a concrete configuration-change workflow for a small self-hosted deployment. The fixed-version quickstart documents the file mode.

The lab mounted the resource directory so host-side file replacement remained visible in the container. We added an allowed alias, reloaded the file, and received 200 from that alias. We then supplied malformed configuration and signaled another reload. AISIX logged rejection of the update, and the newly added alias still returned 200 using the last valid snapshot. The gateway process ID and container restart count remained unchanged.

This gives operators a tested update path that retains a working snapshot after a malformed change. The serial checks are not a continuous-load zero-downtime measurement. Larger deployments can choose another configuration source; this experiment did not test their control plane, configuration propagation, or outage behavior.

Let one alias own the panel-and-judge workflow

For applications that need multiple candidate answers and a synthesis step, AISIX's built-in ensemble model puts that calling sequence behind one model name. The application submits one request; gateway configuration describes the panel models and the judge. This moves the fan-out and synthesis handoff into the gateway. The executor includes per-member sampling overrides, a minimum successful response count, and deadlines.

The tested ensemble excerpt was:

{
  "models": [
    {
      "display_name": "qwen-ensemble",
      "ensemble": {
        "panel": [
          {
            "model": "panel-a",
            "temperature": 0.2
          },
          {
            "model": "panel-b",
            "temperature": 0.8
          }
        ],
        "judge": {
          "model": "judge"
        },
        "min_responses": 2,
        "timeout_ms": 5000
      }
    }
  ]
}

The three referenced direct models must exist first. In this non-streaming experiment they pointed at synthetic upstream IDs, not three real AI models. The test produced two panel calls with temperatures 0.2 and 0.8, followed by one judge call at 0.2. Its synthesis prompt included both panel responses. One client request therefore produced three upstream calls in this successful case.

This workflow has its own parameter contract: the client's max_tokens value of 64 reached both panels but was absent from the judge request. The pinned executor constructs the judge request separately and sets its temperature to 0.2. Verify a judge-specific output cap before deployment. The configured timeout_ms is a per-subcall deadline in this implementation; 5000 does not cap the complete panel-plus-judge workflow at five seconds. The successful fixture case did not exercise that deadline.

Each synthetic response carried 10 input Tokens and 2 output Tokens. AISIX returned aggregate usage of 30 input, 6 output, and 36 total Tokens for the ensemble. This checks orchestration and counter aggregation. It does not establish better answers, real Token accuracy, or lower cost; a real panel and judge add provider work that must be priced and evaluated.

The application-facing gain is one alias for this configured workflow. Panel selection and the judge handoff can be maintained with the gateway models, while the application continues to make a single model request. Adopting the workflow still requires accepting its separate judge policy and evaluating the real models it will call.

Move the profile, then adopt the gateway work you need

Use the migration to establish a stable application-facing profile first. Then adopt the gateway capabilities that match the work you want to centralize: target-specific effort rules, reviewable resource updates, or a panel-and-judge workflow. The examples above demonstrate those configuration and request paths under the stated lab conditions.

Keep the cutover checklist short:

• Confirm every alias's upstream model, defaults, and client-override rules, including the observed null difference.
• Verify authentication, quota rejection, and every fallback attempt's parameters.
• Reconcile requested aliases, served targets, and usage; finish pricing and persistence checks separately.
• Verify panel and judge parameter limits separately, then evaluate real-model quality and cost before enabling this workflow for production traffic.

Before cutover, replay these profiles against your own upstream. Keep the application alias stable, validate the effective parameters, and add the gateway-managed rules and orchestration that your application needs. That is the practical outcome to seek from this migration.