Semantic Routing with AISIX: Intent, Cost, and Risk

Key Takeaways

• Semantic routing selects an LLM from the meaning of a request instead of relying only on static rules, round-robin distribution, or application-side model selection.
• A stable model alias keeps provider and model decisions out of application code, while the AI Gateway owns embeddings, thresholds, target selection, and fallback.
• Production accuracy depends on representative route examples, carefully tuned thresholds, a reliable default route, and continuous evaluation against real traffic.
• Semantic routing adds an embedding dependency and can be influenced by adversarial prompts, so input guardrails and explicit failure behavior are essential.
• AISIX exposes routing decisions through response headers and connects them to gateway logs, usage events, policies, and rate limits. AISIX Cloud adds centralized usage and cost views, budgets, and audit history.

AI applications rarely have one model that is best at every task. A strong reasoning model may be appropriate for contract analysis. A smaller multilingual model may handle translation at lower cost. A fast general model may be the right default for routine questions.

The simplest implementation asks the application to choose a model for every request. That works until model names, providers, prices, risk policies, and application teams multiply. The selection logic becomes duplicated across services, and a routing change requires application releases.

Semantic routing moves that decision into AISIX AI Gateway. Applications call one stable model alias. AISIX examines the request, estimates its intent, and sends it to an appropriate model under a centrally managed policy.

That sounds simple, but production semantic routing is not just an embedding demo. It requires route design, threshold tuning, failure handling, security controls, and observability. This guide explains those operational details and shows how AISIX implements them.

What Is Semantic Routing in an AI Gateway?

Semantic routing is a model-selection method that uses the meaning of a request as a routing signal. Instead of matching a URL, header, tenant ID, or explicit model name, the gateway converts text into an embedding and compares it with examples associated with each route.

Consider a gateway-facing alias named prod-chat. The application always sends that alias. Behind it, the platform team defines three possible outcomes:

• Legal and contract questions go to a high-accuracy reasoning model.
• Translation requests go to a lower-cost multilingual model.
• Everything else goes to a general-purpose default model.

This separates the application contract from the upstream model topology. The application does not need to know which provider serves each model, how the model is authenticated, or when a route changes.

Semantic Routing vs Static and Weighted Routing

Static routing uses deterministic metadata. A premium tenant might always use a premium model, or requests from one region might stay in that region. Weighted routing distributes traffic according to configured percentages. Round-robin routing spreads load. Failover selects a backup after an eligible failure.

Those mechanisms remain useful. AI Gateway load balancing is the right approach when targets are interchangeable or when availability is the main concern.

Semantic routing solves a different problem: targets are not interchangeable, and the content of the request determines which model is appropriate. It is closer to an intent classifier embedded in the request path.

The two approaches can coexist at different layers. A semantic route can select a model class such as legal-reasoning, while a routing group behind that target distributes traffic across equivalent provider deployments.

Semantic Routing vs Semantic Caching

Semantic routing and semantic caching both use embeddings and similarity scores, but they make different runtime decisions.

Semantic routing asks: which upstream model should generate a new response for this request? The result is a target selection, followed by a model call.

Semantic caching asks: is an earlier response similar enough, fresh enough, and authorized for this request? A cache hit may return that response without calling a model at all.

This article covers the first problem. Cache keys, response reuse, TTL, streaming replay, and cache isolation belong to semantic caching and require a separate policy design. Treating the two mechanisms as interchangeable can produce incorrect architecture and misleading cost estimates.

Why Applications Should Call a Stable Model Alias

A stable alias is more than a naming convenience. It is the boundary between application ownership and platform ownership.

Application teams own prompts, user experience, retrieval, and business behavior. Platform teams own provider credentials, approved models, routing, reliability, cost policy, and observability. An alias lets those responsibilities evolve independently.

The AISIX model resource represents that boundary. Callers send the alias they are allowed to use, while the gateway resolves it to a direct model, a multi-target routing policy, an ensemble, or a semantic router.

flowchart LR
    A[AI application] -->|model: prod-chat| G[AISIX AI Gateway]
    G --> K[Caller key policy]
    K --> S[Semantic router]
    S -->|legal intent| L[Reasoning model]
    S -->|translation intent| T[Multilingual model]
    S -->|no match| D[Default model]
    L --> O[Logs and usage]
    T --> O
    D --> O

How Semantic Routing Works

In AISIX, a semantic router is a model resource with a semantic configuration. It references an embedding model, a default target, match settings, and one or more routes.

For each chat request, the gateway performs four main steps.

1. Embed the Latest User Message

AISIX extracts text from the latest user message and sends it to the configured embedding model. When the message has several text parts, they are concatenated. Non-text content is ignored for the routing decision.

System messages, assistant history, and tool turns do not affect the current semantic match. This is an important design boundary. Route examples should represent what users actually ask, not hidden system instructions.

2. Compare Against Route Examples

Each route contains example utterances. AISIX embeds those examples when the configuration is created or changed, then caches the vectors in the data plane.

At request time, the gateway compares the prompt embedding with route example embeddings using cosine similarity. For each route, it keeps the highest score among that route's examples.

The steady-state request cost is therefore one prompt embedding call plus local vector comparisons. This is lighter than embedding every route example on every request, but it still adds latency and an upstream dependency that must be measured.

3. Apply Thresholds and Select a Target

A route matches only when its score reaches its effective threshold. A route-specific threshold overrides the router-level threshold. Among all routes that qualify, the highest score wins.

This prevents a weak similarity from forcing a specialized model selection. A legal route can use a higher threshold than a translation route if false positives are more costly.

4. Use a Default Route

If no route clears its threshold, AISIX sends the request to the configured default model. A good default is not an afterthought. It should safely handle mixed traffic and make ambiguous requests observable for later route tuning.

sequenceDiagram
    participant App
    participant Gateway
    participant Embed as Embedding model
    participant Target as Selected LLM
    App->>Gateway: POST /v1/chat/completions (prod-chat)
    Gateway->>Embed: Embed latest user message
    Embed-->>Gateway: Prompt vector
    Gateway->>Gateway: Score examples and apply thresholds
    alt Route matched
        Gateway->>Target: Forward to route target
    else No route matched
        Gateway->>Target: Forward to default model
    end
    Target-->>Gateway: Model response
    Gateway-->>App: Response + routing headers

Designing a Production Routing Policy

The quality of a semantic router depends more on policy design than on configuration syntax.

Separate Routes by Task, Cost, and Risk

Start with routes that have clear operational consequences. Useful dimensions include:

• Task: translation, coding, summarization, legal analysis, or customer support.
• Cost: routine tasks go to a smaller model; difficult tasks go to a stronger model.
• Risk: regulated or high-impact requests use an approved model and stricter policy.
• Capability: requests whose latest user text explicitly describes a multimodal or tool requirement use models verified for those features.

Avoid creating several routes whose example sets express nearly the same intent. Overlapping classes produce unstable boundaries and make threshold tuning difficult.

Semantic routing should also not replace deterministic policy. If a tenant is prohibited from using a provider, enforce that with identity-aware access policy. Do not rely on prompt meaning to satisfy a hard compliance requirement.

AISIX ignores non-text content when calculating the route. An attached image or tool definition cannot select a capable model by itself; express the requirement in the latest user text or use deterministic application or gateway metadata for that decision.

Choose an Embedding Model

The embedding model defines the geometry of the routing decision. It must match the configured vector dimensions and support the languages present in production traffic.

Cross-language matching is possible with a multilingual embedding model, but it must be tested with representative prompts. Do not assume that English route examples will reliably classify requests in every supported language.

The Sentence Transformers documentation provides useful background on semantic similarity and cosine scoring. In production, the relevant benchmark is still your own route set and traffic distribution.

Write Representative Examples

Examples should express the range of requests a route should accept. For a legal route, include contract review, liability analysis, and policy interpretation. For translation, include several phrasings and supported languages.

Do not put confidential production prompts into configuration. Use sanitized or synthetic examples that preserve intent.

Negative testing matters as much as positive examples. A question such as "translate this contract clause" may sit between translation and legal analysis. Decide the desired outcome, add it to a labeled evaluation set, and tune the policy accordingly.

Tune Thresholds

Thresholds are not universal constants. Scores depend on the embedding model, route examples, language, and prompt style.

Start with offline prompts labeled by expected route. Measure precision, recall, default-route rate, and confusion between route pairs. Raise thresholds when false positives are costly. Lower them only when missed routing opportunities are acceptable.

The AISIX Cloud dashboard provides Test routing to inspect route scores for a prompt and Auto-detect thresholds to estimate starting points from example geometry. For these helpers, the embedding endpoint must be reachable from the AISIX Cloud control-plane services. They accelerate setup, but they do not replace evaluation against live or production-like traffic.

Define Embedding Failure Behavior

An embedding service can time out, rate limit requests, or return errors. A semantic router therefore needs an explicit operational choice.

AISIX supports an embedding_timeout_ms value and three failure patterns through on_embedding_failure:

• default: send the request to the normal default model.
• fail: reject the request with 503.
• Explicit target: send the request to a designated safe model.

Use default for general workloads where availability is more important than specialization. Use fail when serving a default model after an embedding failure would violate the application's availability or quality contract. An explicit safe target is useful when degraded operation must stay within a tested model class. Authorization, data residency, and compliance requirements must still use deterministic policies independent of semantic matching.

Example Open-Source AISIX Semantic Router

The following YAML is an excerpt from the open-source AISIX gateway's resources.yaml file. Open-source configuration references models by display_name; AISIX Cloud uses resource IDs instead. Before loading this excerpt, define the referenced provider key, bge-m3 as an embedding model, and general-chat, safe-chat, legal-reasoning, and multilingual-small as direct models in the same resources file. The caller key must also allow the prod-chat alias.

models:
  - display_name: prod-chat
    semantic:
      embedding_model: bge-m3
      default: general-chat
      embedding_timeout_ms: 500
      on_embedding_failure:
        target: safe-chat
      match:
        distance_metric: cosine
        aggregation: max
        threshold: 0.75
      routes:
        - name: legal
          target: legal-reasoning
          threshold: 0.82
          examples:
            - analyze this contract for liability risk
            - review this NDA for unusual obligations
            - explain the indemnification clause
        - name: translation
          target: multilingual-small
          examples:
            - translate this paragraph to French
            - convert this email into Japanese

Use the current AISIX semantic routing documentation as the source of truth for both the open-source resources.yaml format and the AISIX Cloud Admin API schema.

Verify the Selected Route

After AISIX validates and reloads the resources file, call the stable alias exactly as an application would. The following request uses an OpenAI-compatible chat endpoint; replace the gateway URL and caller key with values from your AISIX environment.

curl --include --request POST 'https://<your-aisix-host>/v1/chat/completions' \
  --header 'Authorization: Bearer <caller-api-key>' \
  --header 'Content-Type: application/json' \
  --data '{
    "model": "prod-chat",
    "messages": [
      {
        "role": "user",
        "content": "Review this NDA and identify unusual indemnification obligations."
      }
    ]
  }'

Inspect the response headers rather than inferring the route from the answer:

• x-aisix-route identifies the named semantic route when one matched.
• x-aisix-served-by identifies the direct model that ultimately handled the request.

Repeat the request with positive examples, negative examples, ambiguous prompts, and prompts expected to use the default. A routing policy is ready for a canary only when those observed decisions match the labeled evaluation set.

Security, Reliability, and Observability

Embedding-based routing creates a policy input that users can influence. An adversarial prompt may imitate a route's examples to reach a model with different capabilities or cost.

Run input guardrails before semantic routing when the route affects sensitive access, cost, or behavior. Keep similarity scores as operator signals rather than exposing them as a control surface to untrusted callers. The OWASP Top 10 for LLM Applications is a useful baseline for prompt injection and excessive-agency risks.

Every routing decision should be observable. AISIX can return x-aisix-route when a named route matches and x-aisix-served-by for the direct model that served the request. Logs should also retain:

• Caller identity and requested alias
• Matched route or default outcome
• Resolved provider and model
• Embedding latency and failure status
• Model latency, tokens, cost, and error
• Guardrail decision and fallback attempts

Track route distribution over time. A sudden increase in the default route may indicate new user behavior, weak examples, an embedding change, or an attack. Compare cost and latency by route, not just by model. Connect those signals to AI Gateway observability so policy changes can be evaluated like software releases.

flowchart TD
    P[Production prompts] --> R[Routing decisions]
    R --> M[Metrics and logs]
    M --> E[Evaluate accuracy, cost, and latency]
    E --> X[Review confused and defaulted prompts]
    X --> C[Update examples or thresholds]
    C --> T[Test and canary]
    T --> R

When Semantic Routing Is the Right Choice

Use semantic routing when one application entry point serves meaningfully different tasks, and model choice can be inferred from user intent. It is especially useful when platform teams want to optimize cost without asking every application developer to build a classifier.

Do not use it when:

• A deterministic attribute such as tenant, region, or data classification must control the decision.
• Every request needs the same model and only availability differs.
• The route depends on full conversation state that the router does not evaluate.
• The additional embedding latency is unacceptable.
• No team owns evaluation, threshold tuning, and incident response.

In those cases, static routing, weighted routing, application-side orchestration, or a dedicated classifier may be more appropriate.

Conclusion

Semantic routing turns a stable model alias into an intent-aware model selection layer. It can send routine work to efficient models, specialized work to capable models, and ambiguous requests to a safe default without spreading provider logic across applications.

The production value comes from controls around the routing algorithm: caller identity, input guardrails, representative examples, tuned thresholds, explicit embedding failure behavior, and decision-level observability.

AISIX brings those controls into the same gateway path as model access, provider credentials, traffic policy, and usage telemetry. Start with two or three clearly separated routes, validate them against a labeled prompt set, and expand only when the operational evidence supports it.

Explore AISIX AI Gateway, then use the semantic routing guide to test a production model alias.