---
title: "How to Differential-Test LLM Routing Changes - API7.ai"
description: "Build a behavior contract for an LLM gateway router, compare old and new decisions, and catch retry, fallback, streaming, and side-effect regressions."
url: "https://api7.ai/blog/llm-routing-differential-testing"
---

[Blog](https://api7.ai/blog)

[Products](https://api7.ai/category/products)[Technology](https://api7.ai/category/technology)[Ecosystem](https://api7.ai/category/ecosystem)[Case Study](https://api7.ai/category/usercase)[Team & Community](https://api7.ai/category/community)

Search

⌘ + K

[Blog](https://api7.ai/blog)

Technology

# How to Differential-Test LLM Routing Changes

[![Yilia Lin](https://github.com/Yilialinn.png)](https://github.com/Yilialinn)

[Yilia Lin](https://github.com/Yilialinn)

November 16, 2026

[Technology](https://api7.ai/category/technology)

An LLM router can return the same final `200` response after making a very different decision. It may choose another deployment, add a retry, cross a provider boundary, consume a fallback, or start a stream before a later failure. A status-code comparison will miss those regressions.

**Differential testing** runs the same case against a reference router and a candidate router, normalizes their observable results, and compares the behaviors that must remain stable. It is especially useful for a rewrite, version upgrade, configuration migration, or policy refactor whose stated goal is “change the implementation without changing the caller contract.”

LiteLLM described this approach in its November 11, 2026 account of [moving routing to a Rust core](https://docs.litellm.ai/blog/router-rust): each behavior-preserving step received a differential test, and the comparison included more than the selected deployment. This article turns that engineering idea into a gateway-operator acceptance pack that works with any router whose decisions and upstream attempts can be observed.

## Define “Same Behavior” Before You Run Traffic

The old and new routers do not need identical internal data structures or log text. They need equivalent outcomes in the dimensions your applications and operators depend on.

Start with a contract table:

| Dimension                     | Compare when                                        | Example normalized value                          |
| ----------------------------- | --------------------------------------------------- | ------------------------------------------------- |
| Caller status                 | Every case                                          | `200`, `429`, `502`                               |
| Caller-visible model or route | The API exposes one                                 | `chat-prod`                                       |
| Selected target               | Routing choice matters                              | `primary-us`                                      |
| Attempt sequence              | Retry or fallback is enabled                        | `primary-us, primary-us, backup-eu`               |
| Terminal stream state         | The response streams                                | `complete`, `upstream_break`, `client_cancel`     |
| Upstream request shape        | Translation or passthrough may change               | normalized method, path, headers, and body fields |
| Policy result                 | Identity, budget, or guardrail applies              | `allowed`, `rate_limited`, `blocked`              |
| Side effects                  | The router writes cooldown, spend, or session state | normalized state delta                            |
| Latency and cost              | Only for a separately designed benchmark            | distribution and method, not one exact value      |

Do not put latency or cost into an equality oracle unless the experiment controls provider, model, cache, concurrency, geography, and pricing. A differential contract test is not automatically a performance benchmark.

## Use Four Pieces: World, Case, Channels, Oracle

A useful test harness has four explicit pieces:

1. **World:** the starting configuration and dependencies—deployments, aliases, health state, cooldowns, Redis rows, credentials, and mock upstream behavior.
2. **Case:** one ordered sequence of client requests and upstream responses.
3. **Channels:** the places where behavior is observed—client response, gateway access log, upstream capture, database or Redis state, and usage ledger.
4. **Oracle:** the rule that decides whether normalized reference and candidate results are equivalent.

This vocabulary helps prevent a common mistake: comparing only the channel that is easiest to collect. If the final client response matches but the candidate sends three upstream attempts instead of one, the caller channel passes while the attempt and cost channels fail.

![Pinned fixtures run through reference and candidate routers. Their client response, attempt sequence, selected target, policy result, and state changes are normalized before an oracle compares them. A negative example shows both routers returning 200 from the backup while the candidate makes an extra primary attempt, which is a regression.](https://static.api7.ai/uploads/2026/11/16/ZO6P3zjM_llm-routing-differential-pipeline.webp?imageMogr2/format/webp)

## Pin Both Sides of the Comparison

Record exact artifacts before collecting a result:

- reference gateway image, tag, or commit;
- candidate gateway image, tag, or commit;
- configuration and secret revision hashes;
- test-harness commit;
- mock-upstream version and scenario file;
- database and cache seed;
- client protocol and streaming mode.

For AISIX, also record whether the alias is a direct model, multi-target model, or semantic router, and whether the deployment is standalone or managed. Pin the reference and candidate environments to explicit releases or build identifiers, because retry, fallback, session-affinity, and telemetry behavior can change between versions.

Do not compare “old production” with “whatever main does today.” A difference without pinned inputs is not reproducible.

## Build the Minimum Case Matrix

Cover the branch points that can change a decision, not merely a list of successful prompts.

| Case                    | Upstream behavior                                   | What the oracle should inspect                                         |
| ----------------------- | --------------------------------------------------- | ---------------------------------------------------------------------- |
| Primary success         | First target returns a valid success                | Selected target, one attempt, caller contract                          |
| Retryable `5xx`         | First attempt returns `503`, next succeeds          | Same-target retry count, fallback boundary, final status               |
| Upstream `429`          | Target returns a rate limit                         | Behavior with `retry_on_429` off and on                                |
| Non-retryable `4xx`     | Target returns `400` or `401`                       | Error returned without spraying invalid traffic to more targets        |
| Connect failure         | Connection fails before response                    | Retry and fallback sequence                                            |
| First-byte timeout      | Stream has not committed bytes                      | Whether another eligible target can be attempted                       |
| Mid-stream break        | Bytes have already reached the caller               | Terminal stream state; no claim of transparent failover after delivery |
| All targets unavailable | Health or policy removes every target               | Configured `503` or documented try-anyway behavior                     |
| Caller not authorized   | Caller cannot use the alias or target               | Denial before upstream dispatch                                        |
| Cooldown transition     | Repeated failure crosses a health threshold         | State delta and next-request selection                                 |
| Alias or wildcard edge  | Requested name resolves indirectly                  | Resolved target and caller-visible name                                |
| Empty or unusual input  | Empty string, long value, Unicode, or missing field | Validation and stable error contract                                   |

Add cases for every custom policy: region tags, team overrides, price-based ranking, sticky assignment, semantic routes, provider-specific headers, or session state.

AISIX documents streaming retry boundaries that should become separate cases rather than one generic “streaming” row. In current routing documentation, a streaming request can move to another target before response bytes are committed but does not fail over after a stream has started. Verify the exact endpoint and version you operate.

## Normalize Before Comparing

Raw observations contain noise: request IDs, timestamps, generated response IDs, host ports, log order, and floating-point durations. Strip only fields that the contract declares irrelevant.

Keep values that reveal a semantic change:

- selected target and attempt order;
- status and structured error code;
- streamed terminal outcome;
- upstream method, path, model, tool fields, and material headers;
- policy decision and authenticated caller scope;
- cooldown, spend, or session-state changes.

Never normalize away a difference merely because it makes the test fail. If a generated ID is irrelevant, write that rule before viewing the candidate result.

## Run a Small, Auditable Comparator

The script below compares normalized JSON Lines produced by two test runs. Each line must contain a stable `case_id`. It ignores no fields by default; pass `--ignore` only for fields already classified as noise.

```
#!/usr/bin/env python3
import argparse
import json
import sys
from pathlib import Path

def load(path: Path):
    rows = {}
    for number, line in enumerate(path.read_text().splitlines(), 1):
        if not line.strip():
            continue
        row = json.loads(line)
        case_id = row.pop("case_id", None)
        if not case_id:
            raise ValueError(f"{path}:{number}: missing case_id")
        if case_id in rows:
            raise ValueError(f"{path}:{number}: duplicate case_id {case_id}")
        rows[case_id] = row
    return rows

def remove_path(value, dotted_path):
    parts = dotted_path.split(".")
    cursor = value
    for part in parts[:-1]:
        if not isinstance(cursor, dict) or part not in cursor:
            return
        cursor = cursor[part]
    if isinstance(cursor, dict):
        cursor.pop(parts[-1], None)

def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("reference", type=Path)
    parser.add_argument("candidate", type=Path)
    parser.add_argument("--ignore", action="append", default=[])
    args = parser.parse_args()

    reference = load(args.reference)
    candidate = load(args.candidate)
    failed = False

    for case_id in sorted(reference.keys() | candidate.keys()):
        if case_id not in reference:
            print(f"EXTRA candidate case: {case_id}")
            failed = True
            continue
        if case_id not in candidate:
            print(f"MISSING candidate case: {case_id}")
            failed = True
            continue

        left = reference[case_id]
        right = candidate[case_id]
        for path in args.ignore:
            remove_path(left, path)
            remove_path(right, path)

        if left != right:
            failed = True
            print(f"DIFF {case_id}")
            print("  reference:", json.dumps(left, sort_keys=True))
            print("  candidate:", json.dumps(right, sort_keys=True))

    if failed:
        return 1
    print(f"PASS: {len(reference)} cases are equivalent")
    return 0

if __name__ == "__main__":
    sys.exit(main())
```

Create a passing reference:

```
{"case_id":"primary-200","status":200,"attempts":["primary"],"target":"primary","terminal":"complete"}
{"case_id":"primary-503-fallback","status":200,"attempts":["primary","backup"],"target":"backup","terminal":"complete"}
```

Then introduce a candidate regression:

```
{"case_id":"primary-200","status":200,"attempts":["primary"],"target":"primary","terminal":"complete"}
{"case_id":"primary-503-fallback","status":200,"attempts":["primary","primary","backup"],"target":"backup","terminal":"complete"}
```

Run the comparison:

```
python3 compare_routes.py reference.jsonl candidate.jsonl
```

The command exits `1` and reports the extra primary attempt. That difference matters even though the final status and target match. After correcting the candidate fixture to use the valid JSON array `["primary", "backup"]`, the command exits `0` and prints `PASS: 2 cases are equivalent`.

The comparator has intentionally narrow scope. It verifies the normalized artifacts you provide; it does not collect gateway traffic, decide which fields matter, or prove model quality.

## Collect Multiple Channels Without Confusing Them

For each case, collect evidence at the point where it is authoritative:

- **Client:** final HTTP status, headers, response shape, stream frames, and terminal error.
- **Gateway:** authenticated caller, resolved alias, selected route, fallback reason, served target, and policy decision.
- **Upstream fixture:** every request received, arrival order, body, material headers, and programmed response.
- **State store:** cooldowns, session pins, rate-limit counters, or other expected changes.
- **Usage ledger:** request and attempt accounting, but only after defining what one row represents.

In AISIX 1.10.0, `x-aisix-served-by` is a debug header that requires `proxy.expose_debug_headers: true`, which is off by default and read at startup. Even when enabled, it appears only on qualifying successful `/v1/chat/completions` routing responses, not as a universal response field. Use `x-aisix-request-id` to correlate the request with the access log's `served_by_model`, or use the mock upstream capture, as the default serving-target oracle.

## Separate Equivalence from Intentional Change

An upgrade may contain both behavior-preserving work and deliberate product changes. Label each case:

- `must_match`: any difference is a regression;
- `expected_change`: reference and candidate should differ in a predeclared way;
- `informational`: record the result but do not gate the rollout yet.

For an expected change, write a directional oracle. If the candidate is supposed to stop retrying authentication errors, the test should require one `401` attempt and no fallback, not merely allow any difference.

Keep the old implementation as a reference only while it represents the desired contract. Differential testing can preserve an existing bug perfectly. Pair equivalence with specification tests for behavior that is meant to be correct.

## Add Noise Detection

Run the reference against itself twice before trusting a reference-versus-candidate difference. If two identical builds disagree, the case is nondeterministic or the normalizer is incomplete.

Common sources of noise include:

- weighted or random selection without a fixed seed;
- shared cooldown or rate-limit state from another case;
- unordered asynchronous logs;
- real provider responses;
- wall-clock TTLs and backoff;
- retries performed by the client SDK rather than the gateway.

Use isolated state, deterministic mock upstreams, fixed clocks or bounded time assertions, and one owner for retries. If randomness is the contract, compare distributions in a separate statistical test instead of forcing one request to match.

## Roll Out with a Bounded Shadow

After deterministic cases pass, replay a permitted, sanitized sample of representative request shapes through the candidate in a non-serving path. Do not duplicate sensitive prompts or side-effecting tool calls without explicit authorization and controls.

Compare normalized routing decisions and upstream request shapes. Keep candidate responses away from users until the mismatch classes are understood. A shadow run is evidence about the sampled workload, not proof that every production edge case is covered.

## Acceptance Checklist

Before switching routers or configurations, require all of the following:

- reference and candidate versions are pinned;
- the behavior contract names caller, attempt, stream, policy, and side-effect dimensions;
- every retry and fallback branch has a deterministic case;
- streaming tests distinguish before-first-byte and after-delivery failure;
- unauthorized and malformed requests prove no upstream dispatch;
- the reference-versus-reference run is stable;
- expected changes use directional assertions;
- mismatches have an owner and disposition;
- no result is presented as a cost, latency, or quality benchmark unless that separate method was run;
- rollback restores both router implementation and compatible state/configuration.

Differential testing is valuable because it makes “behavior-preserving” falsifiable. For an LLM gateway, the observable behavior is larger than the final response: target choice, attempts, policy, streaming terminal state, and side effects are all part of the contract. Capture them explicitly before a router rewrite, upgrade, or configuration migration reaches production.

For an AISIX rollout, start with the current [Routing and Failover documentation](https://docs.api7.ai/ai-gateway/routing/routing-and-failover) and adapt the matrix to the exact endpoint, strategy, and version you operate.

Tags:

[AI Gateway](https://api7.ai/tags/ai-gateway)

[LLM Routing](https://api7.ai/tags/llm-routing)

[Testing](https://api7.ai/tags/testing)

[Reliability](https://api7.ai/tags/reliability)

[Platform Engineering](https://api7.ai/tags/platform-engineering)

Share article link

Share

Related Posts

[![Claude Code Gateway: Govern Coding Agent Traffic](https://static.api7.ai/uploads/2026/10/08/QMnsXVp6_claude-code-gateway.webp)](https://api7.ai/blog/claude-code-gateway)

## [Claude Code Gateway: Govern Coding Agent Traffic](https://api7.ai/blog/claude-code-gateway)

[![AI Gateway High Availability: Survive Control-Plane Outages](https://static.api7.ai/uploads/2026/09/29/1wugfBAj_ai-gateway-high-availability.webp)](https://api7.ai/blog/ai-gateway-high-availability)

## [AI Gateway High Availability: Survive Control-Plane Outages](https://api7.ai/blog/ai-gateway-high-availability)

Table of Contents

- [Define “Same Behavior” Before You Run Traffic](#define-same-behavior-before-you-run-traffic)
- [Use Four Pieces: World, Case, Channels, Oracle](#use-four-pieces-world-case-channels-oracle)
- [Pin Both Sides of the Comparison](#pin-both-sides-of-the-comparison)
- [Build the Minimum Case Matrix](#build-the-minimum-case-matrix)
- [Normalize Before Comparing](#normalize-before-comparing)
- [Run a Small, Auditable Comparator](#run-a-small-auditable-comparator)
- [Collect Multiple Channels Without Confusing Them](#collect-multiple-channels-without-confusing-them)
- [Separate Equivalence from Intentional Change](#separate-equivalence-from-intentional-change)
- [Add Noise Detection](#add-noise-detection)
- [Roll Out with a Bounded Shadow](#roll-out-with-a-bounded-shadow)
- [Acceptance Checklist](#acceptance-checklist)

## Related Posts

[![AI Gateway Configuration Changes: Validate, Reload, Verify, and Roll Back](https://static.api7.ai/uploads/2026/11/09/dGL3lHkE_ai-gateway-configuration-change-runbook-v2.webp)](https://api7.ai/blog/ai-gateway-configuration-change-runbook)

Technology

November 9, 2026

#### [AI Gateway Configuration Changes: Validate, Reload, Verify, and Roll Back](https://api7.ai/blog/ai-gateway-configuration-change-runbook)

Use a four-gate runbook to validate an AI gateway configuration, apply it safely, prove the effective state, and roll back without confusing readiness with freshness.

[AI Gateway](https://api7.ai/tags/ai-gateway)

[AISIX](https://api7.ai/tags/aisix)

[Platform Engineering](https://api7.ai/tags/platform-engineering)

[![Incremental API Gateway Configuration: Rebuild Less, Verify More](https://static.api7.ai/uploads/2026/11/03/5tHRA5Tn_incremental-api-gateway-configuration.webp)](https://api7.ai/blog/incremental-api-gateway-configuration)

Technology

November 3, 2026

#### [Incremental API Gateway Configuration: Rebuild Less, Verify More](https://api7.ai/blog/incremental-api-gateway-configuration)

Apply targeted-build lessons to API gateway configuration: classify changes, combine work, retain full reconciliation, and prove equivalent output.

[API Gateway](https://api7.ai/tags/api-gateway)

[Platform Engineering](https://api7.ai/tags/platform-engineering)

[GitOps](https://api7.ai/tags/gitops)

[![Codex Proxy: Route OpenAI Codex Through AISIX](https://static.api7.ai/uploads/2026/10/13/9A4TPjrT_codex-proxy-aisix.webp)](https://api7.ai/blog/codex-proxy-aisix)

Technology

October 13, 2026

#### [Codex Proxy: Route OpenAI Codex Through AISIX](https://api7.ai/blog/codex-proxy-aisix)

Route Codex through AISIX with scoped credentials and a tested Responses contract, while preserving tools, streaming, continuation, and useful errors.

[AI Gateway](https://api7.ai/tags/ai-gateway)

[AISIX](https://api7.ai/tags/aisix)

[OpenAI Codex](https://api7.ai/tags/openai-codex)
