AISIX Rate Limit Policies: Tokens, Teams, and Budgets

Key Takeaways

• AISIX conditional rate limit policies match traffic by team, member, caller API key, model, model name, or provider.
• Policies can enforce requests per second, minute, hour, or day; tokens per minute or day; and concurrent requests.
• A policy can split counters by team, member, caller API key, model, or provider. A route is not an AISIX policy dimension.
• AISIX returns HTTP 429 when a matching limit has no remaining capacity; token counters settle after the response or stream reports usage.
• AISIX Cloud budgets are separate spend controls. They should not be described as APISIX plugins or API7 Enterprise policies.
• Use the AI Gateway Rate Limiting guide for the product-neutral design model. This article explains the AISIX implementation.

Start with the AISIX Policy Model

AISIX has two layers of rate limiting:

1. Limits attached directly to caller API keys or models.
2. Rate limit policies managed separately from those resources.

AISIX evaluates every applicable layer. A request proceeds only when the direct limits and every matching policy have capacity. For new configurations, the current AISIX documentation recommends conditional policies. Classic single-scope policies remain available for existing configurations.

A conditional policy answers three questions:

• Which requests match? Conditions select traffic by supported identity or model dimensions.
• How are counters divided? group_by creates independent quota buckets.
• How much can each bucket use? limits sets request, token, or concurrency caps.

The exact behavior in this article follows the AISIX 1.4 documentation. Check the AISIX rate limit policy documentation for the version you deploy.

Configure a Team Policy with Per-Member Buckets

The following conditional policy matches one team and gives each bound member an independent request bucket:

{
  "name": "team-member-default",
  "conditions": [
    {
      "dimension": "team",
      "operator": "in",
      "value": ["<team-id>"]
    }
  ],
  "group_by": ["member"],
  "limits": {
    "rpm": 20,
    "tpm": 100000,
    "concurrency": 4
  }
}

In AISIX Cloud, team, member, api_key, and model conditions use resource IDs. In an open-source AISIX resources file, API key and model conditions use the corresponding entry's display_name; team and member conditions match the caller key's team_id and user_id.

The example works only when the caller API key carries a member binding. If group_by references a dimension absent from the request, that request is not subject to the policy. Validate identity bindings before treating the policy as active.

Shared Team Quota and Individual Maximum

AISIX can evaluate several conditional policies for one request. To enforce both a shared team pool and a per-member maximum:

• Create one policy matching the team with no group_by; the team shares one bucket.
• Create a second policy matching the team with group_by: ["member"]; each member receives an independent bucket.

Both policies must have capacity. This avoids treating a per-member rule as a substitute for the team's total provider quota.

Understand When Counters Are Consumed

Request and token limits do not settle at the same phase.

Request Limits

A policy that uses only caller dimensions such as team, member, or API key reserves once at the request gate. If a conditional policy references a model property, AISIX evaluates it when the dispatched model is known.

For a request routed through a model group, semantic router, or ensemble, a model-aware policy can reserve capacity for each target attempt, including eligible retries and failovers. One caller request can therefore consume more than one request slot in a model-aware policy.

Token Limits

Token limits settle after a response or stream reports usage. They are not a guaranteed pre-request estimate of the final token count. Test token accounting for:

• Normal non-streamed responses.
• Completed streams.
• Client disconnects.
• Provider failures and partial attempts.
• Retries and failovers.
• Providers that omit usable token information.

Use request and concurrency limits alongside token limits when capacity must be protected before completion.

Verify the Failure Response

Configure and test one policy at a time with a caller and model that match its conditions. A rejected request on an OpenAI-compatible endpoint returns HTTP 429. For a conditional policy, the response identifies the policy that denied the request:

{
  "error": {
    "message": "request limit exceeded (requests) (policy 'team-member-default')",
    "type": "rate_limit_exceeded",
    "policy": {
      "id": "<policy-id>",
      "name": "team-member-default"
    }
  }
}

Anthropic-compatible endpoints preserve the Anthropic error envelope and do not include this policy object. Operators should therefore correlate the response with AISIX telemetry rather than assuming every protocol returns the same diagnostic fields.

Validate at least these cases before rollout:

1. A matching caller below the limit succeeds.
2. The same bucket exceeds the limit and receives 429.
3. Another member or model uses the intended independent bucket.
4. A request without a required grouping dimension behaves as documented.
5. Retries and failovers consume the expected model-aware counters.
6. Token usage appears after non-streamed and streamed completion.
7. Operators can distinguish an AISIX policy denial from an upstream provider 429.

Keep Budgets Separate from Rate Limits

AISIX Cloud budgets track or constrain spend over a defined period. They are separate from gateway rate limit policies and use their own supported scopes, including organization, environment, caller API key, provider key, team, member, or each member in a team.

Use rate limits for runtime request, token, and concurrency capacity. Use budgets for spend control and cost ownership. Confirm the required scope and enforcement behavior in the current AISIX budget documentation.

An application can receive a dedicated caller API key when application-level attribution is required. A URL route or display label does not automatically become a budget identity.

Preserve the Product Boundary

AISIX AI Gateway is a separate Rust-native AI gateway that can run as an open-source gateway or connect to AISIX Cloud. AISIX Cloud provides its management, budget, audit, and dashboard plane.

API7 Gateway and Apache APISIX have a different runtime and configuration model. Their route, Service, Consumer, Consumer Group, and plugin-instance scopes belong to APISIX-based plugins such as ai-rate-limiting; they are not AISIX policy dimensions. Do not copy APISIX plugin configuration into an AISIX policy or imply that AISIX is built on the API7 Enterprise control plane.

Operator Checklist

• Choose a direct resource limit or conditional policy deliberately.
• Use only documented AISIX match and bucket dimensions.
• Confirm caller keys carry the team or member bindings used by the policy.
• Test shared and per-identity buckets separately.
• Account for per-attempt consumption in model-aware routing policies.
• Pair post-response token accounting with pre-request request or concurrency controls.
• Verify 429 behavior for each supported inbound protocol.
• Keep provider quota failures distinct from AISIX policy denials.
• Configure AISIX Cloud budgets separately from rate limit policies.
• Re-test behavior after changing gateway versions, policy forms, or routing topology.

For the broader design decisions, continue with AI Gateway Rate Limiting. For product configuration and current schemas, use the AISIX documentation and evaluate AISIX AI Gateway.