Announcing API7 Enterprise 3.7.0

API7 Enterprise 3.7.0 brings key advancements for AI plugins, pushing closer to the realization of an 'AI Gateway.' This update focuses on improved integration and management of AI services within existing API infrastructures. This new version also enhances security by supporting Kubernetes secrets for sensitive data storage, such as certificates.

Rate Limiting Based on AI Tokens

Problem:

Due to the widespread adoption of token-based billing for large language models, uncontrolled API access to AI providers can cause significant, unexpected cost increases. Excessive token consumption by users or applications—often from inefficient prompts or runaway loops—degrades system performance and carries substantial financial consequences.

Directly tracking and controlling tokens at the AI provider level can be complex and inefficient, particularly when dealing with numerous concurrent requests and varying usage patterns.

Solution:

API7 Enterprise 3.7.0 introduces a new AI plugin, AI Rate Limiting, to enforce token-based rate limiting for requests sent to LLM services. This plugin is crucial for managing API usage by precisely controlling the number of tokens consumed within a specified time frame. This ensures fair resource allocation, prevents excessive load on the service, and is particularly valuable in scenarios where AI gateways charge based on token consumption.

Active Health Check in AI Proxy Multi Plugin

Problem:

When using multiple AI large language models concurrently, consistent service reliability is essential. Fluctuations in individual model health, including downtime and performance degradation, can severely compromise overall system stability, negatively affecting performance and user experience. This is especially critical in high-availability, real-time applications.

Solution:

By actively monitoring the health of upstream AI providers, the AI Proxy Multi plugin identifies and excludes unhealthy AI services based on health check results. This targeted exclusion of faulty services directly improves the success rate of client requests, ensuring consistent performance and minimizing disruptions.

Supported OpenAI-compatible Providers in AI Proxy and AI Proxy Multi Plugin

Problem:

In the rapidly evolving AI landscape, users often require access to diverse language models and AI functionalities. However, current systems may limit the integration of various AI providers, restricting user choice. Compatibility issues with emerging providers, particularly those using standards like the OpenAI API, can further hinder seamless adoption. This can prevent users from efficiently leveraging the latest AI advancements.

Solution:

This update significantly expands the range of language models and AI functionalities accessible to users. Specifically, it extends the provider field of ai-proxy and ai-proxy-multi to include the openai-compatible value, indicating support for providers that adhere to the OpenAI API standard.

Enhanced AI Proxy Plugin to Proxy Embedding APIs

Problem:

While AI advancements have largely centered on chat-based interactions, many applications require deeper integration of AI functionalities, such as semantic search and AI-driven data analysis. Current limitations often restrict users to chat APIs, hindering the full potential of AI in sophisticated data mining, contextual search, and dynamic information retrieval, thus impeding innovation across various application domains.

Solution:

Enabling users to seamlessly integrate powerful semantic search and AI-driven data analysis into their applications, not restricted to chat APIs.

Reference Secrets in Kubernetes Secret

While the secret provider has long offered secure management of sensitive data, the need for tighter integration with Kubernetes' native secrets management has grown. Existing methods may not fully leverage Kubernetes' robust security features, potentially leading to inconsistencies and added operational overhead. Users require a more streamlined and secure way to reference sensitive data within their Kubernetes deployments, aligning with established best practices for containerized environments.

Solution:

API7 Enterprise now supports Kubernetes secrets as a secret provider. This allows you to reference sensitive values from Kubernetes secrets for use in SSL certificates, SSL private keys, consumer credentials, and various plugin configurations. See Reference a secret in Kubernetes Secrets for details.

Getting started with API7 Enterprise 3.7.0

Want to learn more? Dive into our documentation for more technical details. See API7 Enterprise 3.7.0 Release Note

If you have any questions about API7 Enterprise 3.7.0, reach out to us on LinkedIn or X!