APISIX Empowers Network for a Huge CDN Cloud Service Provider

Overview

About UPYUN

UPYUN, a leading Chinese enterprise cloud service provider, specializes in scenario-based CDN (Content Delivery Network) solutions. With 10 data processing centers, over 1000 domestic CDN nodes, nearly 100 global CDN nodes, 40,000 servers, and 10TB of reserved bandwidth, UPYUN provides secure solutions for various scenarios, handling a daily request volume exceeding 150 billion.

Challenges

• UPYUN faced challenges managing traffic connections with Kong, which functioned as its gateway for the public network.

• In terms of the ingress controller, it presented problems for UPYUN with Ingress-Nginx including complex component dependencies, poor portability, and weak semantic capabilities, posing difficulties in gateway maintenance.

Results

• APISIX stands out in handling traffic, fulfilling requirements for backup and traffic load through its dynamic routing capability. UPYUN has also enhanced component uniformity with the help of APISIX, resulting in elevated efficiency in traffic processing and logging.

• Through the use of APISIX plugins, UPYUN has achieved the successful implementation of employee identity authentication and integration with Lark, thereby improving the functionality of its gateway for the public network.

• APISIX has advanced monitoring and health checks for UPYUN's system by supporting Prometheus and SkyWalking.

Background

UPYUN's Business Characteristics

The gateways are indispensable in UPYUN's business ecosystem. It can coordinate, optimize, and ensure the smooth operation of various services, providing UPYUN with efficient, secure, and stable solutions. Therefore, UPYUN has been working on improving its internal gateway architecture.

UPYUN's Business Scenarios

Internally, UPYUN employs APISIX as its gateway for the public network and its ingress controller.

• The gateway for public network focuses on processing internet traffic, serving as the gateway for external services.

• The ingress controller specializes in managing external access to services within the Kubernetes cluster, enabling flexible traffic routing and control through ingress resource configuration.

Pain Points Before Using APISIX

Before adopting APISIX, UPYUN utilized Kong as its gateway for public network deployment and employed Ingress-Nginx as the ingress controller. However, UPYUN swiftly encountered several issues.

• High Burden on Database Connections: Connecting to the PostgreSQL database within the Kong architecture at UPYUN can lead to a considerable number of connections, potentially degrading database performance.

• Intricaties in Controlling Connection System: Despite the addition of proxies on the front end, UPYUN faces challenges in effectively managing the connection system during system updates or restarts, adding complexity to system maintenance.

• Operational Complexity with Ingress-Nginx: Ingress-Nginx poses challenges for UPYUN in plugin development due to complex dependencies, hindering portability and increasing complexity in development and maintenance. Additionally, it exhibits weaknesses in semantic capabilities, impacting flexibility for intricate business scenarios. The requirement for a reload operation with each ingress rule modification introduces unfriendly aspects, especially for scenarios relying on persistent connections.

In navigating the complexities of maintaining existing logic, these issues presented UPYUN with a substantial challenge. Consequently, the team initiated a quest for more streamlined and advanced alternatives in gateways to enhance operational efficiency.

Why UPYUN Chose APISIX

Elevated Flexibility with Robust Plugin Architecture

Utilizing Apache APISIX's plugin system, UPYUN designed a suite of internal plugins, encompassing features such as internal permission system validation and precise rate limiting. This strategic initiative has not only empowered the company with more adaptable functionality customization but has also infused diverse internal support into the gateway clusters.

Stellar Stability

In the realm of cloud-based operations, stability takes precedence, especially for smaller companies or those with limited operations team members. Opting for APISIX as the gateway solution not only guarantees a stable user experience for external users but also effectively manages the operational costs of internal business deployment.

Supportive Open-Source Community

For the UPYUN technical team, deciding whether a gateway is an open source becomes a pivotal consideration. APISIX, being an open-source solution, leverages its vibrant community support, ensuring swift responses and resolutions to reported bugs.

Impressive Scalability

Diverging from closed-source software, APISIX's remarkable scalability offers developers a seamless way to adapt and integrate. The multi-language extension of APISIX, for instance, empowers UPYUN to craft additional functionalities tailored to their business needs in the context of business expansion. This feature not only significantly boosts UPYUN's development efficiency but also brings greater convenience for subsequent feature iterations and maintenance.

Seamless Integration and Streamlined Customization

UPYUN has integrated Ingress-Nginx in certain Kubernetes clusters. In the absence of a plugin system in Ingress-Nginx previously, they customized specific plugins. With a notable functional overlap between Apache APISIX and NGINX in the gateways of internal data centers and containerized environments, UPYUN is set to replace all previously utilized Ingress-Nginx containerized gateways with the Apache APISIX Ingress Controller. This step aims to harmonize components at the gateway level, mitigating potential redundancy in future development and operational endeavors.

Enhanced Reloading Support

APISIX Ingress Controller comes with reload support, elevating operational efficiency to new heights. This feature allows dynamic updates to configurations without service disruptions, enabling swift deployment and flexible operational maneuvers. The result is a substantial boost in operational efficiency, streamlining system maintenance for greater convenience and speed.

Implementation of APISIX

UPYUN's current internal gateway architecture is depicted in the following diagram. External traffic is initially channeled through Apache APISIX, then directed to the APISIX Ingress Controller via APISIX. Finally, it reaches the backend services for subsequent business processing.

Gateway for Public Network

Traffic Control

The public network gateway acts as the primary entrance for external traffic, shouldering a significant responsibility that necessitates precise control and management of every incoming flow into the internal data center. In this crucial aspect, APISIX has offered UPYUN a range of traffic control services through its powerful features.

To begin with, the APISIX gateway has managed traffic by handling API access from CDN edge nodes. This assistance has not only aided in optimizing the performance of CDN nodes but also laid the groundwork for subsequent traffic management. Additionally, APISIX adeptly has processed traffic related to the official website's static pages and technical support, showcasing its versatile traffic control capabilities. This targeted approach to traffic handling has empowered UPYUN to flexibly address diverse access requirements, ultimately boosting the overall efficiency of the system.

Identity Authentication

Plugins play a pivotal role within UPYUN, especially in the context of employee access to the internal platform. Currently, employees go through authentication using methods such as email and Lark. Thanks to the robust openid-connect plugin of Apache APISIX, seamless integration with these platforms has been achieved, facilitating convenient authentication of employee identities. This innovative application has vividly demonstrated the powerful capabilities of APISIX plugins, providing UPYUN with an efficient and unified identity authentication solution for accessing the management platform.

Intelligent Coordination and Security Protection

In more specific scenarios, UPYUN has leveraged the openid-connect plugin in conjunction with the serverless-post-function plugin to achieve intelligent coordination with the Feishu application. Through the collaborative action of these plugins, relevant user information, like username, email, or the unique identifier within the Lark, has been transmitted to the service after passing through the public network gateway. Once the gateway obtains the pertinent identifier information, it efficiently forwards it to the server, enabling functionalities such as Lark's notifications and mentions. Notably, during this process, the consumer-restriction plugin can also impose specific permissions restrictions on users, enhancing the overall system's security and controllability.

Ingress Controller

Internal Architecture Shift

After integrating Apache APISIX Ingress Controller, UPYUN's internal architecture now showcases the following structure.

Diverging from the previously mentioned Ingress-Nginx framework, the foundational data plane has been swapped with an Apache APISIX cluster. The upper-level controller actively monitors changes in the API Server, subsequently disseminating configuration resources across all nodes of the Apache APISIX cluster through etcd.

A pivotal distinction emerges in Apache APISIX's capability for dynamic route modifications, setting it apart from the Ingress-Nginx configuration on the right. In Apache APISIX, all incoming business traffic converges through a unified location, with route selection executed through Lua code. This efficiency results in a streamlined and easily manageable code deployment. On the contrary, the nginx.conf configuration file on the right for Ingress-Nginx is intricate, necessitating a reload operation for each Ingress alteration.

Dynamic Routing and Declarative Configuration

Capitalizing on Apache APISIX's dynamic routing capability, the Apache APISIX Ingress Controller has effectively implemented its functionalities. Its primary role revolves around monitoring resource changes within APIServer, executing meticulous data structure transformations, validation, and computing the crucial DIFF. The final step involves applying these changes to the Apache APISIX Admin API. Also, the Apache APISIX Ingress Controller introduces a high-availability solution directly through the Kubernetes leader-election mechanism, eliminating the need for external components.

In terms of declarative configuration, UPYUN has opted for the CRD Resource, appreciated for its robust semantics. This structured data configuration approach empowers the implementation of any capability supported by Apache APISIX.

Achievements After Using APISIX

Improving Traffic Management

Apache APISIX has significantly boosted UPYUN's traffic handling efficiency. Its robust backup mechanisms and outstanding traffic load-handling capabilities ensure UPYUN's stability, especially when dealing with large-scale traffic volumes. Through meticulous traffic control, APISIX effectively tackles the challenges associated with backup and traffic management, providing UPYUN with reliable business support and optimizing system performance.

Streamlining Integration and Authentication

APISIX excels in identity authentication and plugin integration. Using the openid-connect plugin, APISIX efficiently accommodates diverse authentication methods, offering UPYUN an effective identity authentication solution. The combined functionality of plugins such as serverless-post-function and consumer-restriction facilitates intelligent data transmission and robust access control, enhancing the system's overall integration capabilities significantly.

Enhancing Log Processing Efficiency

Regarding log processing, UPYUN has implemented multiple Apache APISIX clusters internally, ensuring unified usage across data center gateways and container gateways. This coherence establishes a standardized logic for subsequent log processing and consumption. Noteworthy is Apache APISIX's robust functionality for the log plugin. Internally, UPYUN has chosen the Kafka-Logger, a plugin that supports custom log formats.

Optimizing Monitoring and Health Checks

As for monitoring, UPYUN utilizes tools like Prometheus and SkyWalking, with Prometheus being the preferred option. Functioning as a fundamental proxy, Apache APISIX can monitor app status codes and requests.

Summary

As a prominent player in the CDN cloud services sector, UPYUN has integrated Apache APISIX into its infrastructure, particularly in the gateway for its public network and Ingress controller. The robust plugin system and exceptional traffic management capabilities of APISIX have been important in significantly enhancing operational quality and efficiency for UPYUN.

Looking ahead the road, UPYUN anticipates further collaboration with APISIX to empower scenario-based CDN, delivering a suite of services including cloud storage, cloud processing, cloud security, and traffic marketing.