How Apache APISIX Unifies 360's L7 Load Balancing

Overview

About 360 and 360 ZYUN Cloud

360 Security Technology Inc., also branded as Qihoo 360, is a leading Internet and security technology enterprise and the first advocate of free Internet security in China. In the wave of AI, it is committed to helping industries and organizations achieve intelligent digital transformation.

360 ZYUN Cloud is the intelligent data cloud foundation of 360, offering a wide range of products and services, including databases, middleware, storage, big data, artificial intelligence, computing, networking, video, and IoT integration, and communications, as well as one-stop solutions.

Positioned as an open enterprise application service platform with the mission of "aggregating the value of data and enabling an intelligent future," 360 ZYUN Cloud provides robust products and technical services for businesses and applications across industries, helping enterprises and organizations unlock greater commercial value.

Challenges

Within 360, there existed two main challenges that needed to be solved.

• VPC Incompatibility: Its classic network design was obsolete and unsuited for modern cloud tenants who require isolated VPC environments.

• Non-Cloud-Native: It lacked support for containerized backends (e.g., Kubernetes Pods), severely limiting its use with modern application architectures.

Results

• Cost and Resource Efficiency: Eliminated the need for new hardware by reusing existing L4 infrastructure, reducing both deployment costs and operational overhead.

• Unified Multi-Environment Management: Achieved a consistent control plane for managing traffic across containers, virtual machines, and bare-metal servers, simplifying maintenance and improving scalability.

• Real-Time Performance and Reliability: etcd's watch mechanism enables instant, consistent configuration updates across all nodes, enhancing reliability and ensuring high-performance traffic handling.

• Enhanced Operational Simplicity: Reduced deployment risk and streamlined governance by leveraging existing security policies and maintenance processes.

Background

Layer 7 (L7) load balancing operates at the application layer of the OSI model, supporting in-depth parsing of protocols such as HTTP/HTTPS and enabling a wide range of advanced forwarding rules. Unlike Layer 4 (L4) load balancing, which solely focuses on IP addresses and ports, L7 load balancing can interpret application-layer protocol content, allowing for more granular traffic control.

Within 360, there existed two main challenges that needed to be solved.

1. Lack of VPC Support: The previous load balancer was designed for classic network architecture and did not support Virtual Private Cloud (VPC). This makes it incompatible with and unable to meet the requirements of cloud-based tenants who require isolated, customizable network environments.

2. Lack of Cloud-Native Support: The backend server groups can only consist of bare metal servers and virtual machines. This architecture is missing critical support for cloud-native scenarios (e.g., containers, Kubernetes pods, serverless functions), limiting its applicability in modern, containerized application environments.

Based on the challenges above, the Unified L7 Load Balancer project was initiated. It aims to provide a unified load balancer that supports hybrid network architectures and adapts to multiple backend instance types. The load balancer primarily offers the following features:

• Intelligent Routing Decisions: Achieves fine-grained traffic scheduling by parsing HTTP header fields (e.g., User-Agent, Cookie), URL paths, SSL session IDs, and other parameters.

• Session Persistence Mechanism: Ensures user requests are consistently directed to the same backend server by injecting Cookies or generating Session IDs.

• Active Health Checks: Proactively monitors the liveliness of backend upstreams to prevent request failures caused by upstream anomalies.

• Hot Reloading of Configurations: Enables real-time application of unified configuration changes, reducing configuration maintenance complexity.

• High Availability: Ensures instance reliability through cluster disaster recovery, session persistence, and multi-zone active-active mechanisms.

Why 360 Adopted Apache APISIX

During the evaluation process for the core component of their new unified L7 load balancer, 360 ZYUN Cloud conducted a comprehensive comparative analysis of the leading open-source solutions: NGINX, Envoy, and Apache APISIX.

The final decision to build upon Apache APISIX was driven by its superior architectural design and its unique alignment with 360's specific technical requirements and existing infrastructure. The key deciding factors were:

1. Dynamic Configuration Management with Hot Reloading

A critical limitation of traditional load balancers like NGINX is the requirement to reload the entire process to apply configuration changes (nginx -s reload). This process interrupts the forwarding plane, can cause performance loss during the reload, and is not truly dynamic.

Apache APISIX supports hot reloading. This means all configuration changes, including adding, modifying, or removing routes, upstreams, and plugins, are applied instantaneously without restarting any processes or dropping a single packet. This capability is essential for maintaining high availability and enabling agile, continuous deployment in a large-scale, dynamic cloud environment.

2. Seamless Integration with Existing etcd-Based Architecture

The choice of a configuration center is a fundamental architectural decision. Apache APISIX uses etcd by default as its configuration storage center and service discovery backbone. Since 360's infrastructure already uses etcd, the integration would be seamless. This was a decisive factor because:

• Architectural Consistency: It seamlessly integrated with 360's existing technology stack, eliminating the need to introduce and maintain a new configuration management tool.

• Operational Simplicity: It significantly reduced the operational complexity and risk associated with the integration process.

• High Performance & Reliability: Leveraging etcd's watch mechanism, APISIX nodes receive configuration updates in real-time, ensuring consistency and high performance across the entire load balancing cluster.

Unified L7 Load Balancing in 360

1. L7 Load Balancing Architecture

The unified L7 load balancing architecture consists of two main components: the control plane and the data plane, which is also the forwarding plane, as illustrated in the diagram below.

• The control plane, in addition to the existing LBC-API shared between L4 and L7, incorporates the apisix admin API for disseminating L7 rules. It utilizes an etcd cluster for storage, leveraging its millisecond-level change notification mechanism to achieve real-time configuration updates. This design supports cluster management and offers enhanced adaptability to containerized environments. In VPC scenarios, the control plane assigns a unique IP to upstream services within the VPC, replacing their actual VPC IP addresses.

• The second component is the forwarding plane, which is responsible for processing client requests and forwarding user traffic. It supports various advanced features such as authentication, SSL/TLS termination, log analysis, and observability, but does not retain any data storage. In VPC environments, the forwarding plane reuses the FNAT gateway forwarding logic. Based on the mapping between the unique IP and the actual VPC IP, traffic is encapsulated using VXLAN and transmitted to the host where the upstream service is located.

Invoking the LBC-API allows creating, deleting, or modifying L7 resources such as service, route, and upstream.

1. The API then calls the apisix admin API to persist the configuration into the etcd cluster.
2. The LBC-AGENT process listens for changes in etcd and applies VPC IP mapping configurations to the FNAT data plane.
3. The APISIX data plane nodes also listen for changes in etcd, allowing new L7 routing rules to take effect dynamically.
4. Traffic from our on-prem data center (IDC) is forwarded directly to its destination IP, while traffic destined for VPCs is encapsulated and forwarded to its host.
5. To use the service, teams simply point their application's DNS to the corresponding Virtual IP (EIP for public services or a VPC-internal VIP).

2. Service Deployment Architecture

There are several features of this service deployment architecture.

• Unified Interface: The Stack platform serves as a unified entry point for requests, providing an external OpenAPI for business units to call. The Container Cloud Platform performs operations on L7 load balancing resources through this entry point. In the future, all other services will also be required to access and integrate via OpenAPI.

• High Availability: The control plane and storage are deployed at the regional level, while the forwarding plane adopts a cluster deployment model at the Availability Zone (AZ) granularity. In the event of a single server failure, requests are automatically redirected to other servers within the cluster.

• Hybrid L4 and L7 Deployment: This approach leverages existing L4 capabilities, minimizing the need for additional development efforts.

3. Traffic Path

As shown in the figure above, traffic is categorized into two scenarios: on-cloud and off-cloud.

1. Classic Network (On-Prem) L7 Traffic:

Public EIP → L4 Load Balancing Cluster (idc vip) → L7 Load Balancing Gateway → IDC-Reachable IPs (including VMs, Pods, etc.)

2. VPC On-Cloud L7 Traffic:

Public EIP → L4 Load Balancing Cluster (vpc vip) → L7 Load Balancing Gateway (VXLAN encapsulation) → Intra-VPC IPs (including VMs, Pods, etc.)

Achievements after Using Apache APISIX

Significant Cost and Resource Optimization

The decision to build the new L7 load balancer atop the existing, robust L4 infrastructure was a foundational strategic victory. By reusing the proven hardware and network architecture, the project eliminated substantial capital expenditure that would have been required for a net-new cluster. This approach not only accelerated deployment timelines but also drastically reduced operational overhead, as teams could leverage familiar tools, monitoring systems, and maintenance procedures for the new service, avoiding a steep learning curve and redundant management costs.

Unified Traffic Management Across Heterogeneous Environments

A primary achievement was the creation of a single, consistent control plane for all traffic management. The unified gateway seamlessly orchestrates requests across a diverse backend ecosystem comprising containers, virtual machines, and bare-metal servers. This eliminates environment-specific silos and operational inconsistencies. The result is dramatically simplified maintenance, unified observability, and improved scalability, allowing infrastructure teams to manage all application traffic through a single pane of glass regardless of where the service is deployed.

Instantaneous Configuration Synchronization and Enhanced Reliability

Leveraging etcd as the central configuration store was critical to achieving high performance and reliability. The built-in watch mechanism provides a millisecond-level notification system for any configuration change. This ensures that policies, routing rules, and security settings are propagated instantaneously and consistently across the entire global load balancing fleet. This real-time capability eliminates configuration drift, minimizes the window for inconsistent behavior during updates, and forms a highly reliable foundation for handling traffic at scale.

Streamlined Operations and Governance

The architecture was designed for seamless integration into the company's existing operational practices. By aligning with the established etcd-based infrastructure, the load balancer inherited existing security policies, access controls, and auditing workflows. This significantly reduced deployment risk and streamlined governance, as there was no need to design new security models or compliance procedures. Operational teams manage the new L7 service with the same tools and protocols used for other core infrastructure, ensuring operational simplicity and enforcing best practices across the board.

Conclusion

The unified L7 load balancer has now been fully deployed across all three regions of 360, and adaptation to container services has also been completed. Currently, the unified L7 load balancer supports only relatively basic functionalities. There are plans for significant functional expansions, including but not limited to:

• SSL Offloading Optimization
• Supporting Redirects within Forwarding Rules
• Traffic Mirroring
• Support for Additional Protocol Types

Furthermore, the system is gradually evolving toward intelligent operation—transitioning from a simple traffic distribution tool to a smart scheduling hub capable of dynamic and adaptive decision-making.