How Does Zoom Use APISIX Ingress in Its Continuous Delivery Pipeline?

Background

In recent years, many well-known online conferencing software came into being with the development of online meetings and remote working. The typical one, Zoom, has become a popular meeting tool for home office, online teaching, and social scenarios. It has 350 million daily meeting participants and nearly 470,000 paying business customers using the platform. Furthermore, recent data shows that the amount of minutes used for meeting facilitation has reached more than 3.3 trillion per year.

However, like other SaaS and internet companies, Zoom has encountered technical challenges as its business rapidly expands.

• A significant number of microservices: Due to Zoom’s rapid growth of business and teams, over 100 backend services need to be delivered. However, it is hard to manage a large number of microservices efficiently.

• Various deployment environments: SaaS enterprises often encounter scenarios where customers must deploy on dedicated clouds, private clouds, and multi-clouds. Zoom’s business services are all over the world, so there are technical challenges with a large number of hybrid cloud environments.

• Complex infrastructure: Medium-and-large-sized internet enterprises generally have dedicated infrastructure teams responsible for API gateways, configuration centers, key management, logs, monitoring alarms, databases, etc. As that Zoom’s managing R&D teams are distributed globally, it is a great challenge to integrate these complex middleware and infrastructure into the continuous delivery pipeline.

The above challenges are not a simple additive relationship but a multiplicative relationship. In other words, the actual situation is far more complex. However, the open-source tools that Zoom used previously can no longer meet Zoom’s current requirements. That’s why a reliable continuous delivery pipeline is so important.

Below are detailed reasons why Zoom did not continue to choose these previous open-source tools.

Due to the limitation of the above open-source tools, Zoom finally investigated some mainstream solutions for building the underlying continuous delivery pipeline.

After a few rounds of comparison and verification, Zoom ultimately chose APISIX Ingress Controller to support its new continuous delivery pipeline.

Apache APISIX Ingress Controller

What is Apache APISIX Ingress Controller?

Apache APISIX Ingress Controller is a cloud-native ingress controller that uses Apache APISIX as the data plane to carry traffic and extends the functionality of Kubernetes utilizing CRD (CustomResourceDefinition). It is responsible for interacting with the Kubernetes API server, applying for role-based access control (RBAC), monitoring changes, implementing object conversion within the Ingress controller, comparing the changes, and then synchronizing to Apache APISIX. Moreover, it can support custom resources, including APISIX Route, APISIX Upstream, and other native ingresses of Kubernetes, to control external traffic to access services deployed in Kubernetes.

Below is the Timing Diagram of APISIX Ingress Controller.

Why Zoom Chooses APISIX Ingress Controller?

According to the above background, here comes the question: what kind of continuous delivery pipeline does Zoom want to build, and how does it adopt APISIX Ingress to establish its underlying continuous delivery pipeline?

Zoom previously used NGINX as an API gateway. However, with the rapid development of business and the increase in microservices, the current NGINX solution’s limitations are becoming more apparent.

Different teams must maintain NGINX separately, and thousands of lines of NGINX configuration files make maintenance difficult. Besides, NGINX cannot quickly scale when deploying many lines on the cloud server. About NGINX’s reloading mode, you can check this blog. Zoom’s businesses began to evolve towards Ingress Controller.

The API Gateway team researched some open-source solutions. After simulating migration and analysis of actual business configuration in NGINX and a large number of benchmark tests of performance and plugin ecological comparison, Zoom finally chooses the Apache Software Foundation’s APISIX Ingress Controller project, exploring a more advanced Cloud-Native gateway.

Taking its business scenarios into consideration, Zoom attached more emphasis to below two parts, which can be satisfied by APISIX Ingress.

• Data Security: Zoom takes customer privacy and service security seriously; thus, mTLS authentication and verification are widely used in online meeting rooms and phone calls. Nevertheless, many similar API gateways can only provide such a service in their enterprise version, while APISIX Ingress provides great feasibility and convenience to achieve this goal.

• Service Stability: Zoom’s backend services require Multi-AZ (Multi-Availability Zones) deployments for high availability across different regions. Generally, it puts businesses in other DC. If an error occurs in the original DC, the client traffic needs to be transformed to another one, in which APISIX Ingress can successfully meet the requirement.

Features of Apache APISIX Ingress Controller

Taking Apache APISIX as its data plane to carry business traffic, Apache APISIX Ingress Controller inherits the following advantages from Apache APISIX.

• High Performance and Stability: As a high-performance dynamic open-source API gateway, Apache APISIX is resilient and stable in its performance, being used in many large-scale traffic scenarios of enterprises.

• Active Community: As a top-level and the most active open-source API gateway project, Apache APISIX possesses an active community and has maintained an excellent growth rate from the first day.

• Rich Ecosystem: Apache APISIX supports L7 protocols, including HTTP(S), HTTP2, Dubbo, IoT protocol MQTT, etc. In addition, APISIX supports L4 protocols such as TCP/UDP. In addition, full support of ARM64 is supported in Apache APISIX 3.0.

• Multiple Plugins: There are almost 100 plugins released from the APISIX official that users can use by simple drag. The hot reloading and dynamic orchestration of plugins provide users with great convenience.

In addition, the APISIX Ingress Controller also has the following unique advantages:

• Good Compatibility: APISIX Ingress Controller supports multiple versions of ingress resources and can be compatible with different Kubernetes versions.

• Dynamic Update: There is no need to reload service when modifying routes, certificates, and other configurations, ensuring the smooth operation of the business.

• Flexible Scaling: Since the APISIX Ingress Controller adopts a structure that separates the control plane and the data plane, the data plane cluster of Apache APISIX can be expanded independently without scaling the APISIX Ingress Controller.

• Friendly to Operation and Maintenance: Under the current architecture, users can choose to deploy the data plane APISIX cluster in the Kubernetes cluster or the physical bare-metal machine environment according to the actual situation. Furthermore, due to the architectural separation of the APISIX ingress, the APISIX on the data plane carries the traffic while the APISIX ingress controller is the control plane component. Therefore, the APISIX Ingress Controller’s failure will not impact business traffic.

After the gateway selection was completed, the API Gateway team faced a new challenge: how to migrate the original API gateway configuration of hundreds of services to APISIX Ingress? Zoom’s Infrastructure team was developing the continuous delivery pipeline, which can significantly reduce the migration cost of converting from nginx.conf and other Ingress configurations to APISIX Ingress.

The Process and Functions of Building the Continuous Delivery Pipeline

Zoom’s Continuous Delivery Pipeline

The continuous delivery pipeline is an end-to-end application delivery system that implements a single model for declaring all application delivery requirements and arranging and executing all continuous delivery steps in one line.

There are six parts in the continuous delivery pipeline:

1. Prepare: prepare preset resources, including infrastructure, middleware, cloud service resources, etc.;

2. Config: prepare the configuration files and keys required by the application;

3. Deploy: use K8s for deployment in cloud-native scenarios (including information about containers, container images, parameters, versions, and instances);

4. Access: create a Kubernetes Service and automatically configure the routing rules of the Apache APISIX Ingress if the deployment requires external access;

5. Observe: Perform observability-related configurations, such as monitoring, alarming, logging, and tracing analysis;

6. Scale: Declare KEDA (Kubernetes Event-driven Autoscaling)’s dynamic scaling rules through monitoring metrics.

How to Adopt APISIX Ingress in the Pipeline

Project Management

Project Managers pay more attention to R&D iterations and how to better manage the iterative release progress and personnel efficiency on the timeline corresponding to the iterations. Zoom uses a GitOps workflow internally to build API Gateway configuration into the application delivery model. Under this model, the definition of APISIX routing rules is integrated with “Deploy” and other links, handing over the change control to GitHub. With the creation and merging of GitHub branches, the pipeline realizes consistent timelines between application and gateway configuration release and rollback.

# Code Snippet of Zoom's CD Pipeline in Git Repositary
deploy:
  type: Deployment
  replicas: ~{ replicas, 2 }
  version: "latest"
  containers:
    - name: my-app
      image: "busybox"
      command: "echo 'Demo' && sleep 99d"
access:
  - protocol: https
    host: my-domain.my-org.com
    cert: my-tls-cert
    apisix:
      routes:
        http:
          - name: my-api
            authentication:
              # ......
            match:
              paths:
                - /my-api/*

This way simplifies release management into workflow management within GitHub and solves the problem of the time lag between upstream and downstream systems matching changes when multiple iterations process simultaneously.

Application Development

Developers mainly focus on APIs’ routing and authentication capabilities, which are strongly related to business services and should be filled in by developers to achieve the automatic effect. In addition, they pay attention to developing and implementing business functions and the upper-level business, hoping to build an out-of-the-box infrastructure.

From the above-mentioned codes, we can see developers only need to define Authentication and Match in this continuous delivery pipeline. They do not need to know the underlying logic:

• First, translate it into Kubernetes Deployment and Service.
• Then, call the platform API to verify the path’s correctness.
• Last, translate it into ApisixRoute objects.

Integrating APISIX’s configuration and the continuous delivery pipeline workflow provides developers with a more labor-saving way.

Environment Management

Complex environmental management and control requirements often occur in toB scenarios, and delivery issues in some private cloud, dedicated cloud, and hybrid cloud scenarios must be considered.

Some configurations of APISIX ingress were implemented to meet the requirement of shielding environmental differences. In this way, system managers can comprehensively control the differences in some heterogeneous environments. All services deployed to the environment are effective, avoiding the cognitive burden and special deployment operations caused by environmental differences to application and ops developers. For example, Zoom can use customized configuration to disable tracing for some environments, convert ApisixRoute objects to native Ingress objects and NGINX Ingress Annotation, and use different images to pull secrets.

Furthermore, multi-tenant isolation is required when multiple lines of business are using the APISIX environment. APISIX Ingress provides an Annotation selector that enables different ApisixRoute objects to be picked up by other instances of the APISIX Ingress Controller.

Infrastructure Management

The API Gateway team need to control all APISIX instances, configure security policies comprehensively, implement multi-availability zones, and so on.

Each plugin of the pipeline provides configuration items for infrastructure engineers. In the plugin ingress-apisix, there is a defaultPlugins property.

After the API gateway team configures the property, the setting will take effect for all the services, which is suitable for a unified security and risk control strategy.

Conclusion

APISIX Ingress Controller plays an important role in Zoom’s continuous delivery pipeline, releasing the pressure on project management, application development, environment management, and middleware and infrastructure management. The case of Zoom is worth learning for other companies, and we look forward to APISIX Ingress Controller contributing to the innovation of more companies.

Besides, Apache APISIX Ingress officially released V1.5 in August 2022, which unifies the proposal of all resource API Versions and upgrades all CRD API Versions to V2. At the same time, it supports most of the Gateway API resources. Apache APISIX Ingress has enabled Ingress resources to use arbitrary APISIX plugins by adding a new Annotation “k8s.apisix.apache.org/plugin-config-name” to Ingress resources. In this way, it will significantly increase the ease of use of the APISIX Ingress Controller and reduce the cost for users to migrate from other Ingress Controllers to APISIX Ingress Controller.

For further information, please check Apache APISIX Ingress V1.5.