
## Background

In recent years, many well-known online conferencing software came into being with the development of online meetings and remote working. The typical one, Zoom, has become a popular meeting tool for home office, online teaching, and social scenarios. It has 350 million daily meeting participants and nearly 470,000 paying business customers using the platform. Furthermore, recent data shows that the amount of minutes used for meeting facilitation has reached more than 3.3 trillion per year.

However, like other SaaS and internet companies, Zoom has encountered technical challenges as its business rapidly expands.

- **A significant number of microservices**: Due to Zoom’s rapid growth of business and teams, over 100 backend services need to be delivered. However, it is hard to manage a large number of microservices efficiently.

- **Various deployment environments**: SaaS enterprises often encounter scenarios where customers must deploy on dedicated clouds, private clouds, and multi-clouds. Zoom’s business services are all over the world, so there are technical challenges with a large number of hybrid cloud environments.

- **Complex infrastructure**: Medium-and-large-sized internet enterprises generally have dedicated infrastructure teams responsible for API gateways, configuration centers, key management, logs, monitoring alarms, databases, etc.
As that Zoom’s managing R&D teams are distributed globally, it is a great challenge to integrate these complex middleware and infrastructure into the continuous delivery pipeline.

**The above challenges are not a simple additive relationship but a multiplicative relationship. In other words, the actual situation is far more complex.** However, the open-source tools that Zoom used previously can no longer meet Zoom’s current requirements. That’s why a reliable continuous delivery pipeline is so important.

Below are detailed reasons why Zoom did not continue to choose these previous open-source tools.

| Helm/Kustomize  | Terraform/Pulumi  | KubeVela + Crossplane  |
| ------------ | ------------ | ------------ |
| Unable to connect systems except Kubernetes  | Difficult to use Terraform in the Kubernetes and the middleware layers unless developing extra Providers  | Too brand-new technology stack with fast update, not safe and stable enough to be adopted in the production environment  |
| Difficult to centrally control the platform in the Git sub-repository mode  | Increase the cost of learning another configuration management language: hcl  | Kubevela’s definition of Trait + Cuelang templates and its programming expression capabilities are not enough to cover various non-cloud-native middleware platforms outside the Kubernetes system  |
| Difficult to maintain and debug due to Helm Chart Template’s complex logic  | Mono Repo centralized control mode is unsuitable for bulk teams  | /  |
| Not powerful parameter system to manage dynamic parameters in a large number of environments  | /  | /  |

Due to the limitation of the above open-source tools, Zoom finally investigated some mainstream solutions for building the underlying continuous delivery pipeline.

After a few rounds of comparison and verification, Zoom ultimately chose APISIX Ingress Controller to support its new continuous delivery pipeline.

## Apache APISIX Ingress Controller

### What is Apache APISIX Ingress Controller?

[Apache APISIX Ingress Controller](https://github.com/apache/apisix-ingress-controller) is a cloud-native ingress controller that uses Apache APISIX as the data plane to carry traffic and extends the functionality of Kubernetes utilizing CRD (CustomResourceDefinition). It is responsible for interacting with the Kubernetes API server, applying for role-based access control (RBAC), monitoring changes, implementing object conversion within the Ingress controller, comparing the changes, and then synchronizing to Apache APISIX. Moreover, it can support custom resources, including APISIX Route, APISIX Upstream, and other native ingresses of Kubernetes, to control external traffic to access services deployed in Kubernetes.

Below is the [Timing Diagram of APISIX Ingress Controller](https://apisix.apache.org/docs/ingress-controller/design/#timing-diagram).

![Timing Diagram of APISIX Ingress](https://static.apiseven.com/2022/10/27/635a5613af56a.jpeg)

## Why Zoom Chooses APISIX Ingress Controller?

According to the above background, here comes the question: what kind of continuous delivery pipeline does Zoom want to build, and how does it adopt APISIX Ingress to establish its underlying continuous delivery pipeline?

Zoom previously used NGINX as an API gateway. However, with the rapid development of business and the increase in microservices, the current NGINX solution’s limitations are becoming more apparent.

Different teams must maintain NGINX separately, and thousands of lines of NGINX configuration files make maintenance difficult. Besides, NGINX cannot quickly scale when deploying many lines on the cloud server. About NGINX’s reloading mode, you can check [this blog](https://api7.ai/blog/how-nginx-reload-work). Zoom’s businesses began to evolve towards Ingress Controller.

The API Gateway team researched some open-source solutions. After simulating migration and analysis of actual business configuration in NGINX and a large number of benchmark tests of performance and plugin ecological comparison, Zoom finally chooses the Apache Software Foundation’s APISIX Ingress Controller project, exploring a more advanced Cloud-Native gateway.

Taking its business scenarios into consideration, Zoom attached more emphasis to below two parts, which can be satisfied by APISIX Ingress.

- **Data Security**: Zoom takes customer privacy and service security seriously; thus, mTLS authentication and verification are widely used in online meeting rooms and phone calls. Nevertheless, many similar API gateways can only provide such a service in their enterprise version, while APISIX Ingress provides great feasibility and convenience to achieve this goal.

- **Service Stability**: Zoom’s backend services require Multi-AZ (Multi-Availability Zones) deployments for high availability across different regions. Generally, it puts businesses in other DC. If an error occurs in the original DC, the client traffic needs to be transformed to another one, in which APISIX Ingress can successfully meet the requirement.

### Features of Apache APISIX Ingress Controller

Taking Apache APISIX as its data plane to carry business traffic, Apache APISIX Ingress Controller inherits the following advantages from Apache APISIX.

- **High Performance and Stability**: As a high-performance dynamic open-source API gateway, Apache APISIX is resilient and stable in its performance, being used in many [large-scale traffic scenarios of enterprises](https://apisix.apache.org/blog/tags/case-studies/).

- **Active Community**: As a top-level and the most active open-source API gateway project, Apache APISIX possesses an active community and has maintained an excellent growth rate from the first day.

- **Rich Ecosystem**: Apache APISIX supports L7 protocols, including HTTP(S), HTTP2, Dubbo, IoT protocol MQTT, etc. In addition, APISIX supports L4 protocols such as TCP/UDP. In addition, full support of ARM64 is supported in Apache APISIX 3.0.

- **Multiple Plugins**: There are almost 100 plugins released from the APISIX official that users can use by simple drag. The hot reloading and dynamic orchestration of plugins provide users with great convenience.

In addition, the APISIX Ingress Controller also has the following unique advantages:

- **Good Compatibility**: APISIX Ingress Controller supports multiple versions of ingress resources and can be compatible with different Kubernetes versions.

- **Dynamic Update**: There is no need to reload service when modifying routes, certificates, and other configurations, ensuring the smooth operation of the business.

- **Flexible Scaling**: Since the APISIX Ingress Controller adopts a structure that separates the control plane and the data plane, the data plane cluster of Apache APISIX can be expanded independently without scaling the APISIX Ingress Controller.

![APISIX Ingress Controller Architecture](https://static.apiseven.com/2022/10/27/635a56148940c.jpeg)

- **Friendly to Operation and Maintenance**: Under the current architecture, users can choose to deploy the data plane APISIX cluster in the Kubernetes cluster or the physical bare-metal machine environment according to the actual situation. Furthermore, due to the architectural separation of the APISIX ingress, the APISIX on the data plane carries the traffic while the APISIX ingress controller is the control plane component. Therefore, the APISIX Ingress Controller’s failure will not impact business traffic.

After the gateway selection was completed, the API Gateway team faced a new challenge: how to migrate the original API gateway configuration of hundreds of services to APISIX Ingress? Zoom’s Infrastructure team was developing the continuous delivery pipeline, which can significantly reduce the migration cost of converting from nginx.conf and other Ingress configurations to APISIX Ingress.

## The Process and Functions of Building the Continuous Delivery Pipeline

### Zoom’s Continuous Delivery Pipeline

The continuous delivery pipeline is an end-to-end application delivery system that implements a single model for declaring all application delivery requirements and arranging and executing all continuous delivery steps in one line.

![Zoom's Continuous Delivery Pipeline](https://static.apiseven.com/2022/10/27/635a56149b270.jpg)

There are six parts in the continuous delivery pipeline:

1. **Prepare**: prepare preset resources, including infrastructure, middleware, cloud service resources, etc.;

2. **Config**: prepare the configuration files and keys required by the application;

3. **Deploy**: use K8s for deployment in cloud-native scenarios (including information about containers, container images, parameters, versions, and instances);

4. **Access**: create a Kubernetes Service and automatically configure the routing rules of the Apache APISIX Ingress if the deployment requires external access;

5. **Observe**: Perform observability-related configurations, such as monitoring, alarming, logging, and tracing analysis;

6. **Scale**: Declare KEDA (Kubernetes Event-driven Autoscaling)’s dynamic scaling rules through monitoring metrics.

### How to Adopt APISIX Ingress in the Pipeline

![Zoom's Pipeline Architecture Overview](https://static.apiseven.com/2022/10/27/635a5616254ed.jpg)

**Project Management**

Project Managers pay more attention to R&D iterations and how to better manage the iterative release progress and personnel efficiency on the timeline corresponding to the iterations. Zoom uses a GitOps workflow internally to build API Gateway configuration into the application delivery model. Under this model, the definition of APISIX routing rules is integrated with “Deploy” and other links, handing over the change control to GitHub. With the creation and merging of GitHub branches, the pipeline realizes consistent timelines between application and gateway configuration release and rollback.

```yaml
# Code Snippet of Zoom's CD Pipeline in Git Repositary
deploy:
  type: Deployment
  replicas: ~{ replicas, 2 }
  version: "latest"
  containers:
    - name: my-app
      image: "busybox"
      command: "echo 'Demo' && sleep 99d"
access:
  - protocol: https
    host: my-domain.my-org.com
    cert: my-tls-cert
    apisix:
      routes:
        http:
          - name: my-api
            authentication:
              # ......
            match:
              paths:
                - /my-api/*
```

This way simplifies release management into workflow management within GitHub and solves the problem of the time lag between upstream and downstream systems matching changes when multiple iterations process simultaneously.

**Application Development**

Developers mainly focus on APIs’ routing and authentication capabilities, which are strongly related to business services and should be filled in by developers to achieve the automatic effect. In addition, they pay attention to developing and implementing business functions and the upper-level business, hoping to build an out-of-the-box infrastructure.

From the above-mentioned codes, we can see developers only need to define `Authentication` and `Match` in this continuous delivery pipeline. They do not need to know the underlying logic:

- First, translate it into Kubernetes Deployment and Service.
- Then, call the platform API to verify the path’s correctness.
- Last, translate it into ApisixRoute objects.

Integrating APISIX’s configuration and the continuous delivery pipeline workflow provides developers with a more labor-saving way.

**Environment Management**

Complex environmental management and control requirements often occur in toB scenarios, and delivery issues in some private cloud, dedicated cloud, and hybrid cloud scenarios must be considered.

Some configurations of APISIX ingress were implemented to meet the requirement of shielding environmental differences. In this way, system managers can comprehensively control the differences in some heterogeneous environments. All services deployed to the environment are effective, avoiding the cognitive burden and special deployment operations caused by environmental differences to application and ops developers. For example, Zoom can use customized configuration to disable tracing for some environments, convert ApisixRoute objects to native Ingress objects and NGINX Ingress Annotation, and use different images to pull secrets.

Furthermore, multi-tenant isolation is required when multiple lines of business are using the APISIX environment. **APISIX Ingress provides an Annotation selector** that enables different ApisixRoute objects to be picked up by other instances of the APISIX Ingress Controller.

**Infrastructure Management**

The API Gateway team need to control all APISIX instances, configure security policies comprehensively, implement multi-availability zones, and so on.

Each plugin of the pipeline provides configuration items for infrastructure engineers. In the plugin `ingress-apisix`, there is a `defaultPlugins` property.

After the API gateway team configures the property, the setting will take effect for all the services, which is suitable for a unified security and risk control strategy.

## Conclusion

APISIX Ingress Controller plays an important role in Zoom’s continuous delivery pipeline, releasing the pressure on project management, application development, environment management, and middleware and infrastructure management. The case of Zoom is worth learning for other companies, and we look forward to APISIX Ingress Controller contributing to the innovation of more companies.

Besides, Apache APISIX Ingress officially released V1.5 in August 2022, which unifies the proposal of all resource API Versions and upgrades all CRD API Versions to V2. At the same time, it supports most of the Gateway API resources. Apache APISIX Ingress has enabled Ingress resources to use arbitrary APISIX plugins by adding a new Annotation “k8s.apisix.apache.org/plugin-config-name” to Ingress resources. In this way, it will significantly increase the ease of use of the APISIX Ingress Controller and reduce the cost for users to migrate from other Ingress Controllers to APISIX Ingress Controller.

For further information, please check [Apache APISIX Ingress V1.5](https://apisix.apache.org/blog/2022/08/05/apisix-ingress-1.5rc1-release/).


Zoom integrates APISIX Ingress Controller in its continuous delivery pipeline to realize the overall management of project, application, environment and infrastructure management.

How Does Zoom Use APISIX Ingress in Its Continuous Delivery Pipeline?


## Overview

Since May 2021, [vivo](https://www.vivo.com/en) has introduced [Apache APISIX](https://apisix.apache.org/) as its API Gateway. After more than a year of practice in vivo, APISIX has solved many technical and business pain points and has been used on a large scale.

### Pain Points before Using APISIX

- **Complex Management of Business Scenarios and System Maintenance**

    Due to the rapid growth of business, there are various scenarios and systems that serve them, which vivo needs a unified way to manage.

- **Interact between the Data Plane and the Control Plane**

    For medium and large-sized companies like vivo, it’s unexpected that minor trouble that occurs in the data plane will impact the control plane.

- **No Support for Multi-Dimension Resources**

    Diverse projects leads to various domain names and URLs. The business department needs to search according to different resource dimensions.

- **Uncontrollable Impact of Problems**

    As vivo’s projects are complex, and the effect of problems encountered are uncontrollable. The use of some complicated plugins intensifies this.

By replacing APISIX with NGINX, vivo finally successfully made a series of achievements as below.

### Achievements after Using APISIX

- **High Availability**

    No major failure has occurred since APISIX is launched in vivo, and the system availability exceeds **99.99%**.

- **High Performance**

    Undertaking significant online traffic and serving a large number of services, the current online forwarding traffic reaches close to **one million QPS** (Queries-per-second).

- **Rich Features**

    Thanks to APISIX's [rich features](https://apisix.apache.org/blog/2022/11/02/apache-apisix-v3-preview/), APISIX can cover almost common NGINX proxy scenarios. About 50% of the vivo projects is migrated from the NGINX to the APISIX clusters.

- **Supporting the Construction and Development of Cloud-Native**

    The K8s bare metal supporting the containerization has reached a scale of **10,000**. About 40% of the projects has been migrated from bare metal and virtual machines to the K8s container platform, supporting and promoting vivo’s containerization progress.

## vivo's System Design Based on APISIX

Next, let’s look at the vivo’s system design after adopting APISIX.

### Customized Architecture Display on APISIX

![vivo's API gateway architecture with APISIX](https://static.apiseven.com/2022/11/23/637dc2f51318e.jpg)

From the diagram above, we can analyze that vivo has:

- Completed the construction of **Layer 4** and **Layer 7** traffic gateways, which are supported by APISIX
- Realized the traffic access and mixed deployment of bare metal, virtual machines, and containers
- Implemented APISIX cluster management
- Connected internal DevOps platform and business deployment services to access traffic quickly and automatically
- Improved monitoring construction

### Configuration Management and Launching Improvements

In order to better meet the actual needs of the business department, vivo has carried out a series of adaptations on APISIX. Below are some of those adjustments, including control plane alteration, cluster separation management, and data forwarding.

#### Control Plane Alteration

The whole process can be this:

After the data is configured in the A6 changes platform, the information will be delivered through RPC notify to the ManagerAPI, which is built by vivo based on the open-sourced [APISIX Dashboard](https://github.com/apache/apisix-dashboard).

Then the traffic will be sent to `apisix-agent`. APISIX polls apisix-agent regularly through the privileged process to obtain change tasks in batches. Next, the privileged process notifies the `worker` through shared queues to realize the change in memory.

In the meantime, APISIX informs the `apisix-agent` of the result of tasks and then delivers them to ManagerAPI. Additionally, the A6 changes platform can poll the ManagerAPI to get task results.

![vivo's management and publishing mode](https://static.apiseven.com/2022/11/25/638036da44e7f.jpeg)

The [etcd](https://etcd.io/docs/v3.5/learning/why/#comparison-chart "etcd") is a highlight of APISIX, allowing the independent operation of the control and data planes. Considering the uniqueness of its architecture, vivo discarded etcd in the above process. Here are some reasons.

Due to vivo’ projects diversity, there are various domain names and URLs. Besides, the business departments need to query different dimensions. Thanks to APISIX’s adaptability with not only etcd but also varying kinds of databases, vivo could easily utilize databases like MongoDB, to work together with APISIX.

In addition, vivo made the below contributions to be compatible with Apache APISIX.

- **Developing the Agent Component**

    Since May 2021, vivo introduced Apache APISIX. Considering the technical background and context, vivo was unconfident to be incapable of adopting APISIX because vivo has no experience with OpenResty and Lua. In addition, there are many non-forwarding tasks, like log collection and monitoring handling, which might increase the management complexity of the data plane. Consequently, vivo developed the agent component to reduce the complexity of development.

- **Writing Data to Disk**

    To make the system adjustable and enable the data plane runs independently, thus reducing the reliance on the control plane, vivo wrote the configuration file into the disk. When APISIX is started, it supports complete pulling from the configuration center and also supports directly obtaining configuration resources from the file directory of the local disk. This way dramatically improves the independence of data and the robustness of the system. Moreover, it is very intuitive to understand the configured route and upstream information on the disk that is placed on the disk, which is helpful for troubleshooting.

- **Callback Change Task Result**

    As a large-sized company, vivo needs to make sure that changes to resources such as routers and upstreams can be guaranteed to be effective and successful, and the system can report the error even if these changes fail. Such logic of ACK (Acknowledgement Code) ensures that NGINX workers on a machine can call back. When the callback tasks are successful, all workers on APISIX will update resource changes to the relevant memory.

#### Cluster Separation Management

![Cluster Split Management](https://static.apiseven.com/2022/11/25/6380381595b00.jpeg)

The open-source version of APISIX provides [etcd](https://apisix.apache.org/docs/apisix/FAQ/#why-does-apache-apisix-use-etcd-for-the-configuration-center) for everyone to share. However, the company’s projects are complex, and the problems encountered are uncontrollable. In addition, it is inevitable to use complicated plugins, affecting the system’s performance.

Therefore, it is managed by cluster separation to realize the isolation of cluster configuration on APISIX, which can

- Control the fault domain and effectively support projects complexity without affecting other projects
- Effectively reduce the load caused by the APISIX non-forwarding layer when the container nodes change frequently
- Reduce the load impact caused by the health check

#### Increasing the QPS Carried by HTTPS

According to the relevant requirements of the Ministry of Industry and Information Technology in China, external network traffic must go through the HTTPS protocol. As a TLS-based HTTP encryption protocol, HTTPS heavily burdens the CPU in the encryption and decryption process.

When the routing and other configurations are the same, the traffic that HTTPS can carry is about **1/8 - 1/10** of that of HTTP.

After patching the Intel® QAT (QuickAssist Technology) accelerator card, vivo delivers the decryption handling to the QAT accelerator card, which frees up the CPU, thereby increasing the QPS carried by HTTPS on a single machine. As can be seen from the figure below, **the HTTPS load capacity of a single machine is about doubled**.

![data showing vivo's improvment on carrying traffic](https://static.apiseven.com/2022/11/23/637dc306a5fa1.jpg)

## How Does vivo Combine Businesses with APISIX

### Supporting Containerization Development

To support containerization development, vivo self-developed a K8s ingress controller. Below are some functions of it.

1. Adapting to vivo’s modified asynchronous push configuration change mechanism

2. Conducting multi-K8s cluster event processing notifications to APISIX

3. Coping with complex projetcs scenarios such as:

- One server with multiple ports

- When other RPC framework servers, such as Dubbo and gRPC, are connected to K8s, a unified set of processing logic is required to notify APISIX or other frameworks of port information according to the configuration characteristics of the projects

4. Adapting to the particular needs of the company’s internal DevOps and other automation scenarios, facilitating rapid deployment, and enabling traffic

### Helping Projects Migrate from NGINX to APISIX

Vivo’s projects are deployed on the existing NGINX cluster and has been running stably for a long time. However, it brings non-business workloads and instability to projects. Consequently, it’s challenging to conduct migration. So how to promote the projects migration to APISIX?

- First, find a project of a cooperative department, serve the business department well to set a benchmark, and provide technical guidance and training

- Build an easy-to-use control plane system to facilitate business access and multi-dimensional management of business departments
  
- Provide automatic conversion capability and basic configuration converting from NGINX configuration to APISIX configuration

### Upgrading APISIX and Supporting Its Open-Source Version

Based on APISIX 2.4 version, vivo made some adjustments and released the new version, which was upgraded to a newer one in Q2 this year.

On the one hand, thanks to the modular architecture of APISIX, it is relatively easy to integrate vivo’s modified Lua code into the branch of the higher version of APISIX. On the other hand, vivo also keeps upgrading OpenResty section, with about one version per year. Since vivo utilizes a lot of PATCH and some useful functions like QAT, upgrading this component is difficult and laborious.

The free version of NGINX community features is slow to update and is inactive. Vivo is considering whether to build jointly with APISIX. To reduce the manpower needed to do related system testing, vivo adopted Robot Framework, a generic test automation framework for system integration testing. They are promoting the relevant components for the unit test coverage and the development model of TDD (Test-driven development).

## vivo's Future Planning

Next year, vivo plans to extend [APISIX](https://apisix.apache.org/blog/2022/09/13/why-is-apache-apisix-the-best-api-gateway/) as a traffic gateway into an API gateway, utilizing its advantages of [rate limiting](https://apisix.apache.org/docs/apisix/plugins/limit-count/#attributes), [authentication](https://apisix.apache.org/docs/ingress-controller/tutorials/mtls/), circuit breaking, etc. Considering combining APISIX with DPDK-NGINX, vivo will also cultivate technical personnel and join the community establishment. Furthermore, it will consolidate the basic skills to lay a good foundation, building traffic and service governance.

Welcome to learn more about Apache APISIX.

You can contact us at [https://api7.ai/contact](https://api7.ai/contact).


How does the famous mobile manufacturer vivo adopt APISIX as its API gateway and adjust it to fit its businesses?

How Does vivo Integrate with APISIX


## Overview

### Challenges

- A significant challenge in data governance as Airwallex’s business expands throughout the world
- Complicated management of various systems and interconnection cause problems in other systems when one problem occurs

### Results

- Stable performance for a **15-month** production environment running
- **99%** of the response delays are within **23 ms** when dynamic routing, multi-tenant rate-limiting, multi-tenant IP whitelist, authentication, and authorization are enabled
- Safe and efficient support of data sovereignty covering more than **50 currencies** in over **130 countries and regions**

## About Airwallex

As a global fintech company, [Airwallex](https://www.airwallex.com/us) has built a global financial platform, with its payment network covering more than 50 currencies in over 130 countries and regions worldwide, and it provides digital fintech products for enterprises.

![Airwallex Business](https://static.apiseven.com/apisix-webp/202108/1646626634389-7d9a0a82-a76c-4645-93b6-8a431bb1f9a1.webp)

> This article comes from the interview with Yang Li, Ph.D., Committer of Apache APISIX and Technical Platform Lead of Airwallex, who is responsible for the evolution of Airwallex's technology platform.

## Data Sovereignty Poses Challenge to Airwallex

[Data sovereignty](https://api7.ai/blog/leveraging-api-gateway-for-data-sovereignty-and-data-compliance) is becoming increasingly important in cyberspace data governance. Airwallex serves users in various regions, which naturally involves the problem of data sovereignty.

Before, without the requirement of data sovereignty, data could be placed in Europe and synchronized to Asia or any data center in the world. The data interaction diagram is like this:

![Multi-cluster Interaction](https://static.apiseven.com/202108/1635907812257-962b1247-f5ca-448f-904d-fd47c3ea4586.png)

However, in an age where data sovereignty is valued, data is under control, and the previous architecture cannot be adopted. In Europe, only European data can be processed, and similarly, only Asian data can be processed in Asia. The data interaction changes are as below.

![Interaction under data sovereignty environment](https://static.apiseven.com/uploads/2023/02/01/63Z7AhJx_2.webp)

As a result, data sovereignty poses a significant challenge for the global business of Airwallex. Airwallex intends to migrate data from Europe to other regions, and it has to ensure that the requirements of third-party countries for data sovereignty are in line with that of the EU.

## Why Does Airwallex Choose APISIX?

When selecting the API gateway, Airwallex mainly compared the main gateway products from six dimensions:

- **Stability**: Airwallex puts great emphasis on ensuring the stability of the API gateway. Among the top 1000 websites in the world, **62.1% are NGINX-based**, which shows that the NGINX-based web servers have passed the test of complex and diverse scenarios in the production environment. Apache APISIX is based on NGINX. Furthermore, the design of **Apache APISIX’s [full dynamic](https://api7.ai/blog/how-nginx-reload-work)** feature also makes it possible to modify the route without reloading and maintaining the long connection of the client. In addition, **Airwallex also conducted stress tests on Apache APISIX, which can still run stably when the CPU usage reaches more than 70%.**

- **Performance**: Every API request will pass through the API gateway. Reducing the [performance](https://api7.ai/blog/apisix-kong-3-0-performance-comparison) loss of the gateway can significantly reduce Airwallex's overall response time of the APIs.
  
    > "In the PoC, we compared the main gateway products: **the response delay of Apache APISIX is more than 50% lower than that of other gateways**. In addition, Apache APISIX's independent data plane enables independent instances in the clusters, making it inherently support horizontal expansion." Li said.

- **Scalability**: Airwallex needs APISIX to work as an API gateway that supports complex **[authentication](https://api7.ai/blog/api-gateway-authentication), permission control, [service discovery](https://api7.ai/blog/what-is-service-discovery-in-microservices), [rate-limiting](https://api7.ai/blog/rate-limiting-in-api-management), downgrade, load balancing, whitelist, and dynamic routing**.

- **Active Community**: An active community is a key for API Gateway to keep up with the pace of technological evolution.

    > **Apache APISIX’s [community](https://api7.ai/category/community) was active since it was still an Apache incubation project, judging from the number of contributors, issue response time, and the number of Pull Requests.** Li shared.

- **Private deployment**: API gateway is the core technical architecture component of Airwallex, and which deploys the API gateway at the edge of its own private network. **Apache APISIX has good environmental adaptability and can be easily deployed in various environments, including cloud computing platforms.**

- **Business-friendly Open-source License**: The Apache 2.0 License allows Airwallex to modify the code of APISIX and commercialize on it without worrying about vendor lock-in.

## Intelligent Routing Solution of Apache APISIX

**Airwallex chooses Apache APISIX to make an intelligent routing solution**, and it is the Apache APISIX gateway that decides where the data should be processed.

![Airwallex's two-layer gateway](https://static.apiseven.com/uploads/2023/02/01/cGg9YfNp_3.webp)

The gateway is composed of two layers. The first layer is responsible for routing requests, determining which data center the request should reach based on the conditions. The second layer takes charge of traffic forwarding.

Moreover, **Airwallex implemented the main functions as custom [plugin](https://api7.ai/blog/how-to-implement-plugin-orchestration-in-api-gateway)s** to upgrade to a new version of Apache APISIX at any time. This way, there won’t be any race condition between Airwallex’s code library and the core code of Apache APISIX’s main repository. However, modifying the core code is necessary sometimes. At this time, Li would try his best to implement these functions in the open-source community. **The contributors in the community are very enthusiastic about discussing functions implementation, and in most cases, they can quickly provide available solutions.**

## How is Airwallex Benefitting from Using APISIX?

Below is a diagram showing Airwallex’s integration with APISIX.

![Integration Diagram of Airwallex and APISIX](https://static.apiseven.com/uploads/2023/01/31/Q0LbIb0E_88.webp)

Airwallex takes Apache APISIX as the core component of the microservice gateway model. It is deployed at the edge of the network and provides a general gateway function for all traffic entering Airwallex. The problems it solves include:

- **Data Sovereignty**: For financial infrastructure operating across borders, data sovereignty is a critical regulatory requirement. To this end, **Airwallex developed a dynamic regulatory-compliant routing plugin leveraging the capabilities of Apache APISIX dynamic upstream selection**. Dynamic routing can intelligently select upstreams for request distribution according to the characteristics of user requests, abstracting complex multi-data center collaboration issues from the service layer to the gateway layer. The essence of dynamic routing is to answer two questions: how to group upstream and how to match requests and groups.

- **Isolation in Microservices**: Airwallex hopes that the engineering teams of each microservice can independently control their own services, effectively reduce the cost of communication and coordination, and improve engineering efficiency. This architectural concept requires that infrastructure components shared by various teams, such as API Gateway, can support tenant isolation. **APISIX is perfect for satisfying tenant isolation.** While ensuring the robustness and cost control of the entire system, APISIX allows business teams to configure and expand gateway functions according to their own needs, maintaining the independence of microservice teams and services.

- **Multi-Tenant Rate-Limiting**: In a multi-tenant environment, the traffic characteristics of each tenant are different. Applying the same rate-limiting to different tenants cannot meet commercial needs. **Multi-tenant [rate-limiting](https://api7.ai/blog/rate-limiting-in-api-management) can be more appropriate based on user characteristics.**

- **Multi-Tenant IP Whitelist**: The access IP of each tenant varies in a multi-tenant environment. **Multi-tenant IP whitelist allows each tenant to control its own whitelist without worrying about other users’ invasion of their own resources.**

- **Authentication**: Frequent key update is a critical link to ensure the security of user resources. **APISIX supports not only requests [authentication](https://api7.ai/blog/understanding-microservices-authentication-services) but also dynamic key updates.**

- **Authorization**: The permission management of enterprise applications is very complicated, and different application scenarios have completely different permission management models. **APISIX can verify whether the requesting user has sufficient permissions to access the interface according to the routing configuration and intercept illegal traffic the first time.**

Airwallex’s production environment has been running for **15 months**, and the overall performance is extremely stable. About **99% of the response delays are within 23ms** when dynamic routing, multi-tenant rate-limiting, multi-tenant IP whitelist, authentication, authorization, and other functions are enabled.

**Thanks to the excellent plugin mechanism of Apache APISIX, Airwallex can add private plugins that meet business needs without modifying its core code.** The complete testing system further guarantees the software quality, and you can safely increase the personalized requirements of the plugin while ensuring that the original core logic is not damaged.

## Future plan

Airwallex’s follow-up plan mainly includes three aspects:

1. Use a multi-layer network to split gateway logic into different layers, such as distributing traffic according to data sovereignty or other gateway logic.

2. Easy-to-read and easy-to-use routing management are critical to the success of the API gateway. The routing management needs to be friendly to the developers so that developers can easily understand what the gateway can do for them and how to configure and publish it.

3. Use color-code requests to help implement production environment testing. Using color-code requests of API gateway to implement production environment testing can bring more flexibility and ease of use.

## Looking for APISIX Support?

[Apache APISIX](https://api7.ai/apisix) is an open-source, dynamic, scalable, and high-performance cloud-native API gateway for all your APIs and microservices. Being donated to Apache Software Foundation by API7.ai, APISIX has grown into a top-level open-source Apache project.

Do you want to accelerate your development with confidence like Airwallex? To maximize APISIX support, you need [API7](https://api7.ai/). We provide in-depth support for APISIX and API management solutions based on your needs!

Contact us now: [https://api7.ai/contact](https://api7.ai/contact).

More about the interviewee:

![Yang Li, Airwallex](https://static.apiseven.com/uploads/2023/01/31/uGbiZnt5_Airwallex2.webp)


Airwallex resolves technical challenges that data sovereignty brings and makes intelligent routing solutions with the help of Apache APISIX.

What is Apache APISIX?

Trusted by Global Users

Why Apache APISIX?

APISIX vs alternatives

To Maximize APISIX,
You Need API7

Case Studies

How Does Zoom Use APISIX Ingress in Its Continuous Delivery Pipeline?

How Does vivo Integrate with APISIX

How Airwallex Empowers Global Payment Using Apache APISIX?

Looking for Apache APISIX Support?

Trusted by Global Users

Why Apache APISIX?

APISIX vs alternatives

To Maximize APISIX, You Need API7

Case Studies

How Does Zoom Use APISIX Ingress in Its Continuous Delivery Pipeline?

How Does vivo Integrate with APISIX

How Airwallex Empowers Global Payment Using Apache APISIX?

Looking for Apache APISIX Support?

To Maximize APISIX,
You Need API7