Transforming Fund Management Giant with APISIX

Overview

About Invesco Great Wall

Invesco Great Wall Fund Management Co., Ltd. ("IGW") is a Chinese-American fund management company, with multi-asset investment capabilities and leading equities expertise. The company's main businesses include quantitative investment, active income, and fixed income. IGW's asset under management is over 850 billion USD, providing services to more than 60 million investors.

IGW places its primary emphasis on asset management, providing a diverse range of investment strategies spanning multiple asset classes such as equities, fixed income, and quantitative strategies.

Challenges

• The lack of standardized gateways among different business systems led to numerous services associated with gateways and independent operations, resulting in increased maintenance expenses.
• IGW's previous gateways were limited in carrying capacity and capabilities, such as load balancing, and canary release, which are intricate and time-consuming when implemented, potentially causing operational bottlenecks.
• Due to the absence of authentication capabilities in the previous setup, various system interfaces can be accessed directly, posing a substantial security risk to business systems.

Results

• By implementing APISIX, IGW built its unified API gateway, addressing the issues of redundant development efforts and inconsistent quality of features.
• The integration of DevOps and pipelines significantly enhances the stability of route and service deployment, improving the efficiency of routing and service releases.
• APISIX's hot reloading substantially reduces the need for service restarts, thereby alleviating pressure on system resources and reducing downtime.

Background

First, let's delve into the architectural evolution of IGW's business system. Initially, IGW adopted a monolithic service approach, with service applications deployed on physical machines. However, as the services and business grew, the costs of operations and development began to escalate. This prompted the transition to the virtual machine stage.

During the virtual machine phase, there was a lack of uniformity in the gateways used by different business systems. Each business operated independently, resulting in numerous services associated with gateways and high maintenance costs.

In the fund and securities sectors, there are regulatory requirements concerning network zones. Each network zone needs to be divided into different physically isolated secure zones or logically secure zones based on the information security levels, during which the firewall is employed.

Being aligned with the cloud strategy in financial technology and digital transformation, IGW initiated its cloud migration project.

Why IGW Chose APISIX

Due to the financial sector's heightened emphasis on service stability compared to other industries, stability takes precedence. As virtual machine environments experienced rampant growth of gateway services, it became imperative to meet the diverse range of business requirements. Therefore, the scalability of the gateway is highly valued. Furthermore, observability plays a crucial role, with strong demands for logging, tracing, and monitoring of the business system. Lastly, the ability for hot reloading is also essential.

Due to NGINX's limitations in dynamically updating configurations, it requires a reload when facing configuration changes. What's more, in persistent connection scenarios, this process can result in traffic fluctuations within the business system, affecting the user experience.

Therefore, considering these four focal points, the technical team at IGW conducted a comprehensive comparison between APISIX and Kong, two popular and widely recognized API gateways in the market:

• Technical Framework: APISIX and Kong are both developed based on OpenResty. Regarding the configuration center, APISIX adopts etcd, while Kong opts for PostgreSQL. APISIX stands out as a cloud-native distributed gateway due to the cloud-native nature of etcd and APISIX's state features. However, Kong's choice of configuration center may introduce potential single-point-of-failure issues, requiring additional infrastructure support for high availability.

• Community Activity: Both APISIX and Kong have an active community with an average of 20-time commits per day in their repositories.

• Plugins: APISIX has 80+ plugins and only one document for each plugin; Kong has 30+ plugins and 4-5 documents for each plugin; 100+ code for each APISIX plugin and 300+ for Kong.

• Scalability: APISIX supports many programming languages and also supports WebAssembly. Kong only supports using Lua for writing plugins.

• Hot Reloading: This is what the team focuses on. APISIX supports hot reloading at the millisecond level when enabling or disabling plugins, and when adding new plugins (such as custom plugins). APISIX also supports the dynamic modification of gateway parameters. However, Kong did not support this feature when IGW conducted technology selection.

• Observability: Both APISIX and Kong support OpenTelemetry, but APISIX can also be connected to Elasticsearch, Prometheus, and SkyWalking. Kong did not yet provide SkyWalking support when IGW was conducting technology selection.

Based on the above concerns and comparison points, the IGW technical team finally chose APISIX as the API gateway.

User Scenarios and Solutions for IGW

Introduction to IGW's System Architecture

IGW's business system was roughly divided into three parts: transaction area, production area, and management area, each using different API gateways. Originally, IGW used NGINX as the web server and reverse proxy. Businesses in the same network classification all used the same NGINX. As a result, each service change or routing update required manual updating and reloading on NGINX.

The diagram above illustrates the system architecture of IGW. The gateway security cluster layer uses multiple frameworks such as Zuul, Spring Cloud Gateway, Kong, and NGINX. Architectural management was not unified and management was relatively cumbersome.

Solution

In order to resolve these challenges, IGW puts emphasis on transforming the gateway clusters into APISIX. As APISIX is deployed on Kubernetes clusters, it allows for the unified management of APIs through YAML in a declarative manner. Additionally, the APISIX Ingress Controller automatically watches changes in Kubernetes resources, enabling real-time synchronization of APISIX configurations, including ApisixRoute, SSL, and more.

Below is IGW's Route synchronization Sequence Diagram after using APISIX.

User Scenarios

After using APISIX, IGW achieved many benefits, among which the IGW team is more concerned about from the business perspective, including intelligent routing, authentication, observability, and traffic control.

Efficient Intelligent Routing

The intelligent routing is primarily demonstrated in the load balancing in layer 4 and layer 7. As shown in the diagram, the routed information synchronized through the APISIX Ingress Controller is accompanied by specific labels, effectively preventing accidental deletion operations.

After testing in the environment, even if data is deleted, Kubernetes is capable of rapidly and automatically synchronizing with APISIX, thereby resolving the issue of data loss.

Furthermore, the routing updates on APISIX achieve millisecond-level responsiveness, with nearly imperceptible configuration synchronization delay, providing an excellent user experience.

Enhanced Authentication

Prior to introducing the unified gateway, each business unit had to individually develop authentication-related components to ensure the security of their data interfaces. On the business side, each system was burdened with developing redundant authentication functionalities, resulting in higher development costs and varying quality of implemented features. Consequently, for security verification, IGW introduced unified authentication and HTTPS redirect plugins.

Before the introduction of the unified gateway, HTTPS certificates for services were decrypted at the High Availability (HA) endpoint, which increased complexity, operational overhead, and security risks.

Recognizing these challenges, the IGW technical team devised a plan to centralize the authentication functionality within the gateway, thereby addressing the aforementioned issues. This approach significantly enhances the research and development efficiency of business systems, allowing developers to focus more on core business development.

By transitioning to APISIX, these drawbacks can be mitigated. APISIX can handle HTTPS termination, load SSL certificates dynamically, and provide centralized and secure management for handling security-related tasks, improving overall system performance and flexibility.

Beyond Monitoring

Previously, the weak support for observability in the IGW's original business system adversely affected system troubleshooting, performance optimization, reliability, and proactive monitoring.

Therefore, the IGW technical team aimed to swiftly integrate multiple plugins provided by APISIX, such as log monitoring and tracing, in order to enhance observability capabilities.

Moreover, the team was actively researching and leveraging Apache SkyWalking for distributed tracing, Prometheus for monitoring purposes, and ELK for efficient log collection. Surprisingly, APISIX supports all these plugins and thus has proven to be an ideal solution that fully meets the expectations and requirements of the IGW technical team, making it the preferred choice.

The diagram above illustrates the service topology of the IGW after the technical team integrated SkyWalking into production. This comprehensive diagram offers a clear and concise visualization of the inter-service invocation relationships, encompassing crucial information like traffic direction through the gateway and success rates. Leveraging this topology diagram, the IGW team can promptly pinpoint the exact location of any errors or issues within the service chain.

Mastering Traffic Control

APISIX has proven to be an effective solution in facilitating versatile traffic control mechanisms for the IGW. By adopting APISIX, the IGW technical team has seamlessly achieved traffic control through canary release and weight-based strategies. This robust capability allows for efficient management of traffic distribution, enabling the team to easily implement canary releases and adjust traffic weights as needed.

In a scenario based on canary release, the gateway needs to call a downstream interface through an HTTP request to obtain specific data, and judge based on the returned results to determine whether the request needs to be sent to the canary release environment.

Based on the weight policy, the services of virtual machines and containers are provided to the outside in parallel. For example, 90% of the traffic hits the virtual machine service and 10% of the traffic hits the cloud service to verify the stability of the cloud service. Currently, the production environment of IGW has been configured with a weight-based canary release.

Achievements After Using APISIX

Built a Unified API Gateway

By implementing APISIX, the IGW technical team has achieved the standardization of the API gateway framework that provides rich traffic management functions such as load balancing, dynamic upstream, canary release, circuit breaking, authentication, and observability.

IGW has implemented unified authentication functionality on the gateway, addressing the issues of redundant development efforts and inconsistent quality of features. This allowed developers to focus more on development, and greatly improved the research and development efficiency of business systems.

Improved Stability of Deploying Routing and Services

Routing updates on APISIX are capable of delivering millisecond-level responses, ensuring swift and efficient performance. Additionally, the synchronization of configurations is nearly instantaneous, resulting in an enhanced user experience.

Besides, the integration of DevOps and pipelines has significantly enhanced the stability of route and service deployments. This streamlined approach has improved the efficiency of routing and service releases, ensuring more reliable and consistent results.

Hot-Reloading Enhances Performance and Uptime

The IGW technical team highly values the power of hot reloading. APISIX offers a seamless hot deployment process for enabling or disabling plugins, facilitating the addition of custom plugins, and enabling dynamic updates to the gateway's parameters.

As a result, modifications to plugin configurations and gateway settings can be instantly applied without the need for system restarts or interruptions. This remarkable hot reloading feature greatly enhances the flexibility and agility of the APISIX platform, empowering developers to make real-time adjustments to their configurations and customize the gateway to meet their specific requirements.

APISIX's hot-reloading capability has greatly alleviated the pressure of service restarts. This feature enables updates and modifications to be applied to services without the need for a complete restart, resulting in improved system performance and reduced downtime.

Summary

The IGW technical team also put forward expectations on further enhancing the functionality, reliability, and performance of APISIX for seamless operation in the future.

The successful implementation of APISIX at Invesco Great Wall has yielded significant benefits and positive outcomes. With APISIX, the IGW team has achieved a unified API gateway framework, resulting in streamlined operations and reduced costs. The integration has also brought about enhanced stability in routing and services, leading to improved performance and minimized disruptions.

In light of Invesco Great Wall's remarkable results, APISIX proves to be a highly effective API gateway solution tailored for the financial services industry, including the fund and securities sector. This successful case study stands as a compelling testament to the benefits of adopting APISIX in finance and strongly recommends its implementation to other organizations in the industry.