How APISIX Facilitates Localized Deployment of Beeto Social Media Platform in the Middle East

Overview

Challenges

• Monolithic service architecture results in high costs in maintenance and operation
• Complex architecture deployment and service calls, and multiple technology stacks

Results

• Unified north-south and east-west traffic, saving resources and manpower costs and enabling dynamic and unified management
• Simplified the deployment architecture, thus lowering the interaction between the gateway and the users
• APISIX’s multiple extension plugins helped Beeto efficiently manage permission verification, route distribution, and health check functions of services
• APISIX enables Beeto to dynamically launch and migrate services, which is developers-friendly

Introduction to Beeto

For the Middle East market, Beeto is an Arabic social media platform, with localized product design and technical architecture. It was ever placed No. 4 on the Top List of the Saudi Arabian iOS App Store, surpassing veteran social platform giant Facebook.

Internet development in the Middle East is mature, with a very high penetration of active users in social networks. Especially in Saudi Arabia, where 90% of people used the Internet in 2019. The penetration rate of active users in Saudi Arabia ranked 9th in 2020.

The internet market's maturity resulted in popular international social software like WhatsApp, YouTube, and Instagram. Nevertheless, there is no similar localized social software like Twitter. Therefore, aiming at the Middle East's "mature internet but little localization", Beeto launched localized product development there.

Localization Matters to Beeto

Benchmarked against feed stream applications like Twitter and Facebook, Beeto has planned a relatively complete framework for deploying its business architecture in the Middle East. For instance, it was committed to satisfying the relationship interaction of social attributes, content consumption (text, video live broadcast, local advertising, etc.), as well as various businesses such as rewards, withdrawals, voting, and the lottery of financial and service categories, and even the requirements of platform supervision and content security reviews.

As previously indicated, the maturity of the Internet in the Middle East market inevitably requires high-quality products. Therefore, the first version of Beeto's business architecture was a complete product that integrated all the features mainstream social software should have. Meanwhile, Beeto's goal is to become the largest Arabic social platform and the best Arabic community in the Middle East with "features of localization". In realizing the goal, Beeto analyzed the localization requirements as below.

Pain Points in Beeto Architecture Design

The localization includes the existing local social needs at the business level, and localization operations at the technical level, such as service deployment and data storage. Those familiar with Weibo or Twitter would know it takes dozens or hundreds of architectural systems to collaborate behind such a huge information flow product. The pain points in Beeto architecture are shown as follows.

Monolithic Service Architecture Causes High Costs

Beeto's services can be divided into eight parts, as below.

The implementation of these businesses requires localized deployment in the Middle East. Each service is a separate monolithic architecture if splitting each business by service.

The monolithic architecture diagram above shows a common deployment architecture. Take Beeto's Feed streaming service as an example. If we want to realize the user's demand for browsing the feed stream, we must support public network access, that is, north-south traffic access; at the same time, the feed streaming service also provides some internal calls in the form of topic business, namely, east-west traffic calls.

Therefore, the overall service supports external and internal call modes. After the load balancing of Layer 7, the user traffic is distributed to different servers, and then different storage resources are called. The east-west traffic is also similar. The Layer 7 cluster handles north-south and east-west traffic, load balancing, authentication, and node monitoring.

When the services of multiple businesses are combined, the overall architecture can be:

As you can see, services exist in the adaptation, business, and basic service layers. The deployment architecture for each of these services has the monolithic architectural details mentioned above, so there are several Layer 7 clusters in between, which is a very large and complex system architecture.

However, since the Beeto product is in the startup stage and the product is launched in the Middle East, but its R&D team is in China, great server and maintenance costs are required. Furthermore, as business increases later, the number of monolithic services will inevitably increase, making it more difficult to control both the costs and maintenance operations.

Difficulty in Launching Multiple Services

In addition to the complexity of the architecture deployment, the calls between services within the cluster are also very complex.

The north-south traffic is distributed to various service pools, and the east-west traffic is also interlaced among various services, with the call relationships between these services intertwined.

Besides, these call relationships need to be maintained by services, which leads to unclear call tracing and poor management.

In addition to the complex call relationships, there are also differences in technology stacks between each service. For example, on the call protocol, some services provide HTTP, while others provide RPC; In terms of the development of programming languages, there is a mix of Java, Go, and other programming languages.

Such a multi-service architecture system will obviously expose the problem of high deployment and maintenance costs when it is implemented locally. Furthermore, Beeto needs to input server costs on each set of Layer 7 services, while the traffic difference of each service leads to unbalanced traffic, resulting in a low utilization rate of resources such as servers and waste of resources.

Since Beeto focused more on business upgrades and iterations, the architecture is designed to facilitate maintenance and unified management, so how to achieve this goal?

APISIX Gateway Powers Beeto's Architecture

To solve the problems of inconvenient service management and high costs and to benefit from the dynamic performance of APISIX with etcd, which is more in line with Beeto's product requirements, APISIX was introduced as an API gateway in the architecture deployment, and a gateway cluster was built, as shown in the figure below.

The APISIX gateway cluster provides extension tools such as a registry center, service control, service monitoring, protocol forwarding, and plugins for all services. Clusters of services can all be registered with the gateway, and new services can be added and removed directly through the gateway.

After the gateway was introduced, the call tracing of the whole cluster was clearer. The north-south traffic is routed and forwarded by the gateway, and the east-west traffic is by the gateway for services on the intranet. At the functional level, APISIX is responsible for maintaining the authentication called by this traffic so that the gateway layer can clearly understand the performance differences and traffic differences between services.

Of course, since the gateway carries all the traffic in this architecture, the number of services will increase later as the service expands, the maintenance cost of the gateway will increase, and new solutions will need to be considered. However, in the current context, the solution is still the best choice.

Practices after Applying APISIX

Apache APISIX, as an API gateway, can handle a variety of policies at the gateway level, such as authentication, service forwarding, and health checks. Therefore, Beeto has made many attempts at the business practice level after introducing APISIX.

Security: Authentication Plugin

North-South Traffic: Cookie

As we mentioned earlier, the access traffic of public network users passes through the gateway. The authentication of public network users is a user request based on cookie authentication. When a user requests to bring a cookie to the gateway, it is verified by the authentication plugin.

As shown in the flowchart above, the plugin will first perform localization validation and then perform authentication verification of the remote service according to the policy. Once the request is cookie-checked, it will be forwarded to the specified service.

The advantages of doing so are twofold:

1. The security of users' cookies is ensured. Only the gateway layer can receive and process cookies during execution because cookies are sensitive data. This prevents security problems caused by inconsistent business processing rules and effectively avoids cookie leakage caused by human factors and irregularities.

2. Reducing the complexity of cookie authentication for services. Cookies need to be verified locally or remotely and require simultaneous service upgrading when cookies are upgraded. APISIX unified management and verification, saving the cost of verification of business services and indicating the results, which can be used for business rules processing, thus ensuring that each business processing is more focused on the business itself.

East-West Traffic: Token

In the diagram below, Service A calls Service B. Generally speaking, an API is all that is needed to call each other. However, in the internal process, it is necessary to understand "who called the API, how it was called, whether it is necessary to check the authority, and whether it is necessary to control the researcher", and so on, which need to be handled internally.

With the APISIX gateway, the process becomes much simpler. The mutual call of all internal services only needs to be registered on the Auth authentication service, and an App key is issued to each service to indicate the identity ID of the service. At the same time, the mutual call of internal service will also pass through the gateway. After carrying the token through the gateway, the gateway will authenticate the token through the token plugins. After the authentication is passed, the authentication identifier will be passed to the called service. During the whole process, the service system will perform authentication registration and complete a mutual call.

Thanks to the token rule of the App key, the above operation can be easily traced back to the source of the call, which can be used for troubleshooting and permission control, and also provide effective control on illegal calls. Therefore, the advantage of unified authentication is that, whether for north-south or east-west traffic, it ensures the security and uniformity of the cluster, which helps greatly in problem tracing and call control.

Scalability: Stateless Service Expansion and Migration

The overall deployment architecture of Beeto's cluster from top to bottom is composed of: APISIX gateway clusters - business service clusters of stateless services - data center clusters of stateful service. When deployed locally in the Middle East, they are all deployed on major cloud clusters. According to the cloud coverage scale in the Middle East and the cloud manufacturers in different regions, it is necessary to consider the expansion and migration of cloud services when deploying services.

In the migration process, Beeto focused on the migration of stateless services. It is unsuitable for dynamic adjustment due to the limited migration cost of the data center; most of the requests are carried by the stateless service, so it is very important to ensure that the stateless service has good scalability.

In Beeto's architecture, service scalability is mainly reflected in two aspects: monolithic service scalability and overall cluster scalability. For example, if a monolithic service runs out of resources and needs to be expanded, APISIX gateways can dynamically add nodes for scaling. Similarly, in cross-cluster or cross-cloud situations, cluster scalability can be achieved by deploying multiple APISIX gateways and migrating different services to different gateway nodes.

For business services, the overall architecture remains unchanged so that dynamic expansion and contraction of various services and service migration can be realized at the gateway layer. The whole process has a clear plan and objective, and once changes are involved, the overall architecture will not be unstable.

Functional Extensibility: Dynamic Forwarding of Services

In addition to these familiar gateway general-purpose scenarios mentioned above, Apache APISIX also provides significant assistance to Beeto in terms of service dynamic forwarding.

Those familiar with the UI and backend of APPs should know that different product pages are provided by different services. A page comprises different modules, the content of which is all sent from the API. Whatever data the API sends down to the module is rendered on the ends. This is a joint client-side rendering specification to make the client-side rendering process more generic and the business processing more flexible.

In the implementation of PageA in the above figure, for example, the client calls the API of Service A, sends the data of the corresponding module, and completes the rendering of PageA. This operation will cause problems that the client needs to maintain each page and API to each service. If the content changes, it is necessary to be solved by publishing, which is very unfriendly in terms of operability and efficiency.

The solution to the above problem is to implement a unified distribution of services in the overall architecture. The client first requests the API address, forwards all requests of this type to one API, processes the request parameters and URL rules for the URL address at the gateway level, and then introduces the distribution plugin. Finally, the parsed requests are forwarded directly to the specified service at the gateway layer according to the configuration rules.

The client only needs to determine the rendering specification throughout the process, not the source of the data. The gateway layer assumes the responsibility of service distribution and forwards the traffic directly. The plugin configuration file in APISIX can be dynamically updated in real-time, and the forwarding rules can be dynamically adjusted, which is very flexible. In fact, for applications like BFF (Backend for Frontend) architecture, such requirements can be solved at the gateway layer.

Conclusion

This article presents the application practices of Beeto's introduction of Apache APISIX from the perspectives of design thinking and businesses.

APISIX supported Beeto in controlling the resource cost and multiple business product lines and helped Beeto realize localized deployment, unified management and operation, and maintenance-friendly scenarios in the Middle East.