Introduction to Beeto
For the Middle East market, Beeto is a social media platform that concentrates on Arabic, with localized product design and technical architecture. It was ever placed No. 4 on the Top List of the Saudi Arabian iOS App Store, surpassing veteran social platform giant Facebook.
In actuality, the Middle East is one of the more mature regions in terms of Internet development, with a very high penetration of active users in social networks. Especially in Saudi Arabia, where 90% of people used the Internet in 2019, and the penetration rate of active users in Saudi Arabia will already be ranked 9th in 2020.
Due to the Internet market’s maturity, local mainstream social software like WhatsApp, YouTube, and Instagram is now available internationally. But in retrospect, you’ll see that the Middle East lacks localized social software like Weibo in China. Therefore, before the creation of Beeto products, we aimed at the direction of "Middle East Internet is mature and high penetration, but little localization" and started the preparation of the product focusing on "features of localization".
Beeto in the Middle East actually is benchmarked against feed stream applications like Twitter and Facebook, so it has planned a relatively complete framework for the deployment of its business architecture from the beginning. For instance, it can meet the relationship interaction of social attributes, content consumption (text, video live broadcast, local advertising, etc.), as well as various businesses such as rewards, withdrawals, voting and lottery of financial and service categories, and even the requirements of platform side supervision and content security review.
As we previously indicated, the maturity of the Internet in the Middle East market inevitably requires high quality for the release of products. Therefore, it is impossible to launch an application with simple functions to penetrate the Middle East market.
So the first version of Beeto’s business architecture was a complete product with all the features that mainstream social software should have in one place. Meanwhile, Beeto’s goal is to become the largest Arabic social platform and the best Arabic community in the Middle East with "features of localization".
Problems in Beeto Architecture Design
In order to localize Beeto, in addition to meeting the existing local social needs at the business level, we also need to do some localization operations at the technical level, such as service deployment and data storage. Those who are familiar with Weibo or Twitter should know that it takes dozens or even hundreds of architectural systems to collaborate behind such a huge information flow product.
Monomeric Service Architecture Design
The current business of Beeto products can be divided into the above. The implementation of these businesses actually requires localized deployment in the Middle East. If we split each business by service, each service is a separate monomeric architecture.
The diagram above shows a common deployment architecture. Take Beeto’s Feed streaming service as an example, If we want to realize the user’s demand for browsing the feed stream, we must support public network access, that is, north-south traffic access; at the same time, the Feed Streaming Service also provides some internal calls in the form of topic business, namely, east-west traffic calls. Therefore, the overall service attribute clearly supports external and internal call modes. After seven layers of load balancing, the user traffic is distributed to different servers and then different storage resources are called. The east-west traffic is also similar. The whole seven-layer cluster is responsible for handling north-south and east-west traffic, load balancing, security authentication, and node monitoring.
When the services of multiple businesses are combined, the overall architecture is as follows:
As you can see, there are several services in the adaptation, business, and base service layers. The deployment architecture for each of these services has the monomeric architectural details mentioned above, so there are several seven-layer clusters in between, which is very large and complex system architecture.
However, since the Beeto product is still in the startup stage, especially since the product itself is launched in the Middle East, but the R&D team is in China, according to the above scale deployment, it needs to invest a very large server cost and maintenance cost. At the same time, as business increases later, the number of monomeric services will inevitably increase, making it more difficult to control both the cost and operation and maintenance operations.
Difficulty in Launching Multiple Services
In addition to the complexity of the architecture deployment mentioned above, the calls between services within the cluster are also very complex.
The north-south traffic is distributed to each service pool, and the east-west traffic is also interlaced among various services, with the call relationships between these services intertwined. For each set of services, these call relationships need to be maintained, which leads to unclear call links and poor management.
In addition to the complex call relationships, there are also differences in technology stacks between each service. For example, on the call protocol, some services provide HTTP, while others provide RPC; In terms of the development of programming languages, there is a mix of Java, Go, and other programming languages.
From the above details, it can be seen that such a multi-service architecture system will obviously expose the problem of high deployment and maintenance costs when it is implemented locally. At the same time, each set of seven-layer services needs to input server costs, while the traffic difference of each service will lead to unbalanced traffic, resulting in a low utilization rate of resources such as servers and waste of resources.
Since the cost of Beeto is currently focused on business upgrades and iterations, the architecture is designed to facilitate maintenance and unified management, so how to achieve this goal?
APISIX Gateway Powers Beeto’s Architecture
To solve the problems of inconvenient service management and large cost investment, and to benefit from the dynamic performance of APISIX with etcd, which is more in line with Beeto’s product requirements, APISIX was introduced as a gateway in the architecture deployment, and a gateway cluster was built, as shown in the figure below.
The gateway cluster provides extension tools such as registry, service control, service monitoring, protocol transcoding, and application plugins for all services. Clusters of services can all be registered with the gateway, and new services can be added and removed directly through the gateway.
At the same time, after the gateway is introduced, the call link of the whole cluster becomes very clear. The north-south traffic is routed and forwarded by the gateway, and east-west traffic is forwarded through the gateway for services on the intranet. At the functional level, APISIX is responsible for maintaining the authentication called by these traffic, so that the gateway layer can clearly understand the performance differences and traffic differences between services.
To sum up, after the introduction of the APISIX gateway for architectural integration:
- Solving the problem of unified north-south and east-west traffic, saving resources and manpower costs, and enabling dynamic unified management;
- The deployment architecture of business services focuses on the services themselves, thus realizing the independent existence of the gateway and business insensitivity;
- Through extension plugins, the permission verification, route distribution, and health check functions of each service are hosted by the gateway;
- The new service launch and service migration can be done dynamically, which is very friendly to O&M.
Of course, since the gateway carries all the traffic in this architecture, the number of services will increase later as the service expands, the maintenance cost of the gateway will increase, and new solutions will need to be considered. However, in the current context, the solution is still the best choice.
Business Practices after Applying APISIX
Apache APISIX as a gateway can handle a variety of policies at the gateway level, such as security authentication, service forwarding, and health checks. Therefore, Beeto has done a lot of attempts at the business practice level after the introduction of APISIX.
Security: Authentication Plugin
North-South Traffic: Cookie
As we mentioned earlier, the access traffic of public network users passes through the gateway. The authentication of public network users is a user request based on Cookie authentication. When a user requests to bring a Cookie to the gateway, it is verified by the authentication plugin on the gateway.
As shown in the flowchart above, the plugin will first perform localization validation, and then perform authentication verification of the remote service according to the policy. Once the request is Cookie-checked, it is forwarded to the specified service.
The advantages of doing so are twofold:
- Reducing the complexity of Cookie authentication for each service. As mentioned earlier, Cookies need to be verified locally or remotely in the process, and when Cookies are upgraded, each service needs to be upgraded simultaneously. Through the gateway for unified management and verification, the cost of verification is eliminated in the processing of business services, and only the results are concerned, and the results are used for business rules processing, thus ensuring that each business processing is more focused on the business itself.
East-West Traffic: Token
In the diagram below, Service A calls Service B. Generally speaking, an API is all that is needed to call each other. However, in the internal process, it is necessary to understand "who called the API, how it was called, whether it is necessary to check the authority, and whether it is necessary to control the researcher", and so on, which need to be handled internally.
With the APISIX gateway, the process becomes much simpler. The mutual call of all internal services only needs to be registered on the Auth authentication service, and an App key is issued to each service to indicate the identity ID of the service. At the same time, the mutual call of internal service will also pass through the gateway. After carrying the Token through the gateway, the gateway will conduct Token authentication through the Token plugin. After the authentication is passed, the authentication identifier will be passed to the called service. During the whole process, the service system will perform authentication registration and complete a mutual call.
Thanks to the Token rule of the App key, the above operation can be easily traced back to the source of the call, which can be used for troubleshooting and permission control, and also provide effective control on illegal calls.
Therefore, the advantage of unified authentication, whether for north-south or east-west traffic, is that it ensures the security and uniformity of the cluster, which helps a lot in problem tracing and call control.
Scalability: Stateless Service Expansion and Migration
Currently, the overall deployment architecture of Beeto’s cluster is composed of APISIX gateway cluster – stateless service business service cluster – stateful service data center cluster from top to bottom. When deployed locally in the Middle East, they are all deployed on major cloud clusters. According to the cloud coverage scale in the Middle East and the cloud manufacturers in different regions, it is necessary to consider the expansion and migration of cloud services when deploying services.
In the migration process, Beeto focused on the migration of stateless services. Because of the migration cost of the data center, it is not suitable for dynamic adjustment; at the same time, most of the requests are carried by the stateless service, so it is very important to ensure that the stateless service has good scalability.
In Beeto’s architecture, service scalability is mainly reflected in two aspects, namely, monomeric service scalability and overall cluster scalability. For example, if a monomeric service runs out of resources and needs to be expanded, APISIX gateways can be used to dynamically add nodes to achieve the scaling. Similarly, in cross-cluster or cross-cloud situations, cluster scalability can be achieved by deploying multiple APISIX gateways and migrating different services to different gateway nodes.
For business services, the overall architecture remains unchanged, so that dynamic expansion and contraction of various services and service migration can be realized at the gateway layer. The whole process has a clear plan and objective, and once changes are involved, the overall architecture will not be unstable.
Functional Extensibility: Dynamic Forwarding of Services
In addition to these familiar gateway general-purpose scenarios mentioned above, Apache APISIX also provides significant assistance to Beeto in terms of the service dynamic forwarding.
Those who are familiar with the UI and back-end of APPs should know that different product pages are provided by different services. A page is composed of different modules, the content of which is all sent from the API. Whatever data the API sends down to the module is rendered on the end. This is a joint client-side rendering specification to make the client-side rendering process more generic and the business processing more flexible.
In the implementation of PageA in the above figure, for example, the client calls the API of Service A, sends the data of the corresponding module, and completes the rendering of PageA. This operation will cause problems that the client needs to maintain each page and API to each service. If the content changes, it is necessary to be solved by publishing, which is very unfriendly in terms of operability and efficiency.
The main idea to solve the above problem is to implement a unified distribution of services in the overall architecture. The client first requests the API address, forwards all requests of this type to one API, processes the request parameters and URL rules for the URL address at the gateway level, and then introduces the distribution plugin. Finally, the parsed requests are forwarded directly to the specified service at the gateway layer according to the configuration rules.
The client only needs to determine the rendering specification throughout the process, not the source of the data. The gateway layer assumes the responsibility of service distribution and forwards the traffic directly. The plugin configuration file in APISIX can be dynamically updated in real-time, and the forwarding rules can be dynamically adjusted, which is very flexible. In fact, for applications like BFF (Backend for Frontend) architecture, such requirements can be solved at the gateway layer.
This article presents Beeto’s design thinking and some business-related application practices after the introduction of the Apache APISIX gateway from Beeto’s product design perspective.
With the support of the APISIX gateway, on the premise of controlling resource cost and multiple business product lines, it also helps Beeto realize the scenario of localized deployment, unified management, and operation and maintenance-friendly in the Middle East.