Why AISpeech Chooses Apache APISIX Instead of NGINX as k8s Ingress Controller

Preface

Hi everyone, I'm Jin Wei from AISpeech, a high-tech company specialized in computer speech recognition and analysis, and I'm here to talk about the integration of Apache APISIX with K8s instead of native ingress.

At the time of writing, Apache APISIX has already been applied to our production environment and taken over some of the business entry traffic. We are gradually migrating the native ingress traffic over, as shown in the figure below.

Actually, APISIX-ingress-controller registers the pod ip to the upstream node, so that the business traffic can access the pod directly and bypass the kube DNS. You can implement some particular load balancing policies via plugins based on it. Therefore, on one hand, we rely on this dynamic routing capability of Apache APISIX for traffic distribution. On the other hand, we add some custom plugins to meet business needs.

From the diagram above, APISIX-ingress-controller looks repetitive with the native ingress. In this article, we'll briefly explain why we have abandoned the native ingress and built our own ingress controller based on Apache APISIX.

What Is Ingress

In a word, Ingress is a way for Kubernetes to handle external traffic.

What Problem Ingress Solves

Since the services within a Kubernetes cluster are virtual networks, external traffic requires at least a public ip and a properly mapped port to access the internal services inside of the cluster.

Kubernetes has a variety of ways (NodePort, LoadBalancer, Ingress, …) to expose interfaces to access Kubernetes internal services. In comparison, Ingress is definitely a more economical way to achieve reverse proxy by exposing a limited number of public IPs.

Speaking of reverse proxies, we can also build an NGINX directly to do so, but it is significantly more difficult to maintain to synchronize the readily variable service status in Kubernetes in NGINX.

The good news is that Kubernetes officially provides and maintains an NGINX ingress controller to help solve reverse proxy. With this NGINX ingress controller, we can proxy all the traffic that wants to access Kubernetes and point it to the backend services correctly.

Problems with Kubernetes Native Ingress Controller

The NGINX ingress controller helps us maintain status synchronization between the Kubernetes cluster and NGINX, and provides basic reverse proxy capabilities, so why we build our own ingress? We expect more from the ingress controller to meet more business needs.

After using the Kubernetes native ingress controller, we found the following issues to be prominent:

1. Reloading Problem

   Kubernetes native ingress is designed to pass YAML configuration files to the ingress controller, convert them to NGINX configuration files, and then trigger reloading to make the configuration take effect.

   This is unacceptable, especially when traffic uses long connections, which may lead to accidents.

   In contrast, Apache APISIX supports configurations hot reloading, so you can define and modify routes at any time without triggering an NGINX reload.

2. Writing scripts and filling parameters in annotation

   The native ingress controller supports script snippets defined by annotation in YAML file, which feels like a temporary solution to support advanced features and is, frankly, really unmanageable. The large number of annotation scripts causes problems for the DevOps staff.

   In Apache APISIX, we can write logic via plugin code to expose a simple configuration interface that facilitates configuration maintenance and avoids scripting interference with the DevOps staff.

3. Lack of supporting for stateful load balancing

   Advanced load balancing policies are not supported, such as “session persistent”, etc. Kubernetes is an operations-oriented, containerized application management system, which may have something to do with the fact that Kubernetes promotes a stateless deployment approach, so Kubernetes will not officially support these contradictory load balancing policies in the near future. In fact, Google has already tried to address these issues with its service mesh solution (Istio). Istio's architecture is perfect, but at the expense of performance, which may be solved with mixer v2.

   Since Kubernetes supports scaling, we can also extend Apache APISIX to meet our advanced load balancing needs, since Apache APISIX not only supports “session persistent” and other load balancing natively, but also has the ability to scale the balancer phase.

4. Dynamic weights

   Business services often need to control traffic by percentage, which becomes a problem in Kubernetes.

   Although Kubernetes supports IPVS (IP Virtual Server) after 1.8, neither the start parameters of “kube-proxy” nor the annotation of “kube-route” are as easy to use as Apache APISIX, which internally abstracts route, service, consumer, upstream, plugin, and other major objects. Adjusting the weight of such operations is naturally supported, simply by modifying the “node weight” under “upstream”.

5. Inflexible extension capabilities

   Although Ingress was originally designed to address external traffic, there is no less demand for external traffic than for internal traffic.

   Service level canary release, circuit breaking, flow control, authentication, traffic control, and other requirements are more commonly implemented on Ingress.

   Apache APISIX provides plugin support for scalability, and in addition to the official plugins, you can customize plugins to meet your own characteristics.

   There are also some configuration issues caused by ConfigMap and Namespaces, which are related to the way we use them and are not generic, so I won't go into them here.

Apache APISIX Ingress Controller

Due to the powerful routing and scaling capabilities of Apache APISIX, using Apache APISIX as an implementation of Ingress can easily solve the above mentioned pain points and provide the community with an additional Ingress controller option. The timing diagram is as follows:

To implement the Apache APISIX ingress controller, we need to solve two types of fundamental problems: one is to solve the synchronization between the Kubernetes cluster and Apache APISIX status; the other is to define the objects in Apache APISIX in Kubernetes (CRD).

In order to quickly integrate Apache APISIX and take advantage of it, we created the Apache APISIX ingress controller project (everyone is welcome to participate), which currently implements the first type of basic problem: synchronizing Kubernetes pod information to Apache APISIX upstream, while implementing a primary backup to solve its own high availability problems.

Since Kubernetes uses YAML to define cluster status declaratively, we need to define CRD (Custom Resource Definitions) for the objects (route/service/upstream/plugin) in Apache APISIX to be integrated into Kubernetes.

Also, to facilitate the migration of existing Kubernetes ingress users, we will try to be compatible with the existing ingress configuration items.

These features will be the goal of our next efforts, and we welcome your participation.