Exploring the Benefits of Using an API Gateway for Kafka

Yuan Bao

March 31, 2023

Technology

Brief Introduction of Kafka

Kafka was originally created by LinkedIn as a distributed messaging system that relied on Zookeeper coordination and included multiple partitions and replicas. It was later donated to the Apache Software Foundation. Thanks to its exceptional throughput, persistence, and horizontal scaling capabilities, Kafka has become a widely adopted distributed stream-processing platform within the industry.

Kafka has three primary use cases:

  • Messaging system: This is the most common use case, where it is used for message communication between backend microservices. Kafka allows for system decoupling, traffic shaping, and asynchronous communication.
  • Storage system: Kafka stores messages on disk and includes a multi-replica feature, making it a suitable option for use as a data persistence system. This greatly reduces the risk of data loss.
  • Stream-processing platform: Kafka provides a comprehensive library for stream processing, including operations like windowing, joining, and transformations.

Kafka's Technical Architecture

A standard Kafka cluster consists of multiple Producers, Consumers, Brokers, and a Zookeeper cluster. Zookeeper is the core control component of the Kafka cluster, responsible for managing cluster metadata and controller elections. Producers send messages to Brokers, which store the messages on disk, while Consumers subscribe to and consume messages from the Brokers.

Core Concepts

Kafka Topic and Partition

In Kafka, messages are categorized by topics. Producers usually specify a topic when sending messages, while Consumers typically subscribe to a topic to consume messages.

A topic usually consists of multiple partitions, with each partition containing different messages. When a message is sent to Kafka, it is stored in the appropriate partition based on the partitioning rules. By setting appropriate partitioning rules, messages can be evenly distributed across different partitions.

Producer and Consumer

Producer refers to the party that sends messages, responsible for creating messages and sending them to Kafka brokers.

Consumer refers to the party that receives messages, responsible for connecting to the Kafka cluster's broker, subscribing to a particular topic, and consuming messages.

Kafka Brokers

A Broker can be considered as an independent node or instance of Kafka. A Kafka cluster consists of one or more Brokers.

kafka cluster

Why Kafka Needs an API Gateway

In most cases, when using Kafka as a message system between backend microservices, developers need to choose different Kafka SDKs to develop producer or consumer clients, depending on the language used in the project.

However, this approach has limitations in certain scenarios, such as when the client needs to connect directly to Kafka. In such cases, adding an API gateway between the caller and Kafka can help decouple services. This way, the caller does not need to worry about Kafka or specific communication protocols, which reduces the calling costs. Additionally, an API gateway can provide security support and traffic control capabilities for the APIs exposed by Kafka, ensuring the stability of Kafka services.

Connecting Kafka Directly

Directly connecting to Kafka can result in a number of limitations and risks when using it as a message queue middleware:

  1. Different consumers must adapt to Kafka's communication protocol.
  2. Kafka does not provide solutions for securing exposed APIs, traffic control, and other issues. Without rate-limiting policies, some producers or consumers may consume excessive computing power and bandwidth.
  3. Both consumers and producers must be aware of the Kafka cluster topology, which may be impacted by changes to the Kafka cluster.

API Gateway Enhances Kafka's Usability

Communication Protocol Conversion

When an API gateway is used to proxy Kafka, the client communicates with the API gateway using HTTP protocol, while the API gateway communicates with Kafka using Kafka's protocol. The API gateway converts the client's messages to Kafka's protocol, eliminating the need for the client to adapt to different Kafka message protocols. This significantly reduces development costs and makes it more convenient to use.

Rate Limiting

When resources are limited, the service capacity of Kafka nodes is also constrained. Exceeding this limit could cause the service to crash and result in a chain reaction. Rate limiting can prevent this from happening. In traditional Kafka architecture, clients communicate with Kafka via SDKs. However, if the number of clients or requests is high, it may impact the machine load of Kafka nodes and affect the stability of Kafka's functionality. Adding an API gateway to the architecture makes it easy to incorporate various dimensions of rate-limiting function support to the Kafka cluster, as API gateways excel in this area. These capabilities include:

  • Using the fixed time window algorithm to limit the total number of requests a single client can make to the service within a specified time frame.
  • Restricting the number of concurrent requests a client can make to a single service.
  • Using the leaky bucket algorithm to limit the request rate of a single client to the service.

By implementing these rate limiting functions, Kafka nodes can be effectively protected, ensuring their stability.

Authentication Support

Authentication is also a strong feature of an API gateway. In traditional Kafka architecture, most Kafka ports are accessed within an internal network. If access from a public network is required, complex configurations are needed to ensure security. Using the authentication functionality of the API gateway when proxying Kafka can protect the Kafka-exposed ports while also selectively allowing or denying client access.

Monitoring Capability

There are currently many monitoring products available for Kafka, such as Kafka Eagle, Kafka Monitor, Kafka Manager, etc. Each of them has its own advantages and disadvantages. It is challenging to achieve a universal monitoring capability, and customization costs are relatively high. Moreover, integrating them with internal monitoring systems can be difficult. Building a Kafka monitoring system from scratch is also costly, as the monitoring information for Kafka covers many aspects, and the monitoring metrics provided by Kafka itself require complex processing through Java Management Extension (JMX).

By adding an API gateway to the architecture, the client and API gateway communication is based on the HTTP protocol. As a result, the ecosystem of monitoring software based on HTTP protocol is very rich. This enables the provision of comprehensive observability for Kafka services at very low costs.

Kafka Rolling Upgrades

Kafka services are exposed through broker addresses, which producers and consumers need to provide in their configuration information to connect to a Kafka cluster. The address list is typically formatted as host1:port1, host2:port2, and can contain one or more addresses separated by commas.

However, this configuration method has limitations: it requires Kafka broker addresses to remain fixed and the services to remain stable. Kafka clients assume that all broker addresses are available and randomly select one to use. This can cause problems during rolling upgrades as clients are not aware of which broker to use, and changing broker addresses requires all producer and consumer clients to make changes.

With an API gateway for Kafka, the broker addresses can be configured at the gateway layer, and clients no longer need to focus on the specific details of Kafka brokers. Even if the broker address changes, clients remain unaffected. This makes it easy to perform rolling upgrades for Kafka without worrying about affecting clients.

Summary

Thus, by adding an API gateway to Kafka, it becomes easier to provide rate limiting, authentication, monitoring, and rolling upgrade capabilities for Kafka services through the rich functionality of the API gateway.

Using Apache APISIX to Extend Kafka

Apache APISIX is a high-performance, real-time API gateway under the Apache Software Foundation. It offers a range of sophisticated traffic management features, including load balancing, dynamic upstreams, canary release, circuit breaking, authentication, and observability. As an API gateway, APISIX helps enterprises process API and microservice traffic quickly and securely and is widely used in Kubernetes Ingress and service mesh. By adding an APISIX layer in front of a Kafka service, enterprises can leverage the platform's rich plugin system to enable message proxying, log delivery, rate limiting, monitoring, and other capabilities. With APISIX, both north-south traffic from clients to servers and east-west traffic between enterprise microservices can be effectively processed.

APISIX and Kafka

Proxying Kafka Messages

APISIX offers the capability to proxy Kafka messages. Clients can communicate directly with APISIX, which handles the message protocol conversion between the clients and Kafka.

To enable the Kafka proxy feature in APISIX, we just need to add a route as shown below:

curl -X PUT 'http://127.0.0.1:9180/apisix/admin/routes/Kafka' \
    -H 'X-API-KEY: <api-key>' \
    -H 'Content-Type: application/json' \
    -d '{
    "uri": "/Kafka",
    "plugins": {
        "Kafka-proxy": {
            "sasl": {
                "username": "user",
                "password": "pwd"
            }
        },
        "limit-count": {
            "count": 2,
            "time_window": 60,
            "rejected_code": 503,
            "key_type": "var",
            "key": "remote_addr"
        }
    },
    "upstream": {
        "nodes": {
            "Kafka-server1:9092": 1,
            "Kafka-server2:9092": 1,
            "Kafka-server3:9092": 1
        },
        "type": "none",
        "scheme": "Kafka"
    }
}'

This request creates a route with the URI /Kafka in APISIX, which is associated with three Kafka broker nodes as upstream services. The kafka-proxy plugin is used to add SASL/PLAIN authentication to Kafka requests. The configuration in APISIX supports hot updates, so Kafka brokers can be modified without requiring a restart of the API gateway or disrupting the client.

In addition, the limit-count plugin is enabled for this route to provide support for rate limiting. The plugin's configuration specifies that only two requests are allowed to pass through within a 60 second interval, and any additional requests will be rejected by APISIX with a status code of 503.

Log Pushing

APISIX's kafka-logger plugin allows logs to be pushed as JSON objects to an Apache Kafka cluster.

Here is a configuration example:

curl http://127.0.0.1:9180/apisix/admin/routes/1 \
-H 'X-API-KEY: edd1c9f034335f136f87ad84b625c8f1' -X PUT -d '
{
    "plugins": {
       "kafka-logger": {
            "brokers" : [
              {
                "host" :"127.0.0.1",
                "port" : 9092
              },
              {
                "host" :"127.0.0.1",
                "port" : 9093
              }
            ],
           "kafka_topic" : "test2",
           "key" : "key1"
       }
    },
    "upstream": {
       "nodes": {
           "127.0.0.1:1980": 1
       },
       "type": "roundrobin"
    },
    "uri": "/hello"
}'

APISIX offers more than just the ability to proxy Kafka messages. It also provides tracing, metrics, and logging capabilities through various plugins, covering all aspects of observability. With simple plugin configuration on APISIX, integration with other observability services such as Prometheus, Skywalking, and others is possible, enhancing the monitoring capabilities of Kafka clusters.

Summary

This article highlights the advantages of implementing an API gateway for Kafka, and uses Apache APISIX as a case study to demonstrate how it offers Kafka features such as rate limiting, authentication, monitoring, and rolling upgrades.

Tags:
APISIX BasicsPlugin