How API Gateways Integrate with Kafka

Introduction

API gateways traditionally mediate HTTP-based traffic, managing and securing RESTful APIs. However, as modern architectures increasingly embrace event-driven paradigms powered by messaging systems like Apache Kafka, a new question arises: how do API gateways connect the REST world with Kafka-based streams?

In this article, we explore how API gateways can be extended to support Kafka as a backend target, focusing on architectural strategies, implementation patterns, and practical best practices.

Why Integrate API Gateways with Kafka?

Integrating API gateways with Kafka unlocks several benefits:

• Decoupling: HTTP clients can interact with Kafka producers/consumers without directly coupling to Kafka's protocol.
• Security and Governance: Gateways add authentication, authorization, and rate limiting layers before messages reach Kafka.
• Protocol Bridging: Allows RESTful APIs to produce or consume Kafka messages via HTTP endpoints.
• Unified Observability: API-level metrics and logging can be applied to Kafka traffic.

Use Cases

• Ingesting telemetry or logs from edge services to Kafka
• Providing external-facing APIs to publish business events
• BFF (Backend-for-Frontend) translating frontend requests into Kafka messages
• Building a serverless ingestion layer for Kafka via HTTP

Kafka Primer: Key Concepts

Before diving into gateway integration, it's essential to understand core Kafka components:

• Producer: Publishes messages to Kafka topics
• Consumer: Subscribes and processes messages from topics
• Topic: Logical channel for message categorization
• Partition: Unit of parallelism within a topic
• Broker: Kafka server managing topics and partitions

Integration Models

There are three common models for integrating API gateways with Kafka:

1. Gateway as Kafka Producer (Push Model)

In this model, the gateway acts as a Kafka producer, forwarding HTTP request payloads to a specified Kafka topic.

Architecture

sequenceDiagram
  participant Client
  participant APIGateway
  participant KafkaBroker
  Client->>APIGateway: HTTP POST /send-event
  APIGateway->>KafkaBroker: Produce message to topic
  KafkaBroker-->>APIGateway: ACK
  APIGateway-->>Client: 200 OK

Example: Apache APISIX Kafka Plugin

plugins:
  kafka-logger:
    broker_list:
      - "kafka-broker:9092"
    topic: "events"
    key: "${uri}" # optional
    batch_max_size: 10

2. Gateway as Kafka Consumer (Pull Model)

The gateway can also poll Kafka topics and expose the latest message(s) over a REST endpoint.

Architecture

sequenceDiagram
  participant KafkaBroker
  participant APIGateway
  participant Client
  APIGateway->>KafkaBroker: Poll messages
  KafkaBroker-->>APIGateway: Message batch
  Client->>APIGateway: HTTP GET /read-event
  APIGateway-->>Client: Return latest message(s)

This model requires persistent gateway-side consumers and buffering strategies.

3. Proxying a Kafka REST Bridge

Gateways can proxy traffic to Kafka REST bridges like:

• Confluent REST Proxy
• Redpanda Console HTTP API

This simplifies implementation but depends on external services.

Implementation Considerations

Message Format

• JSON over HTTP is most common
• Convert to Avro, Protobuf, etc. using transformation plugins or schema registries

Performance

• Use batching for higher throughput
• Async production (e.g., via background workers or Kafka buffers)

Error Handling

• Retain message on failure (e.g., local queue, DLQ)
• Return meaningful HTTP errors

Security

• Authenticate clients via JWT or OAuth 2.0
• Encrypt traffic via HTTPS + mTLS
• Apply RBAC on topic-level access

Comparison: Kafka vs RabbitMQ Gateway Integration

Kafka is more aligned with streaming use cases and large-scale ingestion, making it preferable for modern event-driven microservices.

Best Practices

• Define clear contract (schemas) for Kafka messages
• Avoid tight coupling between HTTP paths and topic names
• Monitor gateway-Kafka integration separately (metrics, logs)
• Consider schema validation at the gateway
• Use circuit breakers for downstream Kafka failures

Conclusion

API gateways are no longer confined to REST—they're becoming critical bridges to event-streaming platforms like Kafka. Whether you're building event ingestion APIs or exposing Kafka topics securely to clients, understanding integration patterns helps you design scalable and robust systems.

By leveraging built-in plugins, REST bridges, or custom middleware, developers can enable seamless, observable, and secure communication between HTTP clients and Kafka clusters.

FAQ: Kafka and API Gateway Integration

1. Can API gateways produce messages to Kafka in real-time?

Yes, most gateways (like Apache APISIX) support Kafka plugins that allow real-time message production with optional batching.

2. What if the Kafka broker is down—will the API gateway fail?

It depends on configuration. Gateways can use local queues or buffer retries to minimize data loss, but for guaranteed delivery, a persistent layer or dead-letter strategy should be used.

3. How do I map HTTP requests to specific Kafka topics?

Gateways can be configured with route-level rules or plugins that dynamically resolve topic names based on the URI, headers, or request body.

4. Can I consume Kafka messages over HTTP through the API gateway?

Yes, though it requires polling consumers on the gateway side, which increases complexity. Alternatively, use a REST proxy and have the gateway forward requests.

5. Is schema validation supported at the API gateway level?

Some gateways allow plugin-based validation or integration with schema registries. You can validate JSON schemas or Avro definitions before producing the message.

6. What's the difference between using a REST proxy and native Kafka plugin in the gateway?

REST proxies introduce an extra hop but are easier to manage. Native plugins offer tighter integration, better performance, and more control over error handling and message transformation.

Next Steps

Stay tuned for our upcoming column on the API gateway Guide, where you'll find the latest updates and insights!

Eager to deepen your knowledge about API gateways? Follow our Linkedin for valuable insights delivered straight to your inbox!

If you have any questions or need further assistance, feel free to contact API7 Experts.