As a software architect with extensive experience working with microservices systems, I often come across a recurring question: "Should I use RabbitMQ or Kafka?" For some reason, many developers think these technologies are interchangeable. of. While this is true in some cases, there are fundamental differences between RabbitMQ and Kafka.

Therefore, different scenarios require different solutions, and choosing the wrong solution will seriously affect our software development, design and subsequent maintenance of the software.

The goal of this article is to first introduce the basic asynchronous messaging patterns. Then move on to introduce RabbitMQ and Kafka and their internal structures. Part 2 highlights the key differences between these platforms, their various advantages and disadvantages, and how to choose between the two.

Asynchronous messaging pattern

Asynchronous messaging is a messaging scheme in which message production by the producer is decoupled from message processing by the consumer. In messaging systems, we usually divide it into two main messaging modes: queue mode and publish/subscribe mode.

queue mode

In the queue pattern, the queue temporarily decouples producers from consumers. Multiple producers can send messages to the same queue. Then when the consumer processes the message, the message is locked and then removed from the queue, and is no longer available.

Queue mode usually means that a message can only be processed by one consumer.

message queue

As a side note, if a consumer cannot process a message, the messaging platform typically returns the message to the queue for use by other consumers. In addition to decoupling, queues allow us to scale producers and consumers and provide fault tolerance for error handling.

publish/subscribe pattern

In the publish/subscribe pattern, a single message can be received and processed by multiple subscribers simultaneously.

publish/subscribe

For example, this pattern allows a publisher to notify all subscribers that something has happened in the system. In RabbitMQ, a topic is a specific type of pub/sub implementation (an exchange type to be precise), but in this article, I'll refer to the topic as the representation of the entire pub/sub.

Generally speaking, there are two types of subscriptions:

• A temporary subscription, where the subscription is only valid while the consumer is up and running. Once a consumer is closed, their subscriptions and unprocessed messages are lost.
• Durable subscriptions are maintained as long as they are not explicitly deleted. When a consumer shuts down, the messaging platform maintains the subscription and can resume message processing later.

RabbitMQ

RabbitMQ is an implementation of a message broker—often called a service bus. It natively supports the above two messaging modes. Other popular implementations of message brokers include ActiveMQ, ZeroMQ, Azure Service Bus, and Amazon Simple Queue Service (SQS). All of these implementations have a lot in common, and many of the concepts described in this article apply to most of them.

Tails

RabbitMQ supports classic message queues out of the box. Developers define named queues to which publishers can then send messages. In turn, consumers use the same queue to retrieve messages to process them.

Message exchanges

RabbitMQ implements pub/sub through the use of message exchanges. Publishers publish their messages to the message exchange without knowing who the subscribers of these messages are.

Each consumer that subscribes to the exchange creates a queue, and the message exchange queues the resulting messages for consumption by the consumer. It can also filter messages for certain subscribers based on various routing rules.

RabbitMQ message exchange

It is worth noting that RabbitMQ supports temporary subscriptions and durable subscriptions. Consumers can decide the type of subscription they want to use through RabbitMQ's API.

Due to the architecture of RabbitMQ, we can also create a hybrid approach where some subscribers form consumer groups that work together to process messages as competing consumers on a specific queue. In this way we implement a publish/subscribe pattern while also allowing some subscriber extensions to handle received messages.

Publish/subscribe and queue combined

Apache Kafka

Apache Kafka is a distributed stream processing platform.

Unlike RabbitMQ, which is based on queues and exchanges, Kafka's storage layer is implemented using partitioned transaction logs. Kafka also provides Streams API to process streams in real time, and Connectors API to easily integrate with various data sources. However, these are beyond the scope of this article.

Cloud service providers provide alternative solutions for Kafka's storage layer. These solutions include Azure Event Hubs and, to some extent, AWS Kinesis Data Streams. There are also cloud-specific open source alternatives to Kafka's stream processing capabilities, again, these are beyond the scope of this article.

Topics

Kafka does not implement the concept of queues. Kafka stores collections of records in categories called topics.

For each topic, Kafka maintains a partitioned message log. Each partition is an ordered, immutable sequence of records to which messages are continuously appended.

Kafka appends messages to these partitions as they arrive. By default, it uses a round-robin partitioner to spread messages evenly between partitions.

Producers can modify this behavior to create logical message flows. For example, in a multi-tenant application, we might want to create a logical message flow based on the tenant ID of each message. In an IoT scenario, we may want to continuously map the identity of each producer to a specific partition. Ensure that all messages from the same logical flow are mapped to the same partition to guarantee that they are delivered to consumers in order.

Kafka producers

Consumers consume messages by maintaining offsets (or indexes) of these partitions and reading them sequentially.

A single consumer can use multiple topics, and the consumer can scale up to the number of available partitions.

Therefore, when creating a topic, you should carefully consider the expected throughput of messaging for that topic. A group of consumers who consume a certain topic together is called a consumer group. Kafka's API is generally responsible for the balancing of partition processing among consumers in a consumer group and the storage of consumers' current partition offsets.

Kafka consumers

Using Kafka to implement message passing

Kafka's internal implementation actually reflects the pub/sub model very well.

Producers can send messages to specific topics, and multiple consumer groups can consume the same message. Each consumer group can be scaled independently to handle the load. Since consumers maintain their partition offsets, they can choose between durable subscriptions (which maintain their offsets across reboots) or ephemeral subscriptions (i.e. discard offsets and start with the latest record from each partition on each startup Restart).

Kafka is actually not suitable for queue mode message delivery. Of course we can create a topic with only one consumer group to simulate a classic message queue. But this has several disadvantages, which we will discuss in detail in Part 2 of this article.

Part 2 article address: https://betterprogramming.pub/rabbitmq-vs-kafka-1779b5b70c41

It is worth noting that Kafka retains messages in partitions for a pre-configured time period regardless of whether the consumer consumes the messages. This retention means consumers are free to reread past messages. In addition, developers can also use Kafka's storage layer to implement mechanisms such as event sourcing and audit logs.

Conclusion

Although RabbitMQ and Kafka are sometimes interchangeable, their implementations are quite different. Therefore, we cannot consider them as members of the same category of tools. One is a message broker and the other is a distributed streaming platform.

As solutions architects, we should recognize these differences and actively consider what types of solutions should be used for a given scenario.