Set up local Apache Kafka and connect it with Docker

2024.02.18

I. Introduction

Many companies have large amounts of data and need a way to streamline and dump all incoming data to a different location. This is called a Kafka Sink.

In a production environment, it is not recommended to host Kafka yourself. Instead, it is recommended to pay a cloud service provider to host it on AWS. This is Amazon Managed Streaming for Apache Kafka (MSK). However, it is very inconvenient to test Kafka locally. Therefore, you can choose to use Docker to quickly spin up an Apache Kafka locally to develop what you need before deploying it to a test or production environment. This allows for faster development and verification of code without worrying about environmental issues.

2. Use Docker and Docker Compose to quickly start Kafka

1. Configure Docker Compose file

First, create a setup to run single-node Kafka using Docker and Docker Compose. The following is an example Docker Compose file:

version: "2"
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 11.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
twenty one.
twenty two.
twenty three.

2. Check the status of Kafka and Zookeeper settings

Once completed, you can use the following two commands to check the status of your Kafka and Zookeeper settings.

nc -z localhost 22181
nc -z localhost 290921.
2.

If everything runs smoothly, both tests should return success.

3. Use Offset Explorer 2 to view Kafka

1. Download Offset Explorer 2

Now, you can download the Kafka tool (Offset Explorer 2, download address: https://kafkatool.com/download.html), which is a UI tool for Kafka that supports viewing Kafka in UI mode without having to remember the command line.

2. Configure Offset Explorer 2

After downloading the tool, you need to configure the settings. In the configuration settings, go to the Advanced tab and set localhost:29092 as the boot gateway, and then you can start using it!

NEWS