How to allow access to both internal and external networks when deploying kafka service with docker-compose?

background

Recently I am learning Kafka related knowledge and need to build my own Kafka environment. After comprehensive consideration, I decided to use docker-compose to manage and maintain this environment.

docker-compose.yml

Bitnami's yml file is very good and is used directly here.

version: "2"

services:
  zookeeper:
    image: docker.io/bitnami/zookeeper:3.8
    ports:
      - "2181:2181"
    volumes:
      - "zookeeper_data:/bitnami"
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: docker.io/bitnami/kafka:3.4
    ports:
      - "9092:9092"
    volumes:
      - "kafka_data:/bitnami"
    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

volumes:
  zookeeper_data:
    driver: local
  kafka_data:
    driver: local

First experience with docker-compose

After downloading or creating the above yml file, start:

$ ls
docker-compose.yml

$ docker-compose up -d
Creating network "kafka_default" with the default driver
Pulling kafka (docker.io/bitnami/kafka:3.4)...
3.4: Pulling from bitnami/kafka
55154658374f: Pull complete
Digest: sha256:659549c08f8a1cfce344d31b608ec2d039a66a9b610423c4bc390c486a8cebbd
Status: Downloaded newer image for bitnami/kafka:3.4
Creating kafka_zookeeper_1 ... done
Creating kafka_kafka_1     ... done

$ docker ps -a       
CONTAINER ID   IMAGE                   COMMAND                  CREATED         STATUS         PORTS                                                  NAMES
4fe1da00ef96   bitnami/kafka:3.4       "/opt/bitnami/script…"   2 minutes ago   Up 2 minutes   0.0.0.0:9092->9092/tcp                                 kafka_kafka_1
bc018108f95e   bitnami/zookeeper:3.8   "/opt/bitnami/script…"   2 minutes ago   Up 2 minutes   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp   kafka_zookeeper_1

After the service is started, you can enter the container to access the kafka service

$ docker exec -it 4fe1 /bin/bash

Allow external access

We use our java client to connect to the kafka service in the above docker container, but the result keeps reporting errors.

Error connecting to node 4fe1da00ef96:9092 (id: 1001 rack: null)
java.net.UnknownHostException: 4fe1da00ef96: nodename nor servname provided, or not known

This is because by default, the kafka service only allows local connections.

Update docker-compose.yml to open external access.

Add environment variables

    environment:
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
+     - KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CLIENT:PLAINTEXT,EXTERNAL:PLAINTEXT
+     - KAFKA_CFG_LISTENERS=CLIENT://:9092,EXTERNAL://:9093
+     - KAFKA_CFG_ADVERTISED_LISTENERS=CLIENT://kafka:9092,EXTERNAL://localhost:9093
+     - KAFKA_CFG_INTER_BROKER_LISTENER_NAME=CLIENT

exposed port

    ports:
-     - '9092:9092'
+     - '9093:9093'

Stop and delete containers, and restart new ones

$ docker-compose down
Stopping kafka_kafka_1     ... done
Stopping kafka_zookeeper_1 ... done
Removing kafka_kafka_1     ... done
Removing kafka_zookeeper_1 ... done
Removing network kafka_default

$ docker-compose up -d
Creating network "kafka_default" with the default driver
Creating kafka_zookeeper_1 ... done
Creating kafka_kafka_1     ... done

$ docker ps -a
CONTAINER ID   IMAGE                   COMMAND                  CREATED          STATUS          PORTS                                                  NAMES
13ba5ed71886   bitnami/kafka:3.4       "/opt/bitnami/script…"   24 seconds ago   Up 22 seconds   0.0.0.0:9092-9093->9092-9093/tcp                       kafka_kafka_1
e4ca7940e7ab   bitnami/zookeeper:3.8   "/opt/bitnami/script…"   25 seconds ago   Up 24 seconds   2888/tcp, 3888/tcp, 0.0.0.0:2181->2181/tcp, 8080/tcp   kafka_zookeeper_1

Update hosts

View the host IP information and map the kafka service name managed by docker-compose to the current host IP

$ ifconfig | grep 192 
	inet 192.168.10.241 netmask 0xffffff00 broadcast 192.168.10.255

$ cat /etc/hosts
# docker
192.168.10.241 kafka

At this point, we can connect to the kafka service in java code.

properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9092");
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "127.0.0.1:9093");
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.10.241:9092");
properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.10.241:9093");

The test found that all of these methods can be connected. Why is this? Welcome to leave a message and share.

Another way to configure parameters:

- KAFKA_CFG_LISTENERS=PLAINTEXT://0.0.0.0:9092

- KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://192.168.77.159:9092