Can't connect to local kafka broker using docker compose

See original GitHub issue

Hi, I want to try kafka-ui but it didn’t work for me 😿 Could you help me with it? 🙏

Describe the bug

I wasn’t able to connect to local Kafka broker using docker-compose in this document.

But I was able to connect to local Kafka with Spring Kafka and Conduktor. (sorry for mentioning another tool)

Set up I just followed the kafka-ui docker-compose.md.

The version of app was v0.3.3 and I used latest docker image (and also I tried master tag too)

Steps to Reproduce

run kafka docker compose (from https://github.com/conduktor/kafka-stack-docker-compose/blob/master/zk-single-kafka-single.yml)

# export environment before running
export DOCKER_HOST_IP=127.0.0.1

docker-compose -f zk-single-kafka-single.yml up

zk-single-kafka-single.yml

version: '2.1'

services:
  zoo1:
    image: confluentinc/cp-zookeeper:6.2.1
    hostname: zoo1
    container_name: zoo1
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: zoo1:2888:3888

  kafka1:
    image: confluentinc/cp-kafka:6.2.1
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9092"
      - "9999:9999"
    environment:
      KAFKA_ADVERTISED_LISTENERS: LISTENER_DOCKER_INTERNAL://kafka1:19092,LISTENER_DOCKER_EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_DOCKER_INTERNAL:PLAINTEXT,LISTENER_DOCKER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_DOCKER_INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_JMX_PORT: 9999
      KAFKA_JMX_HOSTNAME: ${DOCKER_HOST_IP:-127.0.0.1}
    depends_on:
      - zoo1

run kafka-ui docker compose

version: '2'
services:
  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    ports:
      - "8080:8080"
    restart: always
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=localhost:9092
      - KAFKA_CLUSTERS_0_ZOOKEEPER=localhost:2181

Additional context

$ docker-compose up

Creating network "kafka-ui_default" with the default driver
Creating kafka-ui ... done
Attaching to kafka-ui
kafka-ui    | 
kafka-ui    |   _  __      __ _               _   _ ___
kafka-ui    |  | |/ /__ _ / _| | ____ _      | | | |_ _|
kafka-ui    |  | ' // _` | |_| |/ / _` |_____| | | || |
kafka-ui    |  | . \ (_| |  _|   < (_| |_____| |_| || |
kafka-ui    |  |_|\_\__,_|_| |_|\_\__,_|      \___/|___|
kafka-ui    | 
kafka-ui    | 
kafka-ui    | 2022-01-28 07:05:04,766 INFO  [background-preinit] o.h.v.i.u.Version: HV000001: Hibernate Validator 6.2.0.Final
kafka-ui    | 2022-01-28 07:05:04,810 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 13.0.9 on 469a81e5874b with PID 1 (/kafka-ui-api.jar started by root in /)
kafka-ui    | 2022-01-28 07:05:04,812 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v2.5.6, Spring v5.3.12
kafka-ui    | 2022-01-28 07:05:04,813 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to default profiles: default
kafka-ui    | 2022-01-28 07:05:07,669 INFO  [main] o.s.d.r.c.RepositoryConfigurationDelegate: Bootstrapping Spring Data LDAP repositories in DEFAULT mode.
kafka-ui    | 2022-01-28 07:05:07,756 INFO  [main] o.s.d.r.c.RepositoryConfigurationDelegate: Finished Spring Data repository scanning in 71 ms. Found 0 LDAP repository interfaces.
kafka-ui    | 2022-01-28 07:05:09,247 INFO  [main] o.s.b.a.s.r.ReactiveUserDetailsServiceAutoConfiguration: 
kafka-ui    | 
kafka-ui    | Using generated security password: ba99fb2b-1edd-42fd-8674-a8142756488d
kafka-ui    | 
kafka-ui    | 2022-01-28 07:05:09,652 INFO  [main] c.p.k.u.s.DeserializationService: Using SchemaRegistryAwareRecordSerDe for cluster 'local'
kafka-ui    | 2022-01-28 07:05:10,663 INFO  [main] o.s.b.a.e.w.EndpointLinksResolver: Exposing 2 endpoint(s) beneath base path '/actuator'
kafka-ui    | 2022-01-28 07:05:10,953 WARN  [main] c.p.k.u.c.a.DisabledAuthSecurityConfig: Authentication is disabled. Access will be unrestricted.
kafka-ui    | 2022-01-28 07:05:11,280 INFO  [main] o.s.l.c.s.AbstractContextSource: Property 'userDn' not set - anonymous context will be used for read-write operations
kafka-ui    | 2022-01-28 07:05:11,862 INFO  [main] o.s.b.w.e.n.NettyWebServer: Netty started on port 8080
kafka-ui    | 2022-01-28 07:05:11,923 INFO  [main] c.p.k.u.KafkaUiApplication: Started KafkaUiApplication in 8.655 seconds (JVM running for 9.886)
kafka-ui    | 2022-01-28 07:05:12,003 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: local
kafka-ui    | 2022-01-28 07:05:12,038 INFO  [parallel-1] o.a.k.c.a.AdminClientConfig: AdminClientConfig values: 
kafka-ui    |   bootstrap.servers = [localhost:9092]
kafka-ui    |   client.dns.lookup = use_all_dns_ips
kafka-ui    |   client.id = 
kafka-ui    |   connections.max.idle.ms = 300000
kafka-ui    |   default.api.timeout.ms = 60000
kafka-ui    |   metadata.max.age.ms = 300000
kafka-ui    |   metric.reporters = []
kafka-ui    |   metrics.num.samples = 2
kafka-ui    |   metrics.recording.level = INFO
kafka-ui    |   metrics.sample.window.ms = 30000
kafka-ui    |   receive.buffer.bytes = 65536
kafka-ui    |   reconnect.backoff.max.ms = 1000
kafka-ui    |   reconnect.backoff.ms = 50
kafka-ui    |   request.timeout.ms = 30000
kafka-ui    |   retries = 2147483647
kafka-ui    |   retry.backoff.ms = 100
kafka-ui    |   sasl.client.callback.handler.class = null
kafka-ui    |   sasl.jaas.config = null
kafka-ui    |   sasl.kerberos.kinit.cmd = /usr/bin/kinit
kafka-ui    |   sasl.kerberos.min.time.before.relogin = 60000
kafka-ui    |   sasl.kerberos.service.name = null
kafka-ui    |   sasl.kerberos.ticket.renew.jitter = 0.05
kafka-ui    |   sasl.kerberos.ticket.renew.window.factor = 0.8
kafka-ui    |   sasl.login.callback.handler.class = null
kafka-ui    |   sasl.login.class = null
kafka-ui    |   sasl.login.refresh.buffer.seconds = 300
kafka-ui    |   sasl.login.refresh.min.period.seconds = 60
kafka-ui    |   sasl.login.refresh.window.factor = 0.8
kafka-ui    |   sasl.login.refresh.window.jitter = 0.05
kafka-ui    |   sasl.mechanism = GSSAPI
kafka-ui    |   security.protocol = PLAINTEXT
kafka-ui    |   security.providers = null
kafka-ui    |   send.buffer.bytes = 131072
kafka-ui    |   socket.connection.setup.timeout.max.ms = 30000
kafka-ui    |   socket.connection.setup.timeout.ms = 10000
kafka-ui    |   ssl.cipher.suites = null
kafka-ui    |   ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
kafka-ui    |   ssl.endpoint.identification.algorithm = https
kafka-ui    |   ssl.engine.factory.class = null
kafka-ui    |   ssl.key.password = null
kafka-ui    |   ssl.keymanager.algorithm = SunX509
kafka-ui    |   ssl.keystore.certificate.chain = null
kafka-ui    |   ssl.keystore.key = null
kafka-ui    |   ssl.keystore.location = null
kafka-ui    |   ssl.keystore.password = null
kafka-ui    |   ssl.keystore.type = JKS
kafka-ui    |   ssl.protocol = TLSv1.3
kafka-ui    |   ssl.provider = null
kafka-ui    |   ssl.secure.random.implementation = null
kafka-ui    |   ssl.trustmanager.algorithm = PKIX
kafka-ui    |   ssl.truststore.certificates = null
kafka-ui    |   ssl.truststore.location = null
kafka-ui    |   ssl.truststore.password = null
kafka-ui    |   ssl.truststore.type = JKS
kafka-ui    | 
kafka-ui    | 2022-01-28 07:05:12,191 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka version: 2.8.0
kafka-ui    | 2022-01-28 07:05:12,192 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka commitId: ebb1d6e21cc92130
kafka-ui    | 2022-01-28 07:05:12,192 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka startTimeMs: 1643353512188
kafka-ui    | 2022-01-28 07:05:12,268 WARN  [kafka-admin-client-thread | adminclient-1] o.a.k.c.NetworkClient: [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
kafka-ui    | 2022-01-28 07:05:12,375 WARN  [kafka-admin-client-thread | adminclient-1] o.a.k.c.NetworkClient: [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.
kafka-ui    | 2022-01-28 07:05:12,478 WARN  [kafka-admin-client-thread | adminclient-1] o.a.k.c.NetworkClient: [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

using Conduktor

Issue Analytics

State:
Created 2 years ago
Comments:5 (2 by maintainers)

Top GitHub Comments

3reactions

kid1412621commented, May 19, 2022

sharing my setup:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    restart: unless-stopped
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  broker:
    image: confluentinc/cp-kafka:latest
    container_name: broker
    restart: unless-stopped
    ports:
    # To learn about configuring Kafka for access across networks see
    # https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc/
      - "9092:9092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_INTERNAL:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_INTERNAL://broker:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
      KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1

  create-topics:
    image: confluentinc/cp-kafka:latest
    hostname: create-topics
    container_name: create-topics
    depends_on:
      - broker
    command: "
      bash -c 'cub kafka-ready -b broker:29092 1 120 && \
      kafka-topics --create --if-not-exists --bootstrap-server broker:29092 --partitions 2 --replication-factor 1 --topic user-event'"
    environment:
      KAFKA_BROKER_ID: ignored
      KAFKA_ZOOKEEPER_CONNECT: ignored

  kafka-ui:
    image: provectuslabs/kafka-ui
    container_name: kafka-ui
    depends_on:
      - broker
    ports:
      - "8092:8080"
    restart: unless-stopped
    environment:
      - KAFKA_CLUSTERS_0_NAME=local
      - KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=broker:29092
      - KAFKA_CLUSTERS_0_ZOOKEEPER=zookeeper:2181

1reaction

azatsafincommented, Jan 28, 2022

Hello @ivvve If you would like to use two different docker-compose files you have to use host IP address for Kafka, and setup communication between Kafka and KafkaUI via it. Docker container can’t connect to host port by ‘localhost’. This is from your log. bootstrap.servers = [localhost:9092]

If would like just to try KafkaUI, it is easiest to use one file which combine all your needs (Kafka, Zookeeper, KafakaUI). For example https://github.com/provectus/kafka-ui/blob/master/docker/kafka-ui.yaml.

Top Results From Across the Web

Why Can't I Connect to Kafka? | Troubleshoot Connectivity

Shut down the Docker containers from above first ( docker rm -f broker; docker rm -f zookeeper ) and then create docker-compose.yml locally...

Cannot connect docker app to kafka docker container [duplicate]

I am trying to connect my Kafka container and my docker image all running locally and don't want to create a container with...

Trouble running Docker Connect with docker-compose #212

I think the issue is that you have the ADVERTISED_LISTENER set to localhost:9092 but you are telling connect the server is on kafka:9092...

Connect to Apache Kafka running in Docker - Baeldung

In this tutorial, we will learn how to configure the listeners so that clients can connect to a Kafka broker running within Docker....

wurstmeister/kafka - Docker Image - Docker Hub

Older compose files using the short-version of port mapping may encounter Kafka client issues if their connection to individual brokers cannot be guaranteed ......