Access KSQL Server from outside of a Docker Container using a different machine (local network)

Question

I have created the following Docker containers to run zookeeper , kafka , ksql , and ksql-cli as well. When I am running the command docker-compose exec ksqldb-cli ksql http://ksqldb-server:8088 from the same machine where Docker is running, ksql-cli can access the ksql-server just fine.

However, I want to have access to the ksql-server outside of the same machine but using a different laptop under the same local network. How do I do that?

Here's the relevant docker-compose.yml file:

version: '3.8'

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    container_name: zookeeper
    networks:
      - kafka_network
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    ports:
      - 22181:2181
  kafka:
    image: confluentinc/cp-kafka:latest
    container_name: kafka
    networks:
      - kafka_network
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
      - 29093:29093
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: EXTERNAL_SAME_HOST://:29092,EXTERNAL_DIFFERENT_HOST://:29093,INTERNAL://:9092
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,EXTERNAL_SAME_HOST://localhost:29092,EXTERNAL_DIFFERENT_HOST://192.168.178.218:29093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL_SAME_HOST:PLAINTEXT,EXTERNAL_DIFFERENT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  ksqldb-server:
    image: confluentinc/cp-ksqldb-server:latest
    container_name: ksqldb-server
    hostname: ksqldb-server
    networks:
      - kafka_network
    depends_on:
      - kafka
    ports:
      - "8088:8088"
    environment:
      KSQL_CONFIG_DIR: "/etc/ksql"
      KSQL_BOOTSTRAP_SERVERS: "kafka:9092"
      KSQL_HOST_NAME: ksqldb-server
      KSQL_LISTENERS: "http://0.0.0.0:8088"
      KSQL_CACHE_MAX_BYTES_BUFFERING: 0
      KSQL_KSQL_SCHEMA_REGISTRY_URL: "http://schema-registry:8081"
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_REPLICATION_FACTOR: 1
      KSQL_KSQL_LOGGING_PROCESSING_TOPIC_AUTO_CREATE: 'true'
      KSQL_KSQL_LOGGING_PROCESSING_STREAM_AUTO_CREATE: 'true'

  ksqldb-cli:
    image: confluentinc/cp-ksqldb-cli:latest
    container_name: ksqldb-cli
    networks:
      - kafka_network
    depends_on:
      - kafka
      - ksqldb-server
    entrypoint: /bin/sh
    tty: true
networks:
  kafka_network:
    name: kafka_docker_sse

When I try accessing the ksql-server from a different laptop that is under the same local network, I get a connection error/connection refused. I tried accessing the ksqldb-server using the Python ksql-python package.

pip install ksql

from ksql import KSQLAPI
client = KSQLAPI('http://ksql-server:8088')

# OR
# client = KSQLAPI('http://0.0.0.0:8088')
# client = KSQLAPI('http://192.168.178.218:8088')

if __name__ == '__main__':
    print(client)

I also tried changing the KSQL_LISTENERS: "http://0.0.0.0:8088" under the ksqldb-server to KSQL_LISTENERS: "http://192.168.178.218:8088" but that doesn't work either.

Any hints would be really helpful as I am currently stuck here for the last two days!

Answer 1

You'll need to keep KSQL_LISTENERS: "http://0.0.0.0:8088" . This binds the container to accept all incoming traffic into port 8088

Then, with ports in Compose, the host 8088 traffic is forwarded to the container 8088.

So, for any external client, you need to connect to that hosts LAN / external IP on port 8088. You may need to explicitly allow TCP traffic into that server host's firewall for that port.