Jupyterlab 容器無法連接到端口 9092 上的 kafka localhost 服務器

Question

我有 docker-compose 包含一個 kafka 服務器、一個 zookeeper、spark 和一個 jupyter-lab。 這是我的 jupyter-lab dockerfile 的代碼

ARG debian_buster_image_tag=8-jre-slim
FROM openjdk:${debian_buster_image_tag}
RUN mkdir -p ${shared_workspace} && \
    apt-get update -y && \
    apt-get install -y python3 && \
    ln -s /usr/bin/python3 /usr/bin/python && \
    rm -rf /var/lib/apt/lists/* && \
    apt-get install -y python3-pip && \
    pip3 install pyspark==3.0.1 jupyterlab==2.1.5

這是我的 docker-compose.yml 代碼

version: "3.6"
volumes:
  shared-workspace:
    name: "hadoop-distributed-file-system"
    driver: local
services:
  jupyterlab:
    image: jupyterlab
    container_name: jupyterlab
    ports:
      - 8888:8888
    volumes:
      - shared-workspace:/opt/workspace

  spark-master:
    image: spark-master
    container_name: spark-master
    ports:
      - 8080:8080
      - 7077:7077
    volumes:
      - shared-workspace:/opt/workspace
  spark-worker-1:
    image: spark-worker
    container_name: spark-worker-1
    environment:
      - SPARK_WORKER_CORES=1
      - SPARK_WORKER_MEMORY=512m
    ports:
      - 8081:8081
    volumes:
      - shared-workspace:/opt/workspace
    depends_on:
      - spark-master
  spark-worker-2:
    image: spark-worker
    container_name: spark-worker-2
    environment:
      - SPARK_WORKER_CORES=1
      - SPARK_WORKER_MEMORY=512m
    ports:
      - 8082:8081
    volumes:
      - shared-workspace:/opt/workspace
    depends_on:
      - spark-master

  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    expose:
    - "2181"

  kafka:
    image: wurstmeister/kafka:latest
    depends_on:
    - zookeeper
    ports:
    - "9092:9092"
    expose:
    - "9093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE

在 jupyter 實驗室中，我無法通過 sparkSession.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9093") 訪問 kafka:9093, localhost:9092...，它顯示了該錯誤：

20/12/03 16:10:23 WARN NetworkClient: [Consumer clientId=consumer-spark-kafka-source-4b4422bc-2a6a-456a-b750-4fba8300328c-37706672-driver-0-1, groupId=spark-kafka-source-4b4422bc-2a6a-456a-b750-4fba8300328c-37706672-driver-0] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected

我還通過 jupyterlab 的 bash 安裝了 telnet 來檢查連接，我可以 telnet 到一些服務器，如telnet www.cyberciti.biz 80 ，但無法連接到 localhost:9092。 請幫我。

Answer 1

根據@onecricketeer，我認為我之前的回答是紅鯡魚。

所以，我運行了你的Dockerfile和docker-compose.yaml ，它對我有用。

ARG debian_buster_image_tag=8-jre-slim
FROM openjdk:${debian_buster_image_tag}

RUN apt-get update -y && \
    apt-get install -y python3 python3-pip && \
    rm -rf /var/lib/apt/lists/* && \
    pip3 install pyspark==3.0.1 jupyterlab==2.1.5

ENTRYPOINT [ "jupyter","notebook","--allow-root","--ip=0.0.0.0"]

筆記

• 刪除${shared_workspace} ，因為它似乎已安裝到/opt/workspace
• 重新排序了apt-get install s
• 添加ENTRYPOINT
• 運行jupyter-notebook --allow-root -ip=0.0.0.0

這會在localhost:8888上公開筆記本服務，但需要令牌。

調整了撰寫：

services:
  jupyterlab:
    image: jupyterlab
    build:
      context: .
      dockerfile: ./Dockerfile
    container_name: jupyterlab
    ports:
      - 8888:8888
    volumes:
      - shared-workspace:/opt/workspace

然后：

docker-compose build && docker-compose up

和docker-compose ps ：

        Name                      Command               State                            Ports                          
------------------------------------------------------------------------------------------------------------------------
65130144_kafka_1       start-kafka.sh                   Up      0.0.0.0:9092->9092/tcp, 9093/tcp                        
65130144_zookeeper_1   /bin/sh -c /usr/sbin/sshd  ...   Up      2181/tcp, 22/tcp, 2888/tcp, 3888/tcp                    
jupyterlab             jupyter notebook --allow-r ...   Up      0.0.0.0:8888->8888/tcp                                  
spark-master           /bin/bash /master.sh             Up      6066/tcp, 0.0.0.0:7077->7077/tcp, 0.0.0.0:8080->8080/tcp
spark-worker-1         /bin/bash /worker.sh             Up      0.0.0.0:8081->8081/tcp                                  
spark-worker-2         /bin/bash /worker.sh             Up      0.0.0.0:8082->8081/tcp          

您可能需要筆記本的令牌，使用docker-compose logs jupyterlab ：

Attaching to jupyterlab
jupyterlab        | [I 17:57:08.412 NotebookApp] Writing notebook server cookie secret to /root/.local/share/jupyter/runtime/notebook_cookie_secret
jupyterlab        | [I 17:57:08.659 NotebookApp] JupyterLab extension loaded from /usr/local/lib/python3.7/dist-packages/jupyterlab
jupyterlab        | [I 17:57:08.659 NotebookApp] JupyterLab application directory is /usr/local/share/jupyter/lab
jupyterlab        | [I 17:57:08.662 NotebookApp] Serving notebooks from local directory: /
jupyterlab        | [I 17:57:08.662 NotebookApp] Jupyter Notebook 6.1.5 is running at:
jupyterlab        | [I 17:57:08.662 NotebookApp] http://1b77e2b01d7b:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
jupyterlab        | [I 17:57:08.662 NotebookApp]  or http://127.0.0.1:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
jupyterlab        | [I 17:57:08.662 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
jupyterlab        | [W 17:57:08.665 NotebookApp] No web browser found: could not locate runnable browser.
jupyterlab        | [C 17:57:08.666 NotebookApp] 
jupyterlab        |     
jupyterlab        |     To access the notebook, open this file in a browser:
jupyterlab        |         file:///root/.local/share/jupyter/runtime/nbserver-1-open.html
jupyterlab        |     Or copy and paste one of these URLs:
jupyterlab        |         http://1b77e2b01d7b:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457
jupyterlab        |      or http://127.0.0.1:8888/?token=e1796ee09331bd31459211ce35ad16fc9a7f4ee854870457

使用第二個鏈接（您需要使用生成的任何令牌），創建一個筆記本並添加：

from pyspark.sql import SparkSession;


spark = SparkSession \
    .builder \
    .appName("test") \
    .getOrCreate()

spark.readStream.format("kafka").option("kafka.bootstrap.servers", "kafka:9093")

並運行它：

<pyspark.sql.streaming.DataStreamReader at 0x7fbf8defe208>

這似乎（？。）好的。

我懷疑 jupyter-lab和 jupyterlab在您的配置中是相同的。

Dockerfile 在啟動時不運行（沒有CMD或ENTRYPOINT ）任何東西，所以我還假設您在其中使用 Z2591C98B70119FE624898B1E424B5E19。 這很好，只是不尋常。

kafka服務的幾件事（注意我只是粗略地熟悉卡夫卡）：

• 在 kafka服務中，您必須將其引用為 localhost
• 你有 INSIDE://kafka:9093但這有兩個問題。 在服務內部，它應該將自己引用為 localhost並且在服務內部我懷疑它位於端口 :8082
• 聲明 expose: - "9093"不是功能性的，而是記錄性的，除非您的圖像也默默地暴露 :9093 ，否則它是不正確的； 這實際上並沒有暴露任何東西，因此它可能是不正確的。