Docker using flink socketwordcount example [apache-flink]

Question

I want to use Flink with the help of the docker. I want to run the Socketwordcount instance as a startup. But at this stage I encounter an error. I cannot connect with socket. so after 20 ms the flink finishes working.

Docker-compose:

version: "3"
services:
  jobmanager:
    image: ${FLINK_DOCKER_IMAGE_NAME:-flink:1.3.2-hadoop24-scala_2.11-alpine}
    expose:
      - "6123"
    ports:
      - "8081:8081"
    command: jobmanager
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

  taskmanager:
    image: ${FLINK_DOCKER_IMAGE_NAME:-flink:1.3.2-hadoop24-scala_2.11-alpine}
    expose:
      - "6121"
      - "6122"
      - "9000"
    depends_on:
      - jobmanager
    command: taskmanager
    links:
      - "jobmanager:jobmanager"
    environment:
      - JOB_MANAGER_RPC_ADDRESS=jobmanager

After I do docker-compose up, I enter into the running jobmanager.

nc -l 6123

I'm running.Then I enter the jobmanager container with the help of another terminal

./bin/flink run examples/streaming/SocketWindowWordCount.jar -port 6123

I'm following instructions in the Flink documentation; What I want to achieve is to count words sent with nc and write results to .out file. But here the flink closes after 20 ms. When I look at the logs with Docker-compose logs, I see that it is closed because there was no connection established. I'm sorry for my bad english. I do not know what is the problem. If you want to know more about the subject, I can publish the order of my work visually.

Answer 1

Here's how I got this working:

First I ran

nc -l 9999

in a terminal on my laptop (not in a container).

And then I ran

./bin/flink run examples/streaming/SocketWindowWordCount.jar --hostname 192.168.1.109 --port 9999

in the jobmanager container, where 192.168.1.109 is my laptop's IP address.

Finally, I used

docker logs stuff_taskmanager_1

to see the job's output (where stuff_taskmanager_1 is the name of the container running the taskmanager, determined by using docker ps ).

Answer 2

The thing is that the actual work happens on taskmanager. Jobmanager only coordinates the work. That said the connection is tried to be establish on the taskmanager node. You might try to run

nc -l 6123

on the taskmanager.

As a tip, the example with Socket connection is not the best for distributed setup.