hadoop on macOS initiating secondary namenode fails due to ssh connection refused

Question

I've successfully gone through initiating single-node in a pseudo-distributed mode described in https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html#Pseudo-Distributed_Operation , under Window's wsl2 environment. After that, I tried to repeat it using MacBookPro. But somehow start-dfs.sh fails. Terminal throws error:

Stopping namenodes on [localhost]
Stopping datanodes
Stopping secondary namenodes [kakaoui-MacBookPro.local]
kakaoui-MacBookPro.local: ssh: connect to host kakaoui-macbookpro.local port 22: Connection refused
2021-06-26 23:01:23,377 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Okay. There are answers saying I should enable ssh connection via system property, but it is already set so and ssh localhost also works fine.

And then thing goes worth; Sometimes it is described that secondary namenode fails as:

Starting secondary namenodes [kakaoui-MacBookPro.local]
kakaoui-MacBookPro.local: ssh: connect to host kakaoui-macbookpro.local port 22: Operation timed out

Then when I leave Mac for a while and again command start-dfs.sh , once in a while it succeeds. And as I do stop-dfs.sh and start-dfs.sh to check, it fails.

Even if I could successfully start-dfs.sh , a lot of problems like not being able to start data node or resourcemanager or nodemanager etc comes after. I couldn't run hadoop environment even once.

Feels like everything is mixed up and things are not stable at all. Tried reinstalling this and that for several times already. Unfortunately most of initiation failure is not even recored in /logs folder.

Currently I'm using:
macOS: Catalina 10.15.6
java: 1.8.0_291
hadoop: 3.3.1

I've spent whole two day just trying. Please help!

Answer 1

Provide ssh-key less access to all your worker nodes in hosts file, even localhost as well as kakaoui-macbookpro.local . Read instruction in the Creating a SSH Public Key on OSX .

At last test access without password by ssh localhost and ssh [yourworkernode] (maybe ssh kakaoui-macbookpro.local ).

Answer 2

Okay, I found the solution that I don't understand. I turned off wifi connection during initiation process and all processes started up. Can't understand how wifi connection interferes ssh localhost though.