[英]Access to datanode running on macOS from a docker container can't be established
I'm not experienced with HDFS and I've run into a problem related to HDFS running on my macbook.我对 HDFS 没有经验,我遇到了与在我的 macbook 上运行的 HDFS 相关的问题。 I have a HDFS client which is launched in a docker container, and every time I try to put or get data to/from HDFS from this container I get the following error:我有一个 HDFS 客户端,它在 docker 容器中启动,每次我尝试从该容器向 HDFS 放入或获取数据时,我都会收到以下错误:
hdfs dfs -put /core-site.xml hdfs://host.docker.internal:9000/abcs
21/03/02 07:28:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/03/02 07:28:48 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
at org.apache.hadoop.hdfs.DFSOutputStream.createSocketForPipeline(DFSOutputStream.java:1610)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.createBlockOutputStream(DFSOutputStream.java:1408)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.nextBlockOutputStream(DFSOutputStream.java:1361)
at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:588)
21/03/02 07:28:48 INFO hdfs.DFSClient: Abandoning BP-1485605719-127.0.0.1-1614607405999:blk_1073741832_1008
21/03/02 07:28:48 INFO hdfs.DFSClient: Excluding datanode 127.0.0.1:9866
21/03/02 07:28:48 WARN hdfs.DFSClient: DataStreamer Exception
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /abcs/core-site.xml._COPYING_ could only be written to 0 of the 1 minReplication nodes. There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
It can be clearly seen that my client(container) receives the wrong IP address of the DataNode (127.0.0.1:9866), it should be something like 192.168.65.2:9866 ie host.docker.internal.可以清楚地看到我的客户端(容器)收到了错误的DataNode(127.0.0.1:9866)的IP地址,应该是192.168.65.2:9866,即host.Z05B6053C41A2130AFD6FC3B158BDA4E6。 or domain name of my laptop (ex. my-laptop)或我的笔记本电脑的域名(例如 my-laptop)
My core-site.xml: (of course my-laptop is binded to 127.0.0.1 in etc/hosts)我的 core-site.xml:(当然 my-laptop 绑定到 etc/hosts 中的 127.0.0.1)
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://my-laptop:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/Users/Ian_Rakhmatullin/localHadoopTmp</value>
</property>
</configuration>
hdfs-site.xml: hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>my-laptop:9866</value>
</property>
<property>
<name>dfs.datanode.http.address</name>
<value>my-laptop:9864</value>
</property>
<property>
<name>dfs.datanode.ipc.address</name>
<value>my-laptop:9867</value>
</property>
</configuration>
One more thing that confuses me is that through HDFS webUI I can see that DataNode is running on localhost:9866 (127.0.0.1:9866)
, but I expect "my-laptop:9866" as well.让我感到困惑的另一件事是,通过 HDFS webUI,我可以看到 DataNode 正在localhost:9866 (127.0.0.1:9866)
上运行,但我也期望“my-laptop:9866”。
Does anyone have any thoughts how to resolve this issue?有没有人有任何想法如何解决这个问题? Thank you.谢谢你。
Seems like I've solved this problem, by following these steps:好像我已经通过以下步骤解决了这个问题:
hdfs-site xml: hdfs 站点 xml:
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
<property>
<name>dfs.permissions.enabled</name>
<value>false</value>
</property>
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.hostname</name>
<value>my-laptop</value>
</property>
core-site xml the same as it is in my question.核心站点 xml 与我的问题相同。
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>true</value>
</property>
192.168.65.2 my-laptop
With this approach Namenode will return host name for your Datanode to the hdfs client, and then, the client will use your mapping to host.docker.internal.通过这种方法,Namenode 会将您的 Datanode 的主机名返回给 hdfs 客户端,然后,客户端将使用您的映射到 host.docker.internal。 And this is what I needed.这就是我所需要的。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.