hdfs-获取文件夹错误

Question

I met an error when run hdfs dfs -get command like this:

[work@myserver ~]$ hdfs dfs -get hdfs://hadoopserver:8020/path/DataLoad/
Exception in thread "main" java.lang.NullPointerException
    at org.apache.hadoop.fs.FsShell.displayError(FsShell.java:304)
    at org.apache.hadoop.fs.FsShell.run(FsShell.java:289)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.fs.FsShell.main(FsShell.java:340)

The "hadoopserver" is the master of a hadoop cluster which is not in the same network with myserver. I connect it by config public network IP in "/etc/hosts".

What's more, this error only happen when getting folder and from out network. Look 2 examples below: The first one is I get the folder from another server which is in the same network with hadoopserver. And the second example is I get a file instead of a folder from the same server(myserver). Both them succeed.

[work@hadoopserver_2 lewis]$ hdfs dfs -get hdfs://hadoopserver:8020/path/DataLoad ./
[work@hadoopserver_2 lewis]$ du -sh DataLoad/
 1.2G   DataLoad/

[work@myserver ~]$ hdfs dfs -get hdfs://hadoopserver:8020/path/DataLoad/part-r-00375-724f4a2e-ed40-4100-8e81-6657d9dacc01.gz.parquet ./
[work@myserver ~]$ ls
part-r-00375-724f4a2e-ed40-4100-8e81-6657d9dacc01.gz.parquet

To add one last point, when I get folder from outer network and get the error, the folder will always be created and sometimes get incomplete files.

[work@myserver DataLoad]$ ls
part-r-00000-724f4a2e-ed40-4100-8e81-6657d9dacc01.gz.parquet  _SUCCESS
[work@myserver DataLoad]$ du -sh
10M .

(There should be many files in DataLoad and larger than 10M. And in most times, there is only a empty file named "_SUCCESS".)

Answer 1

my workmate has solved this problem. We use hdfs --loglevel DEBUG to show the detail error. The root cause is that I just config the master server ip in "/etc/hosts". When myserver tried to get data from a hadoop cluster from outer network. The master server(default 8020 port) tells me which datanode(default 50070 port) I should to connect to. So I can't get the whole folder because the file must in different datanodes. I successed to get the file maybe it just on the master node(which is the only server config in my hosts).

So after I config all node's public IP in hosts. I can get the folder succeessfully.