Hadoop客户端不断从本地文件系统读取

Question

I'm trying to read a file via the Hadoop client from a Java program on windows. I have a valid kerberos keytab, but can't seem to read from HDFS.

I'm using the example from this page and the hadoop client keeps reading from the local file system. I triple checked the URL I'm using & its correct, what am I missing?

Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://test.hdp:8020");
conf.set("hadoop.security.authentication", "kerberos");

UserGroupInformation.setConfiguration(conf);
UserGroupInformation.loginUserFromKeytab("hdfs-user@MYCORP.NET", 
   "c:/temp/hdfs-user.keytab");

FileSystem fs = FileSystem.get(conf);
FileStatus[] fsStatus = fs.listStatus(new Path("/"));
for(int i = 0; i < fsStatus.length; i++){
  System.out.println(fsStatus[i].getPath().toString());
}

Answer 1

I found the issue. The project where the Hadoop Client was reading from the local FS was also trying to integrate Hadoop with Apache & Active MQ. The version of activemq-camel I was using was bring an older Hadoop library that was causing the issue. I updated to the latest versions of Camel & Active MQ & the issue went away. I verified that the newer versions do not bring in any hadoop depedencies.

activemq-camel 5.6 Dependencies ( partial list)
[INFO] \- org.apache.activemq:activemq-camel:jar:5.6.0:compile
[INFO]    +- org.apache.camel:camel-jms:jar:2.9.2:compile
[INFO]    \- org.apache.activemq:activemq-core:jar:5.6.0:compile
[INFO]       +- org.fusesource.fuse-extra:fusemq-leveldb:jar:1.1:compile
[INFO]       |  +- org.apache.hadoop:hadoop-core:jar:1.0.0:compile