Hadoop配置对象未指向hdfs文件系统

Question

Hi I am trying to create small Spark program in Java. I am creating the hadoop configuration object as show below:

Configuration conf = new Configuration(false);
conf.addResource(new Path("/dir/core-site.xml"));
conf.addResource(new Path("/dir/hdfs-site.xml"));
conf.addResource(new Path("/dir/yarn-site.xml"));

When I call the following I get file system but it points to local file system and not the hdfs file system

FileSystem fs = FileSystem.get(conf);

Please guide I am new to Hadoop. Thanks in advance.

Answer 1

I usually keep the core-site.xml (along with the other files) in the classpath and get the configuration as follows:

Configuration conf = new Configuration();
FileSystem localfs = FileSystem.getLocal(conf);
FileSystem hdfs = FileSystem.get(conf);

Here is the content of my core-site.xml :

<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://{hadoop-server-addr}</value>
  </property>
</configuration>

Answer 2

Add your hdfs-site.xml and core-site.xml to class path. And use following code.

Configuration conf = new Configuration();
FileSystem hdfs = FileSystem.newInstance(conf);

It will pick the configs mentioned in the xmls.

Answer 3

To debug the issue better, add this line after creating a new Configuration instance. This will give you a better idea.

Configuration conf = new Configuration(false);
conf.setQuietMode(false);

In most cases, defaulting to local filesystem instead of hdfs occurs when Configuration is not able to load resources specified by the Path strings. In such cases, the default behaviour of Configuration is to fail quietly and return null for requested key. In this case, the requested key is fs.defaultFS , the namenode host. FileSystem instance then defaults to the file:/// scheme which is the local file system.