使用指向HDFS的URI创建文件实例

Question

Is it possible to create a file instance by putting the uri of my HDFS as File class's constructor? For example:

val conf = new Configuration()
conf.addResource(hdfsCoreSitePath)
conf.addResource(hdfsHDFSSitePath)

val uri = conf.get("fs.default.name")
val file = new File(uri + pathtothefile)

Then, with the file instance, I wish to access the file list with the functions provided by File class such as file.list() to returns an array of strings naming the files and directories in the directory denoted by this abstract pathname. I tried the code but it return null on the file.list() .

The method below is not recommended as I am trying to writing the same codebase for normal file system and hdfs to achieve code reusable.

val fileSystem = FileSystem.get(conf)
val status = fileSystem.listStatus(new Path(filepath))
status.map(x => ...

Answer 1

fs.default.name is deprecated. Try to use fs.defaultFS and make sure this property is available in the core-site.xml file you are referring using the below command

conf.addResource(hdfsCoreSitePath)

https://hadoop.apache.org/docs/r2.7.1/hadoop-project-dist/hadoop-common/core-default.xml

Answer 2

The regular built-in Java/Scala File APIs will not work for HDFS files. The protocol and implementation are too different. You have to use the Hadoop API to access HDFS files as in your second example.

The good news, though, is that the Hadoop API will work for non-HDFS files (regular files). So that code is reusable. Just use a URI like: file:///foo/bar for a local file.