[英]In hadoop, how do I initialize the a DistributedFileSystem object via the initialize method?
有两个参数,一个URI和一个Configuration。 我假设客户端设置的JobConf对象应该可以用于Configuration,但是URI呢?
这是我为驱动程序编写的代码:
JobClient client = new JobClient();
JobConf conf = new JobConf(ClickViewSessions.class);
conf.setJobName("ClickViewSessions");
conf.setOutputKeyClass(LongWritable.class);
conf.setOutputValueClass(MinMaxWritable.class);
FileInputFormat.addInputPath(conf, new Path("input"));
FileOutputFormat.setOutputPath(conf, new Path("output"));
conf.setMapperClass(ClickViewSessionsMapper.class);
conf.setReducerClass(ClickViewSessionsReducer.class);
client.setConf(conf);
DistributedFileSystem dfs = new DistributedFileSystem();
try {
dfs.initialize(new URI("blah") /* what goes here??? */, conf);
} catch (Exception e) {
throw new RuntimeException(e.toString());
}
如何获取要提供给上面initialize
的调用的URI?
You could also use as shown below to intialize a file system
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
public static void main(String args[]){
try {
Configuration conf = new Configuration();
conf.set("fs.defaultFS", "hdfs://localhost:54310/user/hadoop/");
FileSystem fs = FileSystem.get(conf);
FileStatus[] status = fs.listStatus(new Path("."));
for(int i=0;i<status.length;i++){
System.out.println(status[i].getPath());
}
} catch (IOException e) {
e.printStackTrace();
}
}
URI是您正在运行的HDFS的位置。 文件系统名称的默认值应在conf / core-site.xml中。 “ fs.default.name”的值应该是您连接到的URI。
如果您还没有看过如何设置简单的单节点系统的教程,我强烈建议您:
http://hadoop.apache.org/common/docs/current/single_node_setup.html
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.