[英]copying directory from local system to hdfs java code
I'm having a problem trying to copy a directory from my local system to HDFS using java code. 我尝试使用java代码将目录从本地系统复制到HDFS时遇到问题。 I'm able to move individual files but can't figure out a way to move an entire directory with sub-folders and files. 我能够移动单个文件,但无法找到一种方法来移动整个目录与子文件夹和文件。 Can anyone help me with that? 任何人都可以帮助我吗? Thanks in advance. 提前致谢。
Just use the FileSystem
's copyFromLocalFile method. 只需使用FileSystem
的copyFromLocalFile方法即可。 If the source Path is a local directory it will be copied to the HDFS destination: 如果源Path是本地目录,它将被复制到HDFS目标:
...
Configuration conf = new Configuration();
conf.addResource(new Path("/home/user/hadoop/conf/core-site.xml"));
conf.addResource(new Path("/home/user/hadoop/conf/hdfs-site.xml"));
FileSystem fs = FileSystem.get(conf);
fs.copyFromLocalFile(new Path("/home/user/directory/"),
new Path("/user/hadoop/dir"));
...
Here is the full working code to read and write in to HDFS. 以下是读取和写入HDFS的完整工作代码。 It takes two arguments 它需要两个参数
Input path ( local / HDFS ) 输入路径(本地/ HDFS)
Output path(HDFS) 输出路径(HDFS)
I used Cloudera sandbox. 我用过Cloudera沙箱。
package hdfsread;
import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.net.URI;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
public class ReadingAFileFromHDFS {
public static void main(String[] args) throws IOException {
String uri = args[0];
InputStream in = null;
Path pt = new Path(uri);
Configuration myConf = new Configuration();
Path outputPath = new Path(args[1]);
myConf.set("fs.defaultFS","hdfs://quickstart.cloudera:8020");
FileSystem fSystem = FileSystem.get(URI.create(uri),myConf);
OutputStream os = fSystem.create(outputPath);
try{
InputStream is = new BufferedInputStream(new FileInputStream(uri));
IOUtils.copyBytes(is, os, 4096, false);
}
catch(IOException e){
e.printStackTrace();
}
finally{
IOUtils.closeStream(in);
}
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.