将JSON文件从本地复制到HDFS

Question

import java.io.BufferedInputStream;
import java.io.FileInputStream;
import java.io.InputStream;
import java.io.OutputStream;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;

public class HdfsWriter extends Configured implements Tool {
 public int run(String[] args) throws Exception {
  //String localInputPath = args[0];
  Path outputPath = new Path(args[0]); // ARGUMENT FOR OUTPUT_LOCATION
  Configuration conf = getConf();
  FileSystem fs = FileSystem.get(conf);
  OutputStream os = fs.create(outputPath);
  InputStream is = new BufferedInputStream(new FileInputStream("/home/acadgild/acadgild.txt")); //Data set is getting copied into input stream through buffer mechanism.
  IOUtils.copyBytes(is, os, conf); // Copying the dataset from input stream to output stream
  return 0;
 }

 public static void main(String[] args) throws Exception {
  int returnCode = ToolRunner.run(new HdfsWriter(), args);
  System.exit(returnCode);
 }
}

Need to Move the data from Local to HDFS.

The above code I got from another blog , it's not working. can anyone help me on this.

Also i need to parse the Json using MR and group by DateTime and move to HDFS

Answer 1

1. Map Reduce is a distributed job processing framework
2. for each mapper local means the local filesytem on the node on which that mapper is running.
3. What you want is reading from local on a given node to be put on to HDFS and then processing it via MapReduce.

There are multiple tools available for copying from Local of one node to HDFS

1. hdfs put localPath HdfsPath (Shell script)
2. flume