如何使用Java代码将文件上传和下载到hdfs

Question

i am newbie to hadoop, and trying to upload and download files to hdfs via. Java code. which should behave as

Data Uploading:

 hadoop fs -put or -copyFromLocal filename directoryName

and Data Downloading

  hadoop fs -get or -copyToLocal filename directoryName

from hdfs. i need this one because datasets contain image, audio, video etc file. above command works fine with all type of data, if i try using Java i/o reader code , it is working fine for text files , but not for images, video. docx etc..

pls any help here.

Edited Here:

public static void main(String[] args) throws IOException {
        // TODO Auto-generated method stub

        Configuration conf=new Configuration();
        FileSystem fs = FileSystem.get(conf);

        Path path=new Path("data");
        Path file=new Path(path,"screenshots.png");

        BufferedImage image = ImageIO.read(new File("/home/hduser/Desktop/screenshots.png"));
        if (!fs.exists(path))
          throw new IOException("Output not found!");

        ImageIO.write(image, "png", fs.open(path));


    }

As asked i have edited here code that i am using to upload image file to hdfs. here ImageIO.write is not accepting arguement fs.open(path) , because is asking for file, but i have to give path here as to read and to write to hdfs we need to give path only. Actually i am in need of a method to upload and download file from hdfs using code for all type of data, so i should not write code and use plugins for all type of file.

Answer 1

ImageIO.write can take an OutputStream as well as a File. However, fs.open is returning an InputStream because it is for reading files only.

You need to call:

ImageIO.write(image, "png", fs.create(file));

The create method will return an OutputStream which ImageIO can write to.

http://hadoop.apache.org/docs/r2.2.0/api/org/apache/hadoop/fs/FileSystem.html

Answer 2

1. If path is already exists than you will overwrite that file with your image. I think that you want to save your image into some existing folder in HDFS . In that case you need to write your image to new Path(path, "SomeImageName.png"); .

2. You don't need to use ImageIO to copy the image from local file system to HDFS . Try to use copyFromLocalFile method of FileSystem :

   fs.copyFromLocalFile(new Path("/home/hduser/Desktop/screenshots.png"), path);