将文件从s3复制并提取到HDFS

Question

I want to copy test.tar.gz file from S3 to HDFS. This can be done by distcp or s3distcp. But my requirement is when I transferring files to HDFS it should be extracted on the fly and in HDFS I should have only the extracted files not tar.gz.

Any suggestions please.

Answer 1

When you transfer by network, it's usually best that the files remain compressed. Imagine transferring a 100GB over instead of transferring a 20GB bz2 compressed file. I would suggest you to use a Hadoop API based code or a MapReduce program to extract your compressed files once the transfer is done to HDFS. Once in HDFS, you have all the power to extract the files without copying them over to the local file system.

1. One solution would be to use a simple Hadoop API based code or a MapReduce code (updated) that decompresses in parallel.

   Addendum: For ZIP you can follow this link . And, you can come up with something similar for tar.gz.

2. In case you file size is huge 100GB.zip, you can probably use a Hadoop API based program which reads a stream of the Zip archive, extracts( check this link for how it was done in the ZipFileRecordReader in addendum above) and then write it back to HDFS. I think, a single ZIP file is not splittable and extractable in parallel (If I'm not mistaken). So, if you have a single zip archive of 100GB, you'll probably not be able to anyway unleash the full potential of a MapReduce program. Hence, not point using it.

3. Other solution is to not decompress at all. For various built-in compressed formats, Hadoop has a command line utility that helps you view the compressed files as is if that is your intention to keep it uncompressed in HDFS.

   hadoop fs -text /path/fileinHDFS.bz2"

Answer 2

What's the problem of using a bash script? I mean:

s3distcp --src [file-location] --dst . #Without the hdfs prefix
tar -zxvf test.tar.gz
hadoop fs -mkdir /input
hadoop fs -mkdir /input/test
hadoop fs -copyFromLocal test/ /input/test

Answer 3

You should be able to achieve this with some clever piping...

Something like this (totally untested):

s3cmd get [s3 path] - | tar -zxfO | hadoop dfs -put - [hadoop path]

s3cmd get [s3 path] - gets the file from S3 and pipes it to stdout ( - ). tar -zxfO gets the piped file content from stdin and extracts it to stdout (the -O option). hadoop dfs -put - [hadoop path] puts the piped in data coming from stdin ( - ) in the provided HDFS file.