How to copy files from HDFS to S3 effectively programatically

Question

My hadoop job generate large number of files on HDFS and I want to write a separate thread which will copy these files from HDFS to S3.

Could any one point me to any java API that handles it.

Thanks

Answer 1

"Support for the S3 block filesystem was added to the ${HADOOP_HOME}/bin/hadoop distcp tool in Hadoop 0.11.0 (See HADOOP-862). The distcp tool sets up a MapReduce job to run the copy. Using distcp, a cluster of many members can copy lots of data quickly. The number of map tasks is calculated by counting the number of files in the source: ie each map task is responsible for the copying one file. Source and target may refer to disparate filesystem types. For example, source might refer to the local filesystem or hdfs with S3 as the target. "

Check out Running Bulk Copies in and out of S3 here http://wiki.apache.org/hadoop/AmazonS3