如何使用Oozie将远程文件复制到HDFS？

Question

I have to copy remote files into HDFS. I want to use Oozie because I need to run this job everyday at a specific time.

Answer 1

Oozie can help you create a workflow. Using oozie you can invoke an external action capable of copying files from your source to HDFS, but oozie will not do it automatically.

Here are a few suggestions:

1. Use a custom program to write files to hdfs, for example using a SequenceFile.Writer.
2. Flume might help.
3. Use an integration component like camel-hdfs to move files to hdfs.
4. ftp files to hdfs node and then copy from local disk to hdfs.

Investigate more options that might be a good fit for your case.