hadoop-streaming：作业完成后自动进行后处理吗？

Question

Step 1- I have a hadoop streaming job that takes variable time based on amount of data to process Step 2- Once the job is done, I need to import all that data dump into mongodb and create a flat csv file out of it

Question
Is there any way I can glue Step 2 to Step 1 using hadoop streaming and avoid doing Step 2 manually?

Answer 1

I would recommend using something like https://github.com/Yelp/mrjob or https://github.com/klbostee/dumbo . Specifically for mrjob and your problem http://packages.python.org/mrjob/job.html#writing-multi-step-jobs