如何划分map-reduce任务？

Question

I have a table containing 200 columns out of which I need around 50 column mentioned in a list, and rows of last 24 months according to column 'timestamp'.

I'm confused what comes under mapper and what under reducer?

As it is just transformation, will it only have mapper phase, or filtering of rows to last 24 months will come under reducer? I'm not sure if this exactly utilises what map-reduce was made for.

I'm using python with hadoop streaming.

Answer 1

So, your have a table with 200 columns(say T), a separate list of entries(say L) to be picked from T and with the last 24-hours(from the timestamp in T).

MapReduce, mapper does give entries from T sequentially. Before your mapper gets into map(), Ie in setup() put the block of code to read from the L and make it handy(use a feasible data structure to hold the list of data). Now, your code should hold two checks/conditions 1) if the entry from T contains/matches with L. If so, then check 2) if the data is within 24-hours range.

Done. Your output is what you have expected. No, reducer is required here, at least to do this much.

Happy Mapreducing.