GAE mapreduce：为作业定义参数

Question

I am fiddling around with GAE mapreduce and have one question:

Is it possible to change a variable only for a certain job in mapreduce?

The reason I am asking is:

The input csv and output csv of my mapreduce job are supposed to have the same header row - however, the header row is somewhere in the output csv, but never at the top. To get the right header row, I inserted a counter into my reduce function that checks the current iteration of the reduce job and if it is 0, it will pass the hard-coded header-row to the pipeline. The counter gets reset when the output csv gets stored in the blobstore.

The problem: More often than not the counter resets itself randomly, probably because I had to define it as global variable "reduce_counter = 0" outside of the function.

Is there any method to chain a variable/parameter to a job or is there any better way to get the header_row?

I don't think that I can work with the DictReader or csv module as the output is stored in the blobstore and blobstore objects cannot be altered as far as I know.

You can find my code on www.github.com/jvdheyden/ste in the main.py document.

Thanks!

Answer 1

You should add the header after the mapreduce job has finished. You can do this by accessing the output CSV after the task is finished:

orig_file = gcs.open(filename_from_mapreduce)
new_file = gcs.open(filename_from_mapreduce + "_with_headers", "r")

new_file.write("your,csv,headers,here")
while True:
   read = orig_file.read()
   if read == '':
      break
   output.write(read)

output.close()
gcs.delete(orig_file)

Your problem happens because the GAE processes mapreduce tasks in multiple shards. The beauty is that each of those small tasks execute in parallel, which gives us huge time advantage on big amounts of data.

This also explains why your CSV header came in random places, while each shard just writes it's output whenever it gets it's job done. Therefore you cannot reliably predict the one that gets to write the first line of the output.