如何提高使用mapreduce分析日志文件的性能

Question

We have to analyze the log files using hadoop as it can handle large data easily. So, I wrote one piece of mapreduce program. But even my mapreduce program is taking lot of time to get the data.

String keys[] = value.toString().split(" ");
        int keysLength = keys.length;
        if(keysLength > 4 && StringUtils.isNumeric(keys[keysLength-5])) {
            this.keyWords.set(keys[0]+"-"+keys[1]+" "+keys[2]+" "+keys[keysLength-5]+" "+keys[keysLength-2]);
            context.write(new IntWritable(1), keyWords);
        }

The requirement is, we will have mostly 10 to 15 of .gz files and every .gz file have one log file inside. we have to pull the data from that log file to analyze it.

Sample input in the log file :

2015-09-12 03:39:45.201 [service_client] [anhgv-63ac7ca63ac] [[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'] INFO TempServerImplementation - || Server: loclhost 121.1.0.0 | Service Category: Testing | Service Method: add | Application Id: Test | Status Code: 200 | Duration: 594ms ||

So could someone help me how can I tune up the performance.

Thanks Sai

Answer 1

You can try using SPARK(We can think this as in memory map reduce), it is 10x to 100x faster than traditional Map reduce. Please check trade-offs between hadoop map-reduce and SPARK before using.

Answer 2

There are two main ways you can speed up your job, input size and variable initialisation .

Input Size

gz is not a splittable format. That means that if you have 15 input gz files, you will only have 15 mappers. I can see from the comments that each gz file is 50MB, so at a generous 10:1 compression ratio, each mapper would be processing 500MB. This can take time, and unless you've got a <15 node cluster, you'll have nodes that are doing nothing. By uncompressing the data before the MR job you could have more mappers which would reduce the runtime.

Variable Initialisation

In the below line:

context.write(new IntWritable(1), keyWords);

you're generating a big overheard by allocating a brand new IntWritable for each output. Instead, why not allocate it at the top of the class? It doesn't change, so it doesn't need allocating each time.

For example:

private static final IntWritable ONE_WRITABLE = new IntWritable(1);
...
context.write(ONE_WRITABLE, keyWords);

The same applies to the strings you use - " " and "-" . Assign them as static variables also and again avoid creating fresh ones each time.