I am running a group by clause in apache pig and it is creating map reduce job,which is failing after 1/3 completion. Is there anyway I can troubleshoot this as logs doesn't give any reason of failure. What i am looking either of following.
1. Some way to find what the exact error is (ie memory error,datatype error etc)
2. Any way to make the logs more verbose to write more error message on the screen.
2016-04-03 22:59:40,252 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1459707650021_0003
2016-04-03 22:59:40,252 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases a,b
2016-04-03 22:59:40,252 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: a[1,4],b[2,4] C: R:
2016-04-03 22:59:40,258 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2016-04-03 22:59:40,258 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1459707650021_0003]
2016-04-03 23:00:10,428 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 33% complete
2016-04-03 23:00:10,428 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1459707650021_0003]
2016-04-03 23:00:15,435 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2016-04-03 23:00:15,435 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1459707650021_0003 has failed! Stop running all dependent jobs
2016-04-03 23:00:15,435 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-04-03 23:00:15,508 [main] INFO org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://compute.internal:8188/ws/v1/timeline/
2016-04-03 23:00:15,511 [main] INFO org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at compute.internal/8050
2016-04-03 23:00:15,515 [main] INFO org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=FAILED. Redirecting to job history server
There is JobId associated with each job u submit to the map reduce framework. Copy that JobId ,paste and search it in your job tracker. For example ,in CDH3 by cluster cloudera in browser they have set up job tracker where u can see all jobs their details.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.