简体   繁体   中英

pig map-reduce job fails after completing 33%

I am running a group by clause in apache pig and it is creating map reduce job,which is failing after 1/3 completion. Is there anyway I can troubleshoot this as logs doesn't give any reason of failure. What i am looking either of following.
1. Some way to find what the exact error is (ie memory error,datatype error etc)
2. Any way to make the logs more verbose to write more error message on the screen.

2016-04-03 22:59:40,252 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1459707650021_0003
2016-04-03 22:59:40,252 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases a,b
2016-04-03 22:59:40,252 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: a[1,4],b[2,4] C:  R:
2016-04-03 22:59:40,258 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2016-04-03 22:59:40,258 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1459707650021_0003]
2016-04-03 23:00:10,428 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 33% complete
2016-04-03 23:00:10,428 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1459707650021_0003]
2016-04-03 23:00:15,435 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2016-04-03 23:00:15,435 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1459707650021_0003 has failed! Stop running all dependent jobs
2016-04-03 23:00:15,435 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-04-03 23:00:15,508 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://compute.internal:8188/ws/v1/timeline/
2016-04-03 23:00:15,511 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at compute.internal/8050
2016-04-03 23:00:15,515 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=FAILED. Redirecting to job history server

There is JobId associated with each job u submit to the map reduce framework. Copy that JobId ,paste and search it in your job tracker. For example ,in CDH3 by cluster cloudera in browser they have set up job tracker where u can see all jobs their details.

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM