简体   繁体   English

完成33%后,猪图减少任务失败

[英]pig map-reduce job fails after completing 33%

I am running a group by clause in apache pig and it is creating map reduce job,which is failing after 1/3 completion. 我在apache pig中运行group by子句,它正在创建map reduce作业,该作业在1/3完成后失败。 Is there anyway I can troubleshoot this as logs doesn't give any reason of failure. 无论如何,我可以解决此问题,因为日志没有给出任何失败原因。 What i am looking either of following. 我正在寻找以下之一。
1. Some way to find what the exact error is (ie memory error,datatype error etc) 1.以某种方式找到确切的错误是什么(即内存错误,数据类型错误等)
2. Any way to make the logs more verbose to write more error message on the screen. 2.任何使日志更加详细的方法都可以在屏幕上写入更多错误消息。

2016-04-03 22:59:40,252 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_1459707650021_0003
2016-04-03 22:59:40,252 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases a,b
2016-04-03 22:59:40,252 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: a[1,4],b[2,4] C:  R:
2016-04-03 22:59:40,258 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2016-04-03 22:59:40,258 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1459707650021_0003]
2016-04-03 23:00:10,428 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 33% complete
2016-04-03 23:00:10,428 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_1459707650021_0003]
2016-04-03 23:00:15,435 [main] WARN  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2016-04-03 23:00:15,435 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_1459707650021_0003 has failed! Stop running all dependent jobs
2016-04-03 23:00:15,435 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-04-03 23:00:15,508 [main] INFO  org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl - Timeline service address: http://compute.internal:8188/ws/v1/timeline/
2016-04-03 23:00:15,511 [main] INFO  org.apache.hadoop.yarn.client.RMProxy - Connecting to ResourceManager at compute.internal/8050
2016-04-03 23:00:15,515 [main] INFO  org.apache.hadoop.mapred.ClientServiceDelegate - Application state is completed. FinalApplicationStatus=FAILED. Redirecting to job history server

There is JobId associated with each job u submit to the map reduce framework. 您要提交给map reduce框架的每个作业都有一个JobId。 Copy that JobId ,paste and search it in your job tracker. 复制该JobId,粘贴并在作业跟踪器中搜索它。 For example ,in CDH3 by cluster cloudera in browser they have set up job tracker where u can see all jobs their details. 例如,在集群cloudera的CDH3中,他们在浏览器中设置了作业跟踪器,您可以在其中查看所有作业的详细信息。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM