[英]Hive Error : FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
I have got twitter data using flume on HDFS.我在 HDFS 上使用水槽获得了 twitter 数据。 Have 3 node cluster and MySQL Metastore for hive.
有 3 节点集群和用于配置单元的 MySQL Metastore。
When i execute below query当我执行以下查询时
select user_name.screen_name, user_name.followers_count from BBMP_election_tweets limit 10;
It is giving proper results.它给出了正确的结果。
But if i run below query但是如果我在查询下面运行
select user_name.screen_name, user_name.followers_count c from BBMP_election_tweets order by c desc;
It throws below error它抛出以下错误
Query ID = hduser_20150827102628_15a405ae-59a4-4c90-aa47-045d58c62f02 Total jobs = 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer= In order to limit the maximum number of reducers: set hive.exec.reducers.max= In order to set a constant number of reducers: set mapreduce.job.reduces= Starting Job = job_1439458788229_0016, Tracking URL = http://hdmaster:8088/proxy/application_1439458788229_0016/ Kill Command = /usr/local/hadoop/bin/hadoop job -kill job_1439458788229_0016 Hadoop job information for Stage-1: number of mappers: 2;
查询 ID = hduser_20150827102628_15a405ae-59a4-4c90-aa47-045d58c62f02 Total jobs = 1 Launching Job 1 out of 1 编译时确定的reduce 任务数:1 为了更改reducer 的平均负载:hivein。 exec.reducers.bytes.per.reducer=为了限制reducer的最大数量:set hive.exec.reducers.max=为了设置恒定的reducer数量:set mapreduce.job.reduces=Starting Job = job_1439458788229_0016 , Tracking URL = http://hdmaster:8088/proxy/application_1439458788229_0016/ Kill Command = /usr/local/hadoop/bin/hadoop job -kill job_1439458788229_0016 Stage-1的Hadoop作业信息:mapper数量:2; number of reducers: 1 2015-08-27 10:27:33,567 Stage-1 map = 0%, reduce = 0% 2015-08-27 10:27:43,908 Stage-1 map = 50%, reduce = 0% 2015-08-27 10:27:45,015 Stage-1 map = 0%, reduce = 0% 2015-08-27 10:28:31,011 Stage-1 map = 100%, reduce = 100% Ended Job = job_1439458788229_0016 with errors Error during job, obtaining debugging information... Examining task ID: task_1439458788229_0016_m_000001 (and more) from job job_1439458788229_0016
减速器数量:1 2015-08-27 10:27:33,567 Stage-1 map = 0%,reduce = 0% 2015-08-27 10:27:43,908 Stage-1 map = 50%,reduce = 0% 2015 -08-27 10:27:45,015 Stage-1 map = 0%,reduce = 0% 2015-08-27 10:28:31,011 Stage-1 map = 100%,reduce = 100% Ended Job = job_143945878816 有错误在作业期间,获取调试信息...正在检查任务 ID:task_1439458788229_0016_m_000001(以及更多)来自作业 job_1439458788229_0016
Task with the most failures(4): ----- Task ID: task_1439458788229_0016_m_000001
失败次数最多的任务(4): ----- 任务ID:task_1439458788229_0016_m_000001
URL:
网址:
http://hdmaster:8088/taskdetails.jsp?jobid=job_1439458788229_0016&tipid=task_1439458788229_0016_m_000001 ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invokehttp://hdmaster:8088/taskdetails.jsp?jobid=job_1439458788229_0016&tipid=task_1439458788229_0016_m_000001 ----- 此任务的诊断消息: 错误:java.lang.RuntimeException at. .setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org. apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446) 在 org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) 在 org.apache.hadoop.mapred.YarnChild$2.run(YarnChild .java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java) :1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 由:java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke 0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at
0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java: 606) 在 org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 导致:java.lang.RuntimeException:在 org.apache.hadoop.util.ReflectionUtils 配置对象时出错。 setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache .hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 更多 14 引起: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 more Caused by: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec
java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more 引起:java.lang.RuntimeException: Map操作符初始化在 org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 处失败 引起:org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException:在 org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) 在 org.apache.hadoop.hive 中找不到类 com.cloudera.hive.serde.JSONSerDe。 ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 更多 引起:java.lang .ClassNotFoundException:在 org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer 的 org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) 中找不到类 com.cloudera.hive.serde.JSONSerDe (PartitionDesc.java:141) 在 org.apache.hadoop.hive.ql.exec .MapOperator.getConvertedOI(MapOperator.java:302) ... 24 more
.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 更多
FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Stage-Stage-1: Map: 2 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total MapReduce CPU Time Spent: 0 msec
失败:执行错误,从 org.apache.hadoop.hive.ql.exec.mr.MapRedTask MapReduce Jobs Launched: Stage-Stage-1: Map: 2 Reduce: 1 HDFS Read: 0 HDFS Write: 0 FAIL Total 返回代码 2 MapReduce CPU 花费时间:0 毫秒
When checked the logs created.检查时创建的日志。 It gives below message
它给出了以下消息
2015-08-27 10:28:48,852 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1439458788229_0016, Ident: (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@988145f) 2015-08-27 10:28:49,023 INFO [main] org.apache.hadoop.mapred.YarnChild: Sleeping for 0ms before retrying again.
2015-08-27 10:28:48,852 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind: mapreduce.job, Service: job_1439458788229_0016, Ident: (org.apache.hadoop.mapreduce.JobTokenIdentifier.token @988145f) 2015-08-27 10:28:49,023 INFO [main] org.apache.hadoop.mapred.YarnChild:在重试之前休眠 0 毫秒。 Got null now.
现在为空。 2015-08-27 10:28:49,627 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-hduser/nm-local-dir/usercache/hduser/appcache/application_1439458788229_0016 2015-08-27 10:28:50,363 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated.
2015-08-27 10:28:49,627 INFO [main] org.apache.hadoop.mapred.YarnChild: mapreduce.cluster.local.dir for child: /tmp/hadoop-hduser/nm-local-dir/usercache/hduser /appcache/application_1439458788229_0016 2015-08-27 10:28:50,363 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: session.id 已弃用。 Instead, use dfs.metrics.session-id 2015-08-27 10:28:51,186 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-08-27 10:28:51,534 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685:0+52974,/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484188899:0+20119InputFormatClass: org.apache.hadoop.mapred.TextInputFormat
相反,使用 dfs.metrics.session-id 2015-08-27 10:28:51,186 INFO [main] org.apache.hadoop.mapred.Task: Using ResourceCalculatorProcessTree : [ ] 2015-08-27 10:28:51,534 INFO [main] org.apache.hadoop.mapred.MapTask: Processing split: Paths:/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685:0+52974,/user/flume/tweets/ BBMP_elections/2015/08/25/11/FlumeData.1440484188899:0+20119InputFormatClass: org.apache.hadoop.mapred.TextInputFormat
2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: PLAN PATH = hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: ***************non-local mode*************** 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: local path = hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Open file to read in plan: hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,719 INFO [main] org.apache.hado
2015-08-27 10:28:51,701 信息 [主要] org.apache.hadoop.hive.ql.exec.Utilities: 计划路径 = hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b [-b3a5-e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4787953-4714-map0107/7070700700000000000000000 .apache.hadoop.hive.ql.exec.Utilities: ***************非本地模式*************** 2015-08 -27 10:28:51,701 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: 本地路径 = hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5- [e326134f9f24/hive_2015-08-27_10-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-91f170-26-28_312_5076847637673296334-1/-mr-10004/6e310852-ea63-4789-93f170-26-28_312_50768476376740.xml-apache-470707040000000xml: hadoop.hive.ql.exec.Utilities:打开文件以读入计划:hdfs://hdmaster:8020/tmp/hive/hduser/55daf291-b44e-414b-b3a5-e326134f9f24/hive_2015-08-268_106373737361 -1/-mr-10004/6e310852-ea63-4789-93f1-754147923e7f/map.xml 2015-08-27 10:28:51,719 INFO [main] org.apache.hado op.hive.ql.log.PerfLogger: 2015-08-27 10:28:51,720 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities: Deserializing MapWork via kryo 2015-08-27 10:28:52,043 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: Processing file hdfs://hdmaster:8020/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.file is deprecated.
op.hive.ql.log.PerfLogger:2015-08-27 10:28:51,720 INFO [main] org.apache.hadoop.hive.ql.exec.Utilities:通过 kryo 反序列化 MapWork 2015-08-27 10:28 :52,043 INFO [main] org.apache.hadoop.hive.ql.log.PerfLogger: 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader: 处理文件hdfs://hdmaster:8020/user/flume/tweets/BBMP_elections/2015/08/25/11/FlumeData.1440484157685 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop. .deprecation:不推荐使用 map.input.file。 Instead, use mapreduce.map.input.file 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.start is deprecated.
相反,使用 mapreduce.map.input.file 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.start 已弃用。 Instead, use mapreduce.map.input.start 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.length is deprecated.
相反,使用 mapreduce.map.input.start 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.conf.Configuration.deprecation: map.input.length 已弃用。 Instead, use mapreduce.map.input.length 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 1 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io.sort.mb: 100 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: soft limit at 83886080 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: bufstart = 0;
相反,使用 mapreduce.map.input.length 2015-08-27 10:28:52,080 INFO [main] org.apache.hadoop.mapred.MapTask: numReduceTasks: 1 2015-08-27 10:28:52,228 INFO [main] ] org.apache.hadoop.mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: mapreduce.task.io. sort.mb: 100 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: 软限制在 83886080 2015-08-27 10:28:52,228 INFO [main] org.apache .hadoop.mapred.MapTask: bufstart = 0; bufvoid = 104857600 2015-08-27 10:28:52,228 INFO [main] org.apache.hadoop.mapred.MapTask: kvstart = 26214396;
bufvoid = 104857600 2015-08-27 10:28:52,228 信息 [主要] org.apache.hadoop.mapred.MapTask: kvstart = 26214396; length = 6553600
长度 = 6553600
2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initializing operator MAP[3] 2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Initialization Done 3 MAP 2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator: Operator 3 MAP initialized 2015-08-27 10:28:52,262 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:446) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Sub
2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator:初始化操作符 MAP[3] 2015-08-27 10:28:52,243 INFO [main] org. apache.hadoop.hive.ql.exec.MapOperator:初始化完成 3 MAP 2015-08-27 10:28:52,243 INFO [main] org.apache.hadoop.hive.ql.exec.MapOperator:Operator 3 MAP 初始化 2015- 08-27 10:28:52,262 WARN [main] org.apache.hadoop.mapred.YarnChild: Exception running child : java.lang.RuntimeException: 在 org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils) 配置对象时出错.java:109) 在 org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) 在 org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) 在 org.apache.hadoop。 mapred.MapTask.runOldMapper(MapTask.java:446) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) ) 在 java.security.AccessController.doPrivileged(Native Method) 在 javax.security.auth.Subject.doAs(Sub ject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more Caused by: java.lang.RuntimeException: Error in configuring object at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 more Caused by: java.lang.ref
ject.java:415) 在 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) 在 org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 引起:java.lang .reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java.lang:43) .reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 9 more 引起:java.lang.RuntimeException:配置对象时出错在 org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) 在 org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) 在 org.apache.hadoop.util.ReflectionUtils.newInstance (ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38) ... 14 更多 引起:java.lang.ref lect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more Caused by: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 more Caused by: java.lang.ClassNot
lect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java. reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) ... 17 more org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:140) ... 22 更多 引起:org.apache.hadoop.hive.ql.metadata.HiveException:java.lang。 ClassNotFoundException:在 org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:328) 处找不到类 com.cloudera.hive.serde.JSONSerDe。 MapOperator.setChildren(MapOperator.java:338) at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:109) ... 22 more 引起:java.lang.ClassNot FoundException: Class com.cloudera.hive.serde.JSONSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1980) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:141) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 more
FoundException: 在 org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer( PartitionDesc.java:141) 在 org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:302) ... 24 更多
2015-08-27 10:28:52,269 INFO [main] org.apache.hadoop.mapred.Task: Runnning cleanup for the task 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping MapTask metrics system... 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system stopped.
2015-08-27 10:28:52,269 INFO [main] org.apache.hadoop.mapred.Task:运行清理任务 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2 .impl.MetricsSystemImpl:停止 MapTask 指标系统... 2015-08-27 10:28:52,278 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl:MapTask 指标系统停止。 2015-08-27 10:28:52,279 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system shutdown complete.
2015-08-27 10:28:52,279 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl:MapTask 指标系统关闭完成。
Please suggest.请建议。 How can this issue be resolved.
如何解决这个问题。 and why is it occuring.
以及为什么会发生。
Thanks谢谢
It is a classpath problem Root cause is java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe这是一个类路径问题根本原因是 java.lang.ClassNotFoundException: Class com.cloudera.hive.serde.JSONSerDe
you can have a look here to add hive-serdes: https://github.com/cloudera/cdh-twitter-example您可以在这里查看添加 hive-serdes: https : //github.com/cloudera/cdh-twitter-example
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.