堆栈内存溢出
  简体   繁体   English

设置hive.exec.pre.hooks时发生ClassNotFoundException

[英]ClassNotFoundException when setting hive.exec.pre.hooks

 原文  2018-08-12 10:08:30  6   1    java/ hadoop/ hive/ hook

问题描述

I am following this document to do hive hook: 我正在按照此文档进行配置单元挂钩：

http://dharmeshkakadia.github.io/hive-hook/ http://dharmeshkakadia.github.io/hive-hook/

But I got this error when show tables 但是当show tables时show tables此错误 

2018-08-12 09:57:38,122 ERROR org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-315]: hive.exec.pre.hooks Class not found: HiveExampleHook
2018-08-12 09:57:38,122 ERROR org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-315]: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(HiveExampleHook)
java.lang.ClassNotFoundException: HiveExampleHook
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.hadoop.hive.ql.hooks.HooksLoader.getHooks(HooksLoader.java:100)
    at org.apache.hadoop.hive.ql.hooks.HooksLoader.getHooks(HooksLoader.java:64)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1674)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
    at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
    at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
    at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

2018-08-12 09:57:38,122 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-315]: </PERFLOG method=Driver.execute start=1534067858120 end=1534067858122 duration=2 from=org.apache.hadoop.hive.ql.Driver>
2018-08-12 09:57:38,122 INFO  org.apache.hadoop.hive.ql.Driver: [HiveServer2-Background-Pool: Thread-315]: Completed executing command(queryId=hive_20180812095757_e6516d83-ddc9-4f82-8151-def7e7f1eb37); Time taken: 0.002 seconds
2018-08-12 09:57:38,122 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-315]: <PERFLOG method=releaseLocks from=org.apache.hadoop.hive.ql.Driver>
2018-08-12 09:57:38,122 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-315]: </PERFLOG method=releaseLocks start=1534067858122 end=1534067858122 duration=0 from=org.apache.hadoop.hive.ql.Driver>
2018-08-12 09:57:38,130 ERROR org.apache.hive.service.cli.operation.Operation: [HiveServer2-Background-Pool: Thread-315]: Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(HiveExampleHook)
    at org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:400)
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:238)
    at org.apache.hive.service.cli.operation.SQLOperation.access$300(SQLOperation.java:89)
    at org.apache.hive.service.cli.operation.SQLOperation$3$1.run(SQLOperation.java:301)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
    at org.apache.hive.service.cli.operation.SQLOperation$3.run(SQLOperation.java:314)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: HiveExampleHook
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:270)
    at org.apache.hadoop.hive.ql.hooks.HooksLoader.getHooks(HooksLoader.java:100)
    at org.apache.hadoop.hive.ql.hooks.HooksLoader.getHooks(HooksLoader.java:64)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1674)
    at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1501)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1285)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1280)
    at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:236)
    ... 11 more

I am sure the last step add jar target/Hive-hook-example-1.0.jar; 我确定最后一步要add jar target/Hive-hook-example-1.0.jar; is wrong. 是错的。

I tried the following: 我尝试了以下方法：

  1. I put the jar file into hdfs /user/hive/ : 我将jar文件放入hdfs / user / hive /中：

    add jar hdfs:///user/hive/Hive-hook-example-1.0.jar; 添加jar hdfs：///user/hive/Hive-hook-example-1.0.jar;

  2. I also set "Hive Auxiliary JARs Directory" as /home/centos/HiveExampleHook/target/Hive-hook-example-1.0.jar in Hiveserver2 node and restart Hive plus beeline. 我还在Hiveserver2节点中将“ Hive辅助JAR目录”设置为/home/centos/HiveExampleHook/target/Hive-hook-example-1.0.jar，然后重新启动Hive加beeline。

  3. Copy the jar file to /opt/cloudera/parcels/CDH/jars/ 将jar文件复制到/opt/cloudera/parcels/CDH/jars/

  4. Copy the jar file to /opt/cloudera/parcels/CDH/lib/hive/lib/ 将jar文件复制到/opt/cloudera/parcels/CDH/lib/hive/lib/

Nothing helps. 没有任何帮助。

Any idea? 任何想法？

UPDATE 1: 更新1：

If I do LIST JARS; 如果我LIST JARS; this would show 这将显示 

+----------------------------------------------------+--+
|                      resource                      |
+----------------------------------------------------+--+
| /tmp/3fe67bb1-5cfd-427f-8faa-cab6524afeb3_resources/Hive-hook-example-1.0.jar |
+----------------------------------------------------+--+

I tried two ways to do CREATE FUNCTION too: 我也尝试了两种方法来CREATE FUNCTION ： 

CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook';
INFO  : Compiling command(queryId=hive_20180812153838_47589f9d-eaeb-410d-80b0-9cf414ae557f): CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook'
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20180812153838_47589f9d-eaeb-410d-80b0-9cf414ae557f); Time taken: 0.002 seconds
INFO  : Executing command(queryId=hive_20180812153838_47589f9d-eaeb-410d-80b0-9cf414ae557f): CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook'
INFO  : Starting task [Stage-0:FUNC] in serial mode
ERROR : FAILED: Class HiveExampleHook not found
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask
INFO  : Completed executing command(queryId=hive_20180812153838_47589f9d-eaeb-410d-80b0-9cf414ae557f); Time taken: 0.003 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask (state=08S01,code=1)

and... 和... 

CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook' USING JAR 'hdfs:///user/hive/Hive-hook-example-1.0.jar';
INFO  : Compiling command(queryId=hive_20180812153939_cf1f31c9-0361-47dc-8903-78221bd12401): CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook' USING JAR 'hdfs:///user/hive/Hive-hook-example-1.0.jar'
INFO  : Semantic Analysis Completed
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling command(queryId=hive_20180812153939_cf1f31c9-0361-47dc-8903-78221bd12401); Time taken: 0.004 seconds
INFO  : Executing command(queryId=hive_20180812153939_cf1f31c9-0361-47dc-8903-78221bd12401): CREATE TEMPORARY FUNCTION test1 AS 'HiveExampleHook' USING JAR 'hdfs:///user/hive/Hive-hook-example-1.0.jar'
INFO  : Starting task [Stage-0:FUNC] in serial mode
INFO  : converting to local hdfs:///user/hive/Hive-hook-example-1.0.jar
INFO  : Added [/tmp/3fe67bb1-5cfd-427f-8faa-cab6524afeb3_resources/Hive-hook-example-1.0.jar] to class path
INFO  : Added resources: [hdfs:///user/hive/Hive-hook-example-1.0.jar]
ERROR : FAILED: Class HiveExampleHook not found
ERROR : FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask
INFO  : Completed executing command(queryId=hive_20180812153939_cf1f31c9-0361-47dc-8903-78221bd12401); Time taken: 0.03 seconds
Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask (state=08S01,code=1)

So clearly it can find the jar but not the class name. 显然，它可以找到jar，但找不到类名。 Am I right? 我对吗？

UPDATE 2: 更新2：

I tried this: 我尝试了这个： 

[Hive-hook-example]# java -cp `pwd`/target/Hive-hook-example-1.0.jar HiveExampleHook

And still got this: 仍然得到这个： 

Error: Could not find or load main class HiveExampleHook

I believe this is some stupid mistake I did. 我相信这是我犯的一些愚蠢的错误。

UPDATE 3: 更新3：

OK I got it figured out. 好吧，我明白了。 You have to use hive CLI and not beeline for this to work. 您必须使用配置单元CLI而不是蜂巢式命令行才能起作用。 

hive> add jar hdfs:///user/hive/Hive-hook-example-1.0.jar;
add jar hdfs:///user/hive/Hive-hook-example-1.0.jar
converting to local hdfs:///user/hive/Hive-hook-example-1.0.jar
Added [/tmp/0a90132d-70cd-4ef0-b4cd-e75dc823e5ca_resources/Hive-hook-example-1.0.jar] to class path
Added resources: [hdfs:///user/hive/Hive-hook-example-1.0.jar]
hive> set hive.exec.pre.hooks=HiveExampleHook;
set hive.exec.pre.hooks=HiveExampleHook
hive> show tables;
show tables
Hello from the hook !!
OK
test1
Time taken: 0.023 seconds, Fetched: 5 row(s)

So the question is how to run this in beeline then? 那么问题是如何在beeline中运行呢？ Because hive CLI is deprecated. 因为不推荐使用Hive CLI。

UPDATE 4: 更新4：

I decided to do this: 我决定这样做： 在此处输入图片说明

Ran beeline and saw this: 跑了beeline并且看见了此： 

2018-08-12 16:39:13,286 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-60]: <PERFLOG method=PreHook.HiveExampleHook from=org.apache.hadoop.hive.ql.Driver>
2018-08-12 16:39:13,286 INFO  org.apache.hadoop.hive.ql.log.PerfLogger: [HiveServer2-Background-Pool: Thread-60]: </PERFLOG method=PreHook.HiveExampleHook start=1534091953286 end=1534091953286 duration=0 from=org.apache.hadoop.hive.ql.Driver>
2

That is some progress although I am not sure what it means and whether the class was ran. 尽管我不确定这意味着什么以及是否已上课，但仍取得了一些进展。 As I see nothing output. 如我所见，没有任何输出。

1 个解决方案

解决方案1
 1  2018-08-12 16:51:03

With beeline, you have to use HDFS path while adding jar. 使用beeline，必须在添加jar时使用HDFS路径。 Remember beeline is just a JDBC CLI, so when you use add jar with local path, it has the reference to you local path, that is not accessible to hive session running on the cluster. 请记住，beeline只是一个JDBC CLI，因此当您将add jar与本地路径一起使用时，它具有对本地路径的引用，集群上运行的hive会话无法访问它。

(Thanks for asking https://twitter.com/quanghoc/status/1028671393376874496 for help. I am the author of the blog you referred to.) （感谢您向https://twitter.com/quanghoc/status/1028671393376874496寻求帮助。我是您提到的博客的作者。）

暂无
暂无

声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

相关问题 球衣：执行前和执行后的挂钩？ - Jersey: pre- and post-exec hooks? 在 Windows 上运行 mvn exec:java 时出现 ClassNotFoundException - ClassNotFoundException when running mvn exec:java on Windows java ClassNotFoundException设置Quartz时的LoggerFactory - java ClassNotFoundException LoggerFactory when setting Quartz up 在Jetbrains TeamCity上执行时，MVN exec：java抛出ClassNotFoundException - mvn exec:java throws ClassNotFoundException when executed on Jetbrains TeamCity 带有 ClassNotFoundException 的 Exec Maven 插件 - Exec Maven Plugin with ClassNotFoundException Maven exec 插件 ClassNotFoundException - Maven exec plugin ClassNotFoundException 引起：java.lang.ClassNotFoundException：org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch出现在flink查询配置单元中 - Caused by: java.lang.ClassNotFoundException: org.apache.orc.storage.ql.exec.vector.VectorizedRowBatch appears in flink query hive 在eclipse和ClassNotFoundException中设置jpf - Setting jpf in eclipse and ClassNotFoundException ClassNotFoundException：org.apache.hive.jdbc.HiveDriver - ClassNotFoundException: org.apache.hive.jdbc.HiveDriver 在subversion（svn）中添加预提交钩子 - adding pre-commit hooks in subversion(svn)
相关标签
 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM