WSO2 BAM問題與蜂巢查詢到jdbc到MySQL外部DB：錯誤

Question

我正在測試通過REST API將數據從Perl推送到BAM中的情況，這似乎運行良好。 現在，下一步就是將聚合測試到另一台主機上的單獨的MySQL數據庫中，並且出現了一個我似乎無法克服的錯誤。 請參閱下面的堆棧跟蹤。

Hive history file=/Users/<user>/local/wso2bam-2.4.1/tmp/hive/wso2-querylogs/hive_job_log_jskogsta_201406141932_1331503753.txt
Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler
[2014-06-14 19:33:00,180] ERROR {org.apache.hadoop.hive.ql.exec.Task} -  Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler
    at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:278)
    at org.apache.hadoop.hive.ql.exec.DDLTask.createTable(DDLTask.java:3360)
    at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:225)
    at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:129)
    at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:62)
    at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1351)
    at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1126)
    at org.apache.hadoop.hive.ql.Driver.run(Driver.java:934)
    at org.apache.hadoop.hive.service.HiveServer$HiveServerHandler.execute(HiveServer.java:201)
    at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:187)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.executeHiveQuery(HiveExecutorServiceImpl.java:577)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:287)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:190)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler
    at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:151)
    at org.apache.hadoop.hive.ql.metadata.Table.getStorageHandler(Table.java:273)
    ... 17 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler
    at org.wso2.carbon.ui.JspClassLoader.findClass(JspClassLoader.java:113)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
    at org.wso2.carbon.ui.JspClassLoader.loadClass(JspClassLoader.java:108)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:249)
    at org.apache.hadoop.hive.ql.metadata.HiveUtils.getStorageHandler(HiveUtils.java:144)
    ... 18 more

FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[2014-06-14 19:33:00,180] ERROR {org.apache.hadoop.hive.ql.Driver} -  FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
[2014-06-14 19:33:00,183] ERROR {org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl} -  Error while executing Hive script.
Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
java.sql.SQLException: Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
    at org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:189)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.executeHiveQuery(HiveExecutorServiceImpl.java:577)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:287)
    at org.wso2.carbon.analytics.hive.impl.HiveExecutorServiceImpl$ScriptCallable.call(HiveExecutorServiceImpl.java:190)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
    at java.util.concurrent.FutureTask.run(FutureTask.java:138)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:695)
[2014-06-14 19:33:00,200] ERROR {org.wso2.carbon.analytics.hive.ui.client.HiveExecutionClient} -  Error while executing Hive script.Query returned non-zero code: 9, cause: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
org.wso2.carbon.analytics.hive.stub.HiveExecutionServiceHiveExecutionException: HiveExecutionServiceHiveExecutionException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at java.lang.Class.newInstance0(Class.java:357)
    at java.lang.Class.newInstance(Class.java:310)
    at org.wso2.carbon.analytics.hive.stub.HiveExecutionServiceStub.executeHiveScript(HiveExecutionServiceStub.java:216)
    at org.wso2.carbon.analytics.hive.ui.client.HiveExecutionClient.executeScript(HiveExecutionClient.java:66)
    at org.apache.jsp.hive_002dexplorer.queryresults_jsp._jspService(org.apache.jsp.hive_002dexplorer.queryresults_jsp:92)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:403)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.wso2.carbon.ui.JspServlet.service(JspServlet.java:155)
    at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:80)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
    at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
    at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:605)
    at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:544)
    at org.eclipse.equinox.http.servlet.internal.RequestDispatcherAdaptor.include(RequestDispatcherAdaptor.java:37)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor$RequestDispatcherAdaptor.include(ContextPathServletAdaptor.java:369)
    at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:1015)
    at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:700)
    at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.tiles.jsp.context.JspUtil.doInclude(JspUtil.java:87)
    at org.apache.tiles.jsp.context.JspTilesRequestContext.include(JspTilesRequestContext.java:88)
    at org.apache.tiles.jsp.context.JspTilesRequestContext.dispatch(JspTilesRequestContext.java:82)
    at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:465)
    at org.apache.tiles.jsp.taglib.InsertAttributeTag.render(InsertAttributeTag.java:140)
    at org.apache.tiles.jsp.taglib.InsertAttributeTag.render(InsertAttributeTag.java:117)
    at org.apache.tiles.jsp.taglib.RenderTagSupport.execute(RenderTagSupport.java:171)
    at org.apache.tiles.jsp.taglib.RoleSecurityTagSupport.doEndTag(RoleSecurityTagSupport.java:75)
    at org.apache.tiles.jsp.taglib.ContainerTagSupport.doEndTag(ContainerTagSupport.java:80)
    at org.apache.jsp.admin.layout.template_jsp._jspx_meth_tiles_insertAttribute_7(org.apache.jsp.admin.layout.template_jsp:603)
    at org.apache.jsp.admin.layout.template_jsp._jspService(org.apache.jsp.admin.layout.template_jsp:335)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:111)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:403)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:492)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:378)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.wso2.carbon.ui.JspServlet.service(JspServlet.java:155)
    at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:80)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
    at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:749)
    at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:487)
    at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:412)
    at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:339)
    at org.eclipse.equinox.http.servlet.internal.RequestDispatcherAdaptor.forward(RequestDispatcherAdaptor.java:30)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor$RequestDispatcherAdaptor.forward(ContextPathServletAdaptor.java:362)
    at org.apache.tiles.servlet.context.ServletTilesRequestContext.forward(ServletTilesRequestContext.java:198)
    at org.apache.tiles.servlet.context.ServletTilesRequestContext.dispatch(ServletTilesRequestContext.java:185)
    at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:419)
    at org.apache.tiles.impl.BasicTilesContainer.render(BasicTilesContainer.java:370)
    at org.wso2.carbon.ui.action.ActionHelper.render(ActionHelper.java:52)
    at org.wso2.carbon.ui.TilesJspServlet.service(TilesJspServlet.java:101)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.eclipse.equinox.http.helper.ContextPathServletAdaptor.service(ContextPathServletAdaptor.java:37)
    at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
    at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
    at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
    at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
    at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
    at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
    at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
    at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
    at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
    at java.lang.Thread.run(Thread.java:695)

我什至下載了MySQL jar文件並放入/ lib：

[19:31:00] [<user>@enterprise ../local/wso2bam-2.4.1/lib]$ ls
ant-contrib-1.0b3.jar           org.wso2.carbon.logging-4.2.0.jar   tomcat
commons-lang-2.6.0.wso2v1.jar       org.wso2.carbon.server-4.2.0.jar    transactions.properties
endorsed                org.wso2.ciphertool-1.0.0-wso2v2.jar
mysql-connector-java-5.1.31.tar     runtimes
[19:45:34] [<user>@enterprise ../local/wso2bam-2.4.1/lib]$

這是我嘗試在BAM admin gui中執行的查詢，但失敗了：

CREATE EXTERNAL TABLE IF NOT EXISTS Summarized_Table ( payload_symbol STRING, payload_min INT, payload_price STRING, payload_volume INT, payload_max INT)
STORED BY 'org.apache.hadoop.hive.jdbc.storage.JDBCStorageHandler'
TBLPROPERTIES ( 'mapred.jdbc.driver.class' = 'com.mysql.jdbc.Driver' ,
'mapred.jdbc.url' = 'jdbc:mysql://<IP>:<PORT>/testdb' ,
'mapred.jdbc.username' = '<USER>' ,
'mapred.jdbc.password' = '<PASS>' ,
'hive.jdbc.update.on.duplicate' = 'true' ,
'hive.jdbc.table.create.query' = 'CREATE TABLE testtable
                                (payload_symbol VARCHAR(10), payload_min INT, payload_price VARCHAR(20), payload_volume INT, payload_max INT) ');

在執行此查詢之前，我正在執行以下查詢：

CREATE EXTERNAL TABLE IF NOT EXISTS TestTable (Row_ID STRING, payload_symbol STRING,
          payload_min INT, payload_price STRING, payload_volume INT, payload_max INT)
STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler'
WITH SERDEPROPERTIES ( "cassandra.host" = "127.0.0.1" ,
"cassandra.port" = "9160" ,
"cassandra.ks.name" = "EVENT_KS" ,
"cassandra.ks.username" = "<USER>" ,
"cassandra.ks.password" = "<PASS>" ,
"cassandra.cf.name" = "stockquote_stream" ,
"cassandra.columns.mapping" = ":key,payload_symbol,payload_min,payload_price,payload_volume,payload_max" );

我已經確認可以從這台計算機連接到遠程數據庫：

[19:51:22] [<USER>@enterprise ../local/wso2bam-2.4.1/lib]$ mysql -u currentcost -h <IP> --port=<PORT> -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 43
Server version: 5.5.34 Source distribution

Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> exit
Bye
[19:51:46] [<USER>@enterprise ../local/wso2bam-2.4.1/lib]$

這是BAM的標准默認安裝，完全沒有執行任何操作。

任何人都可以指出解決問題的方法嗎？ 將不勝感激！

[解決方案更新]我設法解決了這個問題，無論您是否相信... MySQL的第三方.jar的$ CARBON_CLASSPATH問題。 最終將mysql-connector-java-5.1.31-bin.jar文件放在WSO2 BAM安裝中的各個/ lib /文件夾中，並且可以正常工作。 現在是執行的控制台日志，數據現在顯示在遠程MySQL數據庫中。

Hive history file=/Users/<USER>/local/wso2bam-2.4.1/tmp/hive/wso2-querylogs/hive_job_log_<USER>_201406151106_798138044.txt
OK
OK
OK
Copying data from file:/Users/<USER>/projects/ccost/prod.txt
Copying file: file:/Users/<USER>/projects/ccost/prod.txt
Loading data to table default.producttable
OK
OK
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=<number>
Execution log at: /Users/<USER>/local/wso2bam-2.4.1/repository/logs//wso2carbon.log
2014-06-15 11:13:07.935 java[2137:1003] Unable to load realm info from SCDynamicStore
[2014-06-15 11:13:08,473]  WARN {org.apache.hadoop.mapred.JobClient} -  Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
Job running in-process (local Hadoop)
Hadoop job information for null: number of mappers: 0; number of reducers: 0
2014-06-15 11:13:11,720 null map = 0%,  reduce = 0%
2014-06-15 11:13:14,722 null map = 100%,  reduce = 0%
2014-06-15 11:13:17,725 null map = 100%,  reduce = 100%
Ended Job = job_local_0001
Execution completed successfully
Mapred Local Task Succeeded . Convert the Join into MapJoin
OK

因此-到目前為止一切都很好。 現在到下一個問題。

Answer 1

您必須將任何第3方庫復制到<BAM_HOME>/repository/components/lib文件夾中。