堆棧內存溢出
  簡體   English   中英

使用MASTER = yarn-cluster運行HiveFromSpark示例

[英]Run HiveFromSpark example with MASTER=yarn-cluster

 原文  2015-04-15 10:46:05  2   3    hadoop/ apache-spark/ amazon-emr/ apache-spark-sql

問題描述

我正在嘗試在我的EMR Spark / Hive群集上運行HiveFromSpark示例。

問題

使用yarn-client ： 

~/spark/bin/spark-submit --master yarn-client --num-executors=19 --class org.apache.spark.examples.sql.hive.HiveFromSpark  ~/spark/lib/spark-examples-1.3.0-hadoop2.4.0.jar

奇跡般有效。 但是，使用yarn-cluster ： 

~/spark/bin/spark-submit --master yarn-cluster --num-executors=19 --class org.apache.spark.examples.sql.hive.HiveFromSpark ~/spark/lib/spark-examples-1.3.0-hadoop2.4.0.jar

失敗： 

Exception in thread "main" org.apache.spark.SparkException: Application finished with failed status
    at org.apache.spark.deploy.yarn.Client.run(Client.scala:622)
    at org.apache.spark.deploy.yarn.Client$.main(Client.scala:647)
    at org.apache.spark.deploy.yarn.Client.main(Client.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:569)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:166)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:189)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:110)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

並且資源管理器中的錯誤消息顯示： 

User class threw exception: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient

完整日志： 

15/04/15 12:02:14 WARN DataNucleus.Persistence: Error creating validator of type org.datanucleus.properties.CorePropertyValidator
ClassLoaderResolver for class "" gave error on creation : {1}
org.datanucleus.exceptions.NucleusUserException: ClassLoaderResolver for class "" gave error on creation : {1}
    at org.datanucleus.NucleusContext.getClassLoaderResolver(NucleusContext.java:1087)
    at org.datanucleus.PersistenceConfiguration.validatePropertyValue(PersistenceConfiguration.java:797)
    at org.datanucleus.PersistenceConfiguration.setProperty(PersistenceConfiguration.java:714)
    at org.datanucleus.PersistenceConfiguration.setPersistenceProperties(PersistenceConfiguration.java:693)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:273)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:247)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:225)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:416)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:301)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
    at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
    at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
    at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339)
    at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248)
    at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
    at org.apache.spark.sql.hive.HiveContext.sessionState$lzycompute(HiveContext.scala:229)
    at org.apache.spark.sql.hive.HiveContext.sessionState(HiveContext.scala:225)
    at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:241)
    at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:240)
    at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:86)
    at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:49)
    at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
Caused by: java.lang.NullPointerException
    at org.datanucleus.NucleusContext.getClassLoaderResolver(NucleusContext.java:1079)
    ... 58 more
Nested Throwables StackTrace:
java.lang.NullPointerException
    at org.datanucleus.NucleusContext.getClassLoaderResolver(NucleusContext.java:1079)
    at org.datanucleus.PersistenceConfiguration.validatePropertyValue(PersistenceConfiguration.java:797)
    at org.datanucleus.PersistenceConfiguration.setProperty(PersistenceConfiguration.java:714)
    at org.datanucleus.PersistenceConfiguration.setPersistenceProperties(PersistenceConfiguration.java:693)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:273)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:247)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:225)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:416)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:301)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
    at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
    at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
    at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339)
    at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248)
    at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
    at org.apache.spark.sql.hive.HiveContext.sessionState$lzycompute(HiveContext.scala:229)
    at org.apache.spark.sql.hive.HiveContext.sessionState(HiveContext.scala:225)
    at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:241)
    at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:240)
    at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:86)
    at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:49)
    at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
15/04/15 12:02:14 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.    HiveMetaStoreClient
java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:346)
    at org.apache.spark.sql.hive.HiveContext.sessionState$lzycompute(HiveContext.scala:229)
    at org.apache.spark.sql.hive.HiveContext.sessionState(HiveContext.scala:225)
    at org.apache.spark.sql.hive.HiveContext.hiveconf$lzycompute(HiveContext.scala:241)
    at org.apache.spark.sql.hive.HiveContext.hiveconf(HiveContext.scala:240)
    at org.apache.spark.sql.hive.HiveContext.sql(HiveContext.scala:86)
    at org.apache.spark.examples.sql.hive.HiveFromSpark$.main(HiveFromSpark.scala:49)
    at org.apache.spark.examples.sql.hive.HiveFromSpark.main(HiveFromSpark.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:480)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1412)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2453)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2465)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:340)
    ... 12 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1410)
    ... 17 more
Caused by: javax.jdo.JDOFatalInternalException: Unexpected exception caught.
NestedThrowables:
java.lang.reflect.InvocationTargetException
    at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1193)
    at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:808)
    at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:701)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:310)
    at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:339)
    at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:248)
    at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:223)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:58)
    at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStore(HiveMetaStore.java:497)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:475)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:523)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:397)
    at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.<init>(HiveMetaStore.java:356)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:54)
    at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:59)
    at org.apache.hadoop.hive.metastore.HiveMetaStore.newHMSHandler(HiveMetaStore.java:4944)
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:171)
    ... 22 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at javax.jdo.JDOHelper$16.run(JDOHelper.java:1965)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.jdo.JDOHelper.invoke(JDOHelper.java:1960)
    at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1166)
    ... 41 more
Caused by: org.datanucleus.exceptions.NucleusUserException: Persistence process has been specified to use a ClassLoaderResolver of name "datanucleus" yet this has   not   been found by the DataNucleus plugin mechanism. Please check your CLASSPATH and plugin specification.
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:283)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:247)
    at org.datanucleus.NucleusContext.<init>(NucleusContext.java:225)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.<init>(JDOPersistenceManagerFactory.java:416)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:301)
    at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:202)
    ... 49 more
15/04/15 12:02:14 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.RuntimeException: Unable to     instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient)

集群設置

EMR配置

我通過Install Spark （s3：//support.elasticmapreduce/spark/install-spark）引導操作和Setup Hive步驟來設置集群。 

JAR location: s3://elasticmapreduce/libs/script-runner/script-runner.jar
Main class: None
Arguments: s3://elasticmapreduce/libs/hive/hive-script --base-path s3://elasticmapreduce/libs/hive/ --install-hive --hive-versions 0.13.1

軟件版本

hadoop version 

Hadoop 2.4.0-amzn-3
Subversion http://aws.amazon.com/elasticmapreduce/ -r d99c64d21bf338dcdd524bd08d7c393b6d1ac1d8
Compiled by Elastic MapReduce on 2015-02-11T20:22Z
Compiled with protoc 2.5.0
From source with checksum 6c725ed23b3ecb95921fe461587fccf
This command was run using /home/hadoop/.versions/2.4.0-amzn-3/share/hadoop/common/hadoop-common-2.4.0-amzn-3.jar

Spark版本 

Spark 1.3.0 built for Hadoop 2.4.0
Build flags: -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests -Pkinesis-asl -Pspark-ganglia-lgpl -Phadoop-provided -Phive -Phive-thriftserver

3 個解決方案

解決方案1
 5  2015-04-15 19:19:11

提交時使用--jars選項添加datanucleus jar，並將hive-site.xml添加到驅動程序類路徑，以便spark可以讀取hive配置。 

--jars spark/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar,spark/lib_managed/jars/datanucleus-rdbms-3.2.9.jar,spark/lib_managed/jars/datanucleus-core-3.2.10.jar,spark/mysql-connector-java-5.0.8-bin.jar --files hive/conf/hive-site.xml

解決方案2
 0  2015-06-02 18:22:41

對我而言，hadooped的答案實際上失敗了，並顯示以下錯誤： 

15/06/02 17:41:41 ERROR Datastore.Schema: Failed initialising database. Unable to open a test connection to the given database. JDBC url = jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true, username = hive. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception:
------ java.sql.SQLException: No suitable driver found for jdbc:mysql://localhost:3306/hive?createDatabaseIfNotExist=true

但是當我刪除hive-site.xml文件參數並使用正確的路徑（對於2015-6-1的EMR）時，它起作用了： 

--jars spark/lib/datanucleus-api-jdo-3.2.6.jar,spark/lib/datanucleus-core-3.2.10.jar,spark/lib/datanucleus-rdbms-3.2.9.jar,hive/lib/mysql-connector-java-5.1.30.jar

解決方案3
 0  2018-12-19 09:45:18

I faced the same issue while executing Spark Hive code.

Spark version 2.0.2

Hive version 1.2.1

The issue got fixed when I added following jars and hive-site.xml to the classpath:

/usr/hdp/2.5.3.0-37/spark2/jars/datanucleus-api-jdo-3.2.6.jar,   
/usr/hdp/2.5.3.0-37/spark2/jars/datanucleus-core-3.2.10.jar,   
/usr/hdp/2.5.3.0-37/spark2/jars/datanucleus-rdbms-3.2.9.jar

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 使用--master yarn-cluster運行spark-submit：spark-assembly問題 紗線群集模式下Spark作業的ClassNotFoundException Spark在紗簇上提交-Hive錯誤 連接CDH 5.4中的紗線簇上的火花 為報告平台選擇哪種紗線群集或紗線客戶端？ 使用--jars的spark-submit yarn-cluster不起作用？ Spark yarn-cluster模式-讀取通過--files傳遞的文件 使用Spark-SQL通過yarn-cluster的壞hdfs權限 使用yarn-cluster模式初始化SparkContext時出錯 有關故障轉移過程如何在紗線群集模式下為Spark驅動程序（及其YARN容器）工作的資源/文檔
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM