火花不讀hive-site.xml嗎？

Question

我正在嘗試訪問配置單元metastore，為此我正在使用SparkSql。 我已經設置了sparksession，但是當我運行程序並查看日志時，看到此異常

Caused by: java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
    at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:188)
    ... 61 more
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1523)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:86)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:132)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3005)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3024)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)
    ... 62 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1521)
    ... 68 more
Caused by: javax.jdo.JDOFatalDataStoreException: Unable to open a test connection to the given database. JDBC url = jdbc:derby:;databaseName=metastore_db;create=true, username = APP. Terminating connection pool (set lazyInit to true if you expect to start your database after your app). Original Exception: ------
java.sql.SQLException: Failed to create database 'metastore_db', see the next exception for details.

我正在運行一個訪問以下代碼的servlet。

public class HiveReadone extends HttpServlet {
    private static final long serialVersionUID = 1L;

    /**
     * @see HttpServlet#HttpServlet()
     */
    public HiveReadone() {
        super();
        // TODO Auto-generated constructor stub
    }

    /**
     * @see HttpServlet#doGet(HttpServletRequest request, HttpServletResponse response)
     */
    protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {
        // TODO Auto-generated method stub
        response.getWriter().append("Served at: ").append(request.getContextPath());

        SparkSession spark = SparkSession
                .builder()
                .appName("Java Spark SQL basic example")
                .enableHiveSupport()
                .config("spark.sql.warehouse.dir", "hdfs://saurab:9000/user/hive/warehouse")
                .config("mapred.input.dir.recursive", true)
                .config("hive.mapred.supports.subdirectories", true)
                .config("hive.vectorized.execution.enabled", true)
                .master("local")
                .getOrCreate();
        response.getWriter().println(spark);

在瀏覽器上沒有任何打印輸出接受response.getWriter().append("Served at: ").append(request.getContextPath()); 可以Served at: /hiveServ

請看看我的conf/hive-site.xml

<property>
        <name>hive.metastore.schema.verification</name>
        <value>false</value>
    </property>
    <property>
        <name>hive.server2.enable.doAs</name>
        <value>true</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
        <value>jdbc:mysql://saurab:3306/metastore_db?createDatabaseIfNotExist=true</value>
        <description>metadata is stored in a MySQL server</description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
        <description>MySQL JDBC driver class</description>
    </property>
    <property>
        <name>hive.aux.jars.path</name>
        <value>/home/saurab/hadoopec/hive/lib/hive-serde-2.1.1.jar</value>
    </property>
    <property>
        <name>spark.sql.warehouse.dir</name>
        <value>hdfs://saurab:9000/user/hive/warehouse</value>
    </property>
    <property>
        <name>hive.metastore.uris</name>
        <!--Make sure that <value> points to the Hive Metastore URI in your cluster -->
        <value>thrift://saurab:9083</value>
        <description>URI for client to contact metastore server</description>
    </property>
    <property>
        <name>hive.server2.thrift.port</name>
        <value>10001</value>
        <description>Port number of HiveServer2 Thrift interface.
            Can be overridden by setting $HIVE_SERVER2_THRIFT_PORT
        </description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>hiveuser</value>
        <description>user name for connecting to mysql server</description>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>hivepassword</value>
        <description>password for connecting to mysql server</description>
    </property>

據我了解，如果我們配置hive.metastore.uris spark將連接到hive metastore，但是在我的情況下，它沒有連接，並給了我上面的錯誤。

Answer 1

要在hive上配置spark，請嘗試將hive-site.xml復制到spark / conf目錄。