如何在 Oozie 中使用 hive 操作運行 shell 腳本？

Question

我每天都在嘗試在 Oozie 中使用 hive 操作運行 shell 腳本。 我已經在 Oozie 中成功執行了操作，但是 shell 腳本中的 hive 部分不起作用。 當我從 shell 運行腳本時，它工作正常。 位於 HDFS 中的文件。 這是例外

Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1422)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:62)
    at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:72)
    at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2457)
    at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2469)
    at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:341)
    ... 7 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1420)
    ... 12 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: GSS initiate failed
    at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:221)
    at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:297)
    at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
    at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)

這是我的腳本

S=$(hive -S -hiveconf MY_VAR1=$DB -hiveconf MY_VAR2=$avgpay -hiveconf MY_VAR3=$Date_LastDay -hiveconf MY_VAR4=$Date_LastNmonth -f hv.hql)

`mysql ...`
S1=( $( for k in $S ; do echo $k ; done ) )
    cntn=${#S1[@]}
    for (( p=0 ; p<$cntn; p=p+5 ))
     do
     `mysql ...`
     done

這是工作流程

<workflow-app name="shell-wf" xmlns="uri:oozie:workflow:0.4" >
<start to="shellbpxp"/>
<action name="shellbpxp">
<shell xmlns="uri:oozie:shell-action:0.1">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<configuration>
 <property>
    <name>mapred.job.queue.name</name>
    <value>${queueName}</value>
 </property>
</configuration>
 <exec>netcool.sh</exec>
   <file>netcool.sh#netcool.sh</file>
   <file>hv.hql#hv.hql</file>
</shell>
 <ok to="end" />
<error to="fail" />
</action>
 <kill name="fail">
    <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
 </kill>
<end name='end' />
</workflow-app>

Answer 1

您需要將 hive-config.xml 文件作為您的工作流.xml 中的文件參數，如下所示：

<workflow-app name="shell-wf" xmlns="uri:oozie:workflow:0.4" > 
<start to="shellbpxp"/> 
<action name="shellbpxp"> 
<shell xmlns="uri:oozie:shell-action:0.1">

   <job-tracker>${jobTracker}</job-tracker>
   <name-node>${nameNode}</name-node>
   <job-xml>/user/<<your_path>>/hive-config.xml</job-xml> <configuration>  <property>
       <name>mapred.job.queue.name</name>
       <value>${queueName}</value>  </property> </configuration>  <exec>netcool.sh</exec>    <file>netcool.sh#netcool.sh</file>   
   <file>hv.hql#hv.hql</file>   
   <file>/user/<<your_path>>/hive-config.xml#hive-config.xml</file> </shell>  <ok to="end" /> <error to="fail" /> </action>  <kill
   name="fail">
       <message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>  </kill>
   <end name='end' /> </workflow-app>

您可以在 /etc/hive/conf 目錄下找到 hive-config.xml 文件。 您需要將 hive-config.xml 文件中的 fs.defaultFS 節點注釋掉。

hive-config.xml 包含用於連接到數據庫的 Metastore-uris。

Answer 2

在工作流 xml 中的參數標簽之后添加這個

<env-var>HADOOP_USER_NAME=${wf:user()}</env-var>

應該管用