OOZIE workflow: HIVE table did not exists but directory created in HDFS

Question

I am trying to run a HIVE action using a OOZIE workflow. Below is the hive action:

create table abc (a INT);

I can locate the internal table in HDFS (directory abc getting created under /user/hive/warehouse ) but when I trigger the command SHOW TABLES from hive> , I am not able to see the table.

This is the workflow.xml file:

<workflow-app xmlns="uri:oozie:workflow:0.2" name="hive-wf">
 <start to="hiveac"/>
 <action name="hiveac">
    <hive xmlns="uri:oozie:hive-action:0.2">
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <!-- <prepare> <delete path="${nameNode}/user/${wf:user()}/case1/out"/> </prepare> -->
        <!-- <job-xml>hive-default.xml</job-xml>-->
            <configuration>
                <property>
                    <name>oozie.hive.defaults</name>
                    <value>hive-default.xml</value>
                </property>
                <property>
                    <name>mapred.job.queue.name</name>
                    <value>${queueName}</value>
                </property>
            </configuration>
            <script>script.q</script>
            <!-- <param>INPUT=/user/${wf:user()}/case1/sales_history_temp4</param>
            <param>OUTPUT=/user/${wf:user()}/case1/out</param> -->
        </hive>
  <ok to="end"/>
  <error to="fail"/>
 </action>
   <kill name="fail">
   <message>Pig Script failed!!!</message>
   </kill>
   <end name="end"/>
</workflow-app>

This is the hive-default.xml file:

<configuration>
<property>
  <name>javax.jdo.option.ConnectionURL</name>
  <value>jdbc:mysql://localhost/metastore</value>
  <description>JDBC connect string for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionDriverName</name>
  <value>org.apache.derby.jdbc.EmbeddedDriver</value>
  <description>Driver class name for a JDBC metastore</description>
</property>

<property>
  <name>javax.jdo.option.ConnectionUserName</name>
  <value>hiveuser</value>
</property>
<property>
  <name>javax.jdo.option.ConnectionPassword</name>
  <value>password</value>
</property>
<property>
  <name>datanucleus.autoCreateSchema</name>
  <value>false</value>
</property>
<property>
  <name>datanucleus.fixedDatastore</name>
  <value>true</value>
</property>
<property>
  <name>hive.stats.autogather</name>
  <value>false</value>
</property>
</configuration>

This is the job.properties file:

nameNode=hdfs://localhost:8020
jobTracker=localhost:8021
queueName=default
oozie.libpath=/user/oozie/shared/lib
#oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/my/jobhive

The logs did not gave any errors as such:

stderr logs

Logging initialized using configuration in jar:file:/var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/distcache/3179985539753819871_-620577179_884768063/localhost/user/oozie/shared/lib/hive-common-0.9.0-cdh4.1.1.jar!/hive-log4j.properties
Hive history file=/tmp/mapred/hive_job_log_mapred_201603060735_17840386.txt
OK
Time taken: 9.322 seconds
Log file: /var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/training/jobcache/job_201603060455_0012/attempt_201603060455_0012_m_000000_0/work/hive-oozie-job_201603060455_0012.log  not present. Therefore no Hadoop jobids found

I came across a similar thread: Tables created by oozie hive action cannot be found from hive client but can find them in HDFS

But this did not resolved my issue. Please let me know how to resolve this issue.

Answer 1

I haven't used Oozie for a couple months (and did not keep archives because of legal reasons) and anyway it was V4.x so it's a bit of guesswork...

1. upload your valid hive-site.xml to HDFS somewhere
2. tell Oozie to inject all these properties in the Launcher Configuration before running the Hive class, so that it inherits them all, with <job-xml>/some/hdfs/path/hive-site.xml</job-xml>
3. remove any reference to oozie.hive.defaults

Warning: all that assumes that your sandbox cluster has a persistent Metastore -- ie your hive-site.xml does not point to a Derby embedded DB that gets erased every time!