oozie 中 Hive 操作的作业队列

Question

I have a oozie workflow. I am submitting all the hive actions with

<name>mapred.job.queue.name</name>
<value>${queueName}</value>

But for few hive actions, the job launched is not in specified queue; it is invoked in default queue.

Please suggest me the cause behind this behavior and solution.

Answer 1

A. Oozie specifics Oozie propagates the "regular" Hadoop properties to a "regular" MapReduce Action.

But for other types of Action (Shell, Hive, Java, etc.) where Oozie runs a single Mapper task in YARN, it does not consider that it's a real MapReduce job. Hence it uses a different set of undocumented properties always prefixed with oozie.launcher. Look into that post for example.

So in your case the actual property to set would be oozie.launcher.mapred.job.queue.name

B. TEZ specifics Excerpt from HortonWorks documentation :

For example, in Hive you can use the tez.queue.name property in hive-site.xml to specify the queue to use for Hive-on-Tez jobs. To assign Hive-on-Tez jobs to use the "engineering" queue, add the following property to hive-site.xml:

<property>  <name>tez.queue.name</name> <value>engineering</value> </property>

Answer 2

In my case using Oozie it seems that there is no way to add --hiveconf tez.queue.name=myqueue in an xml tag. Trying also by updating the hive-site.xml on hdfs and still no usage of the defined queue. I've end up with updating the .hql file by adding the below at the beginning of the required query:

# in your hql file
SET tez.queue.name=${myqueue};