Oozie 工作流中的 Hive 操作执行参数设置

Question

I am using Hive Action for executing queries through Oozie. I am setting TEZ and MR as execution engine for queries. How can I set maximum possible number of reducers to execute on for query? Currently, I am using mapred.reduce.tasks but it takes a static number.

The real problem is, when I execute same queries on hive CLI, number of reducers chosen by Hive are optimal and not 1; so what setting is my Oozie job missing that it is choosing 1 reducer for all the queries?

Answer 1

Usually the ideal way to control the number of reducers of a Hive query is to use the hive.exec.reducers.bytes.per.reducer property.

The default value is 1 GB, where for every 1gb size of your input files one reducer will be dispatched.

Try to relatively reduce this value according to the expected maximum number of reducers. By this way you may eliminate setting static number of reducers using the mapred.reduce.tasks property.

Answer 2

When running a hive action in oozie you should always set a configurations property for mapred.reduce.tasks = -1 . This will force the optimal reducer value to be created based on your system and available resources.