“ Hive on Spark模式”和“ Spark SQL”有什么区别？ “ Hive on Spark模式”会使用Catalyst Optimizer吗？

Question

• Hive on Spark mode vs MR mode
• Spark SQL
• Catalyst Optimizer/RDD/Tungsten

Answer 1

Hive on Spark is different from running Hive queries using Spark SQL via HiveContext. It doesn't translate the queries to Spark primitives but translates them to MapReduce primitives and executes on Spark.Its main purpose to utilize Spark exeuction engine, without impacting existing code in Hive.

Internally, it translates Hive's logical operators to Spark Tasks , which are mostly RDD transformations and actions and doesn't use Dataframe as yet ( as per official documentation) , so it doesn't utilize Tungsten or Catalyst Optimizer for those.

This document below mentions all design consideration for Hive on Spark as available in official documentation -

Hive on Spark Mode Design