Spark SQL real time on Hive

Question

I'm actually asking my self about performance of using Spark SQL with Hive to do real time analytics. I know that Hive has been created for batch processing, and Spark is use to do fast queries.

But, use Spark SQL with Hive will allow me to do real time queries ? Or it just will make fastest queries but not real time. Should I use an other datawarehouse instead of Hive, like Hbase ?

Thanks in advance, Florian

Answer 1

While Spark can be much faster than hive, its still probably not an ideal solution for say serving a website. So if Spark SQL can do "realtime" queries or not depends largely on what sort of timelines you consider realtime, if your dataset is small enough to cache in memory, and if your queries are able to take advantage of partitioning.