在Spark中计算Hive表的统计信息

Question

I have created a DataFrame to load CSV files and created a temp table to get the column statistics.

However when I try to run the ANALYZE command I am facing the below error The same Analyze command ran in Hive successfully.

Spark Version : 1.6.3

df = sqlContext.read
.format("com.databricks.spark.csv")
.option("header", "true") 
.option("mode", "DROPMALFORMED")
.load("/bn_data/bopis/*.csv")

// To get the statistics of columns
df.registerTempTable("bopis")

val stat=sqlContext.sql("analyze table bopis compute statistics for columns").show()

Error:

    java.lang.RuntimeException: [1.1] failure: ``with'' expected but identifier analyze found

analyze table bopis compute statistics for columns
^

Please let us know on how to achieve the column statistics using Spark

Thanks.!

Answer 1

If you use the FOR COLUMNS option, you have to pass a list of column names, see https://docs.databricks.com/spark/latest/spark-sql/language-manual/analyze-table.html

In any case, even if you do, you are going to get an error because you can't run compute statistics on a temp table. ( you will get a Table or view 'bopis' not found in database 'default' ).

You'll have to create a full blown Hive table, either via df.write.saveAsTable("bopis_hive") , or sqlContext.sql("CREATE TABLE bopis_hive as SELECT * from bopis")