如何在 Spark Scala 中获取 Hive 表的表统计信息到 Dataframe

Question

I am working on a project on Spark scala Can I get table stats of a Hive table into a Dataframe for further calculations?

I could see info of the table using

"show table stats table_name"

But can I get that info into a dataframe. Thanks

Answer 1

I think this is enough.

val df = spark.sql("show table stats table_name")

But, my test for table stats not worked. I used an ANALYZE and DESCRIBE EXTENDED for the test.

spark.sql("analyze table table_name compute statistics")
val df = spark.sql("describe extended table_name")
df.show(false)

This gives a result.

Answer 2

In spark2, you can import hive table data into data frames in either of below methods:

Method1:

val df = spark.sql("select * from databasename.tablename")

Method2:

val df = spark.read.table("databasename.tablename")