在Spark中创建的数据之上创建Hive表

Question

I have created data in ORC format under Spark like this:

var select: String = "SELECT ..."
sqlContext.sql(select).write.format("orc").save("/tmp/out/20160101")
select = "SELECT ..."
sqlContext.sql(select).write.format("orc").save("/tmp/out/20160102")
& so on...

Now I am trying to create an external table in Hive as follows:

CREATE EXTERNAL TABLE `mydb.mytable`
 (`col1` string, 
  `col2` decimal(38,0), 
  `create_date` timestamp, 
  `update_date` timestamp)
  PARTITIONED BY (`my_date` string)
  STORED AS ORC
  LOCATION '/tmp/out/';

When I do:

"select count(*) from mydb.mytable"

I get count value as 0. But under Spark-shell, when I run:

val results = sqlContext.read.format("orc").load("/tmp/out/*/part*")
results.registerTempTable("results")
sqlContext.sql("select count(*) from results").show

I get 500,000 rows as expected.

Seems like the 'Partitions' are not getting recognized or something. How can I create an 'External' Hive table on top of data created in Spark?

Answer 1

Hive will not automatically find new partitions. You need update the hive table after creating a new partition. One the partition is created and added to the hive table, you can add and remove files within that partition as you like and these changes will be reflected immediately without needing to update the metastore.

You can use an ALTER TABLE query to create a new partition in the metastore.

ALTER TABLE mydb.mytable 
ADD PARTITION (my_date='20160101')
LOCATION '/tmp/out/20160101'

You will need to do this query for every output directory so that Hive will pick them up.

However, Hive has a standard naming convention for its partitions <column_name>=<value> . Using this naming scheme has a couple of advantages. Firstly, you can omit the LOCATION clause from the ALTER TABLE query, but it will also allow you to instead use a different query: MSCK REPAIR TABLE <table_name> which will add all directories as partitions into the metastore. This is useful if you want to add many partitions at once and means you don't need to know the values of all the partition columns you are adding.