在嵌套子查询中显示配置单元分区

Question

I have a hive table that is partitioned by day (eg 20151001, 20151002,....). Is there a hive query to list these partitions in a way that it is possible to be used in a nested sub query?

That is can I do something along the line of:

SELECT * FROM (SHOW PARTITIONS test) a where ...

The query-

SELECT ptn FROM test 

returns as many rows as the number of rows in the test table. I want it to return only as many rows as the number of partitions (without using the DISTINCT function)

Answer 1

A potential solution is to find the partitions from the hdfs location for the table of interest by using either shell script/python.

The data that corresponds to the hive table is stored in the hdfs eg

/hive/database/table/partition/datafiles

in your case, /hive/database/table/20151001/datafiles

If the table is bucketed there are as many individual files as the cluster size.

Once you have the above path, create a shell script to loop through the folder (in this case 20151001 etc..)

capture this in a shell variable and pass it as a parameter to the hive query.