Select a large number of ids from a Hive table

Question

I have a large table with format similar to

+-----+------+------+
|ID   |Cat   |date  |
+-----+------+------+
|12   | A    |201602|
|14   | B    |201601|
|19   | A    |201608|
|12   | F    |201605|
|11   | G    |201603|
+-----+------+------+

and I need to select entries based on a list with around 5000 thousand IDs. The straighforward way would be to use the list as a WHERE clause but that would have a really bad performance and probably it even would not work. How can I do this selection?

Answer 1

Using a partitioned table things run fast. Once you partitioned the table add your ids into the where. You can also extract a subtable from the original one selecting all the rows which have their ids between the min and the max of you ids list.