在HIVE中，分区列不属于基础保存数据的一部分吗？

Question

I have some log data, that has fields

1. id, tdate, info

I have created a dynamic partitioned table

CREATE TABLE log_partitioned(id STRING,  info STRING)
PARTITIONED BY ( tdate STRING) 

and then I am loading the data

FROM logs lg
INSERT OVERWRITE TABLE log_partitioned PARTITION(tdate)
SELECT lg.id, lg.info, lg.tdate
DISTRIBUTE BY tdate;

It is successfully loading the data by dynamic partitioning. But when I am trying to look at the data at

hdfs dfs -cat /user/hive/warehouse/log_partitioned/tdate=2000-11-05/part-r-00000

Only two columns values are there.

• id1, info1

• id2, info2 ....

If we run the hive query

select * from log_partitioned limit 10

it is showing all three columns. what should I do that hive also store the partition columns in the underlying data?

Answer 1

I'm fairly certain hive does not do this at all by default. You may be able to accomplish it with a custom SerDe and/or Input/OutputFormat, but it could be tricky. The problem is that anyone can put data in those partition folders, and if they put data containing the wrong value for that column, how would Hive reconcile that?

What is your use case for this? If you are doing the dfs -cat command you were talking about, isn't the tdate obvious from the path you're passing in? If you really want it in the output of a shell command, then something like:

dfs -cat /foo/bar/tdate=2000-11-05/part-r-00000 | sed -e 's/$/  2000-11-05/'

Another possible work-around would be to store the same data in two columns in the table. Like this:

CREATE TABLE log_partitioned(id STRING, info STRING, tdate_1 STRING)
PARTITIONED BY (tdate_2 STRING);

FROM logs lg
INSERT OVERWRITE TABLE log_partitioned PARTITION(tdate)
SELECT lg.id, lg.info, lg.tdate as tdate_1, lg.tdate as tdate_2
DISTRIBUTE BY tdate_2;