在Hive分区中编写子目录

Question

Problem statement

I have files as follows with the schema Event_Time, AD_id

file_20170102-May have records with event_time for 20170101,20170102,20170103
file_20170103-May have records with event_time for 20170102,20170103,20170104

Here event time is the time when the event occurred and the timestamp on filename is when the events were collected.So the timestamp on the filename and the event_time inside the file are out of sync.

When i write this data to hive i definitely need to write data based on event_time partition because the users are interested in queries based on event_time.

so my out put looks as follow

/path/to/output/event_time=20170102/....parquet
/path/to/output/event_time=20170103/....parquet

However i need to be able to keep track of file timestamp because some times a file gets reposted and we want to go delete already processed files based on file timestamp.

is there a way i could write this /path/to/output/event_time=20170101/20170202(file_tiemstamp)

Please note that in the above 20170102(file_timestamp) is a directory and not hive partition.

ALternately can i control the name of the parquet file so when i want to delete a file name its easy to figure out which files to delete

Answer 1

Demo

Files under /home/dmarkovitz/myfiles

myfile_1_20161204.csv

20161204,1
20161203,2

myfile_2_20161205.csv

20161203,3
20161204,4
20161205,5
20161203,6

myfile_3_20161205.csv

20161205,7
20161205,8
20161203,9

hive

create external table myfiles
(
    Event_Time  string
   ,AD_id       int
)
row format delimited
fields terminated by ','
stored as textfile
location 'file:///home/dmarkovitz/myfiles'
;

---

select  * 
       ,input__file__name

from    myfiles 
;

---

+------------+-------+-----------------------------------------------------+
| event_time | ad_id |                  input__file__name                  |
+------------+-------+-----------------------------------------------------+
|   20161204 |     1 | file:/home/dmarkovitz/myfiles/myfile_1_20161204.csv |
|   20161203 |     2 | file:/home/dmarkovitz/myfiles/myfile_1_20161204.csv |
|   20161205 |     7 | file:/home/dmarkovitz/myfiles/myfile_3_20161205.csv |
|   20161205 |     8 | file:/home/dmarkovitz/myfiles/myfile_3_20161205.csv |
|   20161203 |     9 | file:/home/dmarkovitz/myfiles/myfile_3_20161205.csv |
|   20161203 |     3 | file:/home/dmarkovitz/myfiles/myfile_2_20161205.csv |
|   20161204 |     4 | file:/home/dmarkovitz/myfiles/myfile_2_20161205.csv |
|   20161205 |     5 | file:/home/dmarkovitz/myfiles/myfile_2_20161205.csv |
|   20161203 |     6 | file:/home/dmarkovitz/myfiles/myfile_2_20161205.csv |
+------------+-------+-----------------------------------------------------+

---

create table mytable
(
    AD_id   int
)
partitioned by (file_dt date,Event_Time date)
stored as parquet
;

---

set hive.exec.dynamic.partition.mode=nonstrict;

---

insert into mytable partition (file_dt,Event_Time)

select  ad_id
       ,from_unixtime(unix_timestamp(split(input__file__name,'[_.]')[2],'yyyyMMdd'),'yyyy-MM-dd')
       ,from_unixtime(unix_timestamp(Event_Time,'yyyyMMdd'),'yyyy-MM-dd')

from    myfiles
;

---

show partitions mytable
;

---

+------------------------------------------+
|                partition                 |
+------------------------------------------+
| file_dt=2016-12-04/event_time=2016-12-03 |
| file_dt=2016-12-04/event_time=2016-12-04 |
| file_dt=2016-12-05/event_time=2016-12-03 |
| file_dt=2016-12-05/event_time=2016-12-04 |
| file_dt=2016-12-05/event_time=2016-12-05 |
+------------------------------------------+

---

select  *
       ,input__file__name 

from    mytable
;

---

+-------+------------+------------+----------------------------------------------------------------------+
| ad_id |  file_dt   | event_time |                          input__file__name                           |
+-------+------------+------------+----------------------------------------------------------------------+
|     2 | 2016-12-04 | 2016-12-03 | file:/mydb/mytable/file_dt=2016-12-04/event_time=2016-12-03/000000_0 |
|     1 | 2016-12-04 | 2016-12-04 | file:/mydb/mytable/file_dt=2016-12-04/event_time=2016-12-04/000000_0 |
|     9 | 2016-12-05 | 2016-12-03 | file:/mydb/mytable/file_dt=2016-12-05/event_time=2016-12-03/000000_0 |
|     3 | 2016-12-05 | 2016-12-03 | file:/mydb/mytable/file_dt=2016-12-05/event_time=2016-12-03/000000_0 |
|     6 | 2016-12-05 | 2016-12-03 | file:/mydb/mytable/file_dt=2016-12-05/event_time=2016-12-03/000000_0 |
|     4 | 2016-12-05 | 2016-12-04 | file:/mydb/mytable/file_dt=2016-12-05/event_time=2016-12-04/000000_0 |
|     7 | 2016-12-05 | 2016-12-05 | file:/mydb/mytable/file_dt=2016-12-05/event_time=2016-12-05/000000_0 |
|     8 | 2016-12-05 | 2016-12-05 | file:/mydb/mytable/file_dt=2016-12-05/event_time=2016-12-05/000000_0 |
|     5 | 2016-12-05 | 2016-12-05 | file:/mydb/mytable/file_dt=2016-12-05/event_time=2016-12-05/000000_0 |
+-------+------------+------------+----------------------------------------------------------------------+

---

explain dependency
select  *       
from    mytable
where   Event_Time = date '2016-12-04'        
;

{"input_tables":[{"tablename":"local_db@mytable","tabletype":"MANAGED_TABLE"}],"input_partitions":[{"partitionName":"local_db@mytable@file_dt=2016-12-04/event_time=2016-12-04"},{"partitionName":"local_db@mytable@file_dt=2016-12-05/event_time=2016-12-04"}]}

---

bash

tree mytable

---

mytable
├── file_dt=2016-12-04
│   ├── event_time=2016-12-03
│   │   └── 000000_0
│   └── event_time=2016-12-04
│       └── 000000_0
└── file_dt=2016-12-05
    ├── event_time=2016-12-03
    │   └── 000000_0
    ├── event_time=2016-12-04
    │   └── 000000_0
    └── event_time=2016-12-05
        └── 000000_0