Hive从文件名创建表分区

Question

New to Hadoop. I know how to create a table in Hive (Syntax) Creating a table with 3 Partition Key. but the keys are in File Names.

FileName Example : ServerName_ApplicationName_ApplicationName.XXXX.log.YYYY-MM-DD

there are hundreds of file in a directory want to create a table with following Partition Keys from file Name :ServerName,ApplicationName,Date and load all the files in to table Hive Script would be the preference but open to any other ideas

(files are CSV. and I know The schema(column definitions) of the file )

Answer 1

I assume the File Name is in format ServerName_ApplicationName.XXXX.log.YYYY-MM-DD (removed second "applicationname" assuming it to be a typo).

Create a table on the contents of the original file. Some thing like..

create external table default.stack
(col1 string,
 col2 string,
 col3 string,
 col4 int,
 col5 int
 )
 ROW FORMAT DELIMITED
 FIELDS terminated  by ','
 STORED AS INPUTFORMAT                                                  
  'org.apache.hadoop.mapred.TextInputFormat'                           
OUTPUTFORMAT                                                           
  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat' 
 location 'hdfs://nameservice1/location1...';

Create another partitioned table in another location like

create external table default.stack_part
(col1 string,
 col2 string,
 col3 string,
 col4 int,
 col5 int
 )
 PARTITIONED BY ( servername string, applicationname string, load_date string)
 STORED as AVRO  -- u can choose any format for the final file
 location 'hdfs://nameservice1/location2...';

Insert into partitioned table from base table using below query:

set hive.exec.dynamic.partition.mode=nonstrict;
SET hive.exec.compress.output=true;
set hive.exec.parallel=true;
SET mapred.output.compression.codec=org.apache.hadoop.io.compress.SnappyCodec;

Insert overwrite table default.stack_part
partition ( servername, applicationname, load_date)
select *, 
       split(reverse(split(reverse(INPUT__FILE__NAME),"/")[0]),"_")[0] as servername
       ,split(split(reverse(split(reverse(INPUT__FILE__NAME),"/")[0]),"_")[1],'[.]')[0] as applicationname
       ,split(split(reverse(split(reverse(INPUT__FILE__NAME),"/")[0]),"_")[1],'[.]')[3] as load_date
from default.stack;

I have tested this and it works.