用于字符串引用空格分隔文件的Hadoop Hive SerDe行格式

Question

I'm trying to create a Hive table for a log file with the following format.

Log file:

#Software: 1
#Version: 1
#Start-Date: xx
#Date: xx
#Fields: date time time-taken c-ip cs-username cs-auth-group x-exception-id sc-filter-result cs-categories cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id x-bluecoat-application-name x-bluecoat-application-operation
#Remark: 3215330049 "SHPROD24A" "10.0.16.162" "main"
2016-08-12 00:35:31 2 172.28.212.88 - - authentication_failed DENIED "unavailable" -  407 TCP_DENIED CONNECT - tcp psoc.ebayc3.com 443 / - - "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Firefox/45.0" 10.0.10.198 529 296 - "unavailable" "unavailable"

Note:

• The first 6 lines of every log file are comment lines, (starting with a '#')
• There are 27 fields in every line that is not a comment. Some fields are space delimited strings. Other fields are space delimited quoted strings, with spaces within the field. Ex: "str ing"

Simple space delimited files break because of this quirk of having quoted string fields. For this reason, I'm trying to use a SerDe RegEx pattern in the Row format.

HiveQL Create Table Query:

CREATE TABLE test (date_field STRING, 
time_field  STRING, 
time_taken  STRING, 
c_ip  STRING, 
cs_username  STRING, 
cs_auth_group  STRING, 
x_exception_id  STRING, 
sc_filter_result  STRING, 
cs_categories  STRING, 
csReferer  STRING, 
sc_status  STRING, 
s_action  STRING, 
cs_method  STRING, 
rsContent_Type  STRING, 
cs_uri_scheme  STRING, 
cs_host  STRING, 
cs_uri_port  STRING, 
cs_uri_path  STRING, 
cs_uri_query  STRING, 
cs_uri_extension  STRING, 
csUser_Agent  STRING, 
s_ip  STRING, 
sc_bytes  STRING, 
cs_bytes  STRING, 
x_virus_id  STRING, 
x_bluecoat_application_name  STRING, 
x_bluecoat_application_operation  STRING)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
WITH SERDEPROPERTIES (
  "input.regex" = "(\"[^\"]*\"|'[^']*'|[\S]+)+"
)
STORED AS INPUTFORMAT
   'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
TBLPROPERTIES ("skip.header.line.count"="6");

Results: Running

SELECT * FROM test LIMIT 10;

gives me this error: Failed with exception

java.io.IOException:org.apache.hadoop.hive.serde2.SerDeException: Number of matching groups doesn't match the number of columns

I'm confused because I've got 27 fields in my table (verified with DESCRIBE), and I've got 27 matches on the Regex. I've got a table property to ignore first 6 lines, so the comments shouldn't be a problem here. The error message doesn't make too much sense with this logic.

I've tested the RegEx on https://regex101.com/ with positive results. The matches break down the fields as I want them:

I've tried switching the Regex pattern for various other configurations, without any luck.

Any suggestions or hints around what could be going wrong here?

Thanks in advance!

Answer 1

我尝试使用下面的正则表达式，我可以获取26个字段，不确定哪个字段丢失

^(\S+\s\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s*(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(\S+)\s(".*")\s(\S+)\s(\S+)\s(\S+)\s(\S)\s(".*")\s(\S*)