使用带有回车符的HIVE-Error在Hadoop中加载电子邮件数据

Question

I am trying to load text data in hadoop. Following are the detail

File:

"Availing Comp-off on 30th April 2014 and 2nd May 2014.eml"

Date: Tue, 29 Apr 2014 18:09:20 +0530 (IST)|

From: Rahul Tandel/AHD/TCS

To: pravita tolanavar |

Cc: upendra.bhandari@uno1.com|

Subject: Availing Comp-off on 30th April 2014 and 2nd May 2014|

Dear Pravita, I am availing comp-off on Wednesday(30th April 2014) and Friday(2nd May 2014) as i was working on weekend - Saturday(13th April 2014-2nd Shift) and Saturday(19th April 2014-1st Shift). I have informed Kamlesh regarding the same.

Thanks & Regards Rahul Tandel PS PAG PGA Mailto: rahul.tandel@uno1.com Website: http://www.uno1.com

HIVE Table schema:

" create table mail (mail_data STRUCT<DATE_M:STRING, FROM_m:STRING, TO_m:STRING,   CC_M:STRING, SUBJECT_m:STRING, DATA_M:STRING>) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' COLLECTION ITEMS TERMINATED BY '\174' LINES TERMINATED BY '\n';"

THe table is created sucessfully. But when I load data, I see the 19 rows instead of single row. I tried to find the reason, it seems hive does not have a mechanism to detect newline and break the input file.

If anyone has a better solution, please help

enter code here

Regards Upendra

Answer 1

I have the exact same situation and all I have managed to do is to replace new lines and tabulations with regexp_replace function :

regexp_replace(TEXT_HERE, '\\t|\\r|\\n', ' ')

If anyone has a better solution I would appreciate !