[英]Creating structured hive table with unstructured GPS packets in csv format
我有一個如下所述的csv文件。
VTS,51,0071,9739965515,NM,GP,INF01,V,19,072219,291014,0000.0000,N,00000.0000,E,07AE VTS,01,0097,9739965515,SP,GP,18,072253,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,169,B205 VTS,51,0071,9739965515,NM,GP,INF01,V,18,072311,291014,0000.0000,N,00000.0000,E,C24E VTS,01,0097,9739965515,NM,GP,19,072311,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,171,B358 VTS,51,0071,9739965515,NM,GP,INF01,V,18,072319,291014,0000.0000,N,00000.0000,E,012F VTS,51,0071,9739965515,NM,GP,INF01,V,19,072326,291014,0000.0000,N,00000.0000,E,B2E6 VTS,01,0097,9739965515,NM,GP,18,072326,V,0000.0000,N,00000.0000,E,0.0,0.0,291014,0000,00,4000,11,999,173,EAA0 VTS,51,0071,9739965515,NM,GP,INF01,V,18,072333,291014,0000.0000,N,00000.0000,E,9896 VTS,51,0071,9739965515,NM,GP,INF01,V,18,072340,291014,0000.0000,N,00000.0000,E,9B23
這必須與字段映射:
pkt_header,gprs_pkt_id,pkt_length,sim_no,msg_id,gprs_pkt,gsm_sig_strength,utc_time,pkt_validation,latitude,direction_n_s,longitude,direction_e_w,speed,track_angle,utc_date,fuel_adc_values,ignition,odometer_values,supply_int,battery_adc,pkt_id,check_sum
第二個字段,即值為01的gprs_pkt_id描述了一個有效的數據包。 我的用例是僅針對有效數據包過濾CSV數據,而我正在使用正則表達式,但是我無法獲取全部數據。 任何幫助將不勝感激。
使用的Hive查詢如下所示。
CREATE EXTERNAL TABlE sky_track_testing1( pkt_header STRING, gprs_pkt_id STRING, pkt_length STRING, sim_no STRING, msg_id STRING, gprs_pkt STRING, gsm_sig_strength STRING, utc_time STRING, pkt_validation STRING, latitude STRING, direction_n_s STRING, longitude STRING, direction_e_w STRING, speed STRING, track_angle STRING, utc_date STRING, fuel_adc_values STRING, ignition STRING, odometer_values STRING, supply_int STRING, battery_adc STRING, pkt_id STRING, check_sum STRING ) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe' WITH SERDEPROPERTIES ( "input.regex" = "^(VTS,01).*$" ) STORED AS TEXTFILE LOCATION '/user/root/sky_track';
這絕對是錯誤的查詢。 請幫我。
我建議您為此使用Pig :
a = load '/user/root/sky_track' as (pkt_header,gprs_pkt_id,pkt_length,sim_no,msg_id,gprs_pkt,gsm_sig_strength,utc_time,pkt_validation,latitude,direction_n_s,longitude,direction_e_w,speed,track_angle,utc_date,fuel_adc_values,ignition,odometer_values,supply_int,battery_adc,pkt_id,check_sum);
b = filter a by gprs_pkt_id == '01';
store b into '/user/root/sky_track_valid';
是的,根據以上答案,Pig非常適合您的數據。 你可以試試豬。 如果您對配置單元感興趣,請參見以下示例(數據集不需要正則表達式)。
hive> CREATE TABLE sky_track_testing1(
> pkt_header STRING,
> gprs_pkt_id STRING,
> pkt_length STRING,
> sim_no STRING,
> msg_id STRING,
> gprs_pkt STRING,
> gsm_sig_strength STRING,
> utc_time STRING,
> pkt_validation STRING,
> latitude STRING,
> direction_n_s STRING,
> longitude STRING,
> direction_e_w STRING,
> speed STRING,
> track_angle STRING,
> utc_date STRING,
> fuel_adc_values STRING,
> ignition STRING,
> odometer_values STRING,
> supply_int STRING,
> battery_adc STRING,
> pkt_id STRING,
> check_sum STRING
> )
> ROW FORMAT
> DELIMITED FIELDS TERMINATED BY ','
> LINES TERMINATED BY '\n'
> STORED AS TEXTFILE;
OK
Time taken: 0.1 seconds
hive> select *from sky_track_testing1 where gprs_pkt_id='01';
OK
VTS 01 0097 9739965515 SP GP 18 072253 V 0000.0000 N 00000.0000 E 0.0 0.0 291014 0000 00 4000 1999 169 B205
VTS 01 0097 9739965515 NM GP 19 072311 V 0000.0000 N 00000.0000 E 0.0 0.0 291014 0000 00 4000 1999 171 B358
VTS 01 0097 9739965515 NM GP 18 072326 V 0000.0000 N 00000.0000 E 0.0 0.0 291014 0000 00 4000 1999 173 EAA0
Time taken: 14.328 seconds
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.