透視登錄蜂巢或pyspark
[英]Pivot log in hive or pyspark
問題描述
我有這種格式的許多文件日志：
[Windows user ] Pâmela
[Host name ] DV6000
[Local time ] 14:25:07
[System time ] 17:25:07
[ASCWebBrowser info] 1.1.1
[Last Write Time ] 07/19/2016 14:01
[HD Info ] Volume name: , Serial: 1713925408, File System: NTFS, Max Component Length: 255
[Network Info
[Index ] 48
[Type ] 1
[Description ] TAP-Win32 Adapter OAS #6
[Name ] {343D77F2-B3CE-414B-AE01-E248D3FC85F6}
[Ip address ] 169.254.92.162
[MAC Address ] 00-FF-34-3D-77-F2
[Gateway ] 0.0.0.0
[Mask ] 255.255.0.0
[Index ] 38
[Type ] 1
[Description ] TAP-Windows Adapter V9 #6
[Name ] {C81FC3F7-19F9-44DD-9470-4982F48A141D}
[Ip address ] 169.254.96.118
[MAC Address ] 00-FF-C8-1F-C3-F7
[Gateway ] 0.0.0.0
[Mask ] 255.255.0.0
[Index ] 36
[Type ] 1
[Description ] TAP-Win32 Adapter OAS #5
[Name ] {72115AC7-4EE2-4CB3-A8D2-
]
我需要將每一行轉換為一列。 如您所見，有一個或多個網絡信息。 這將是一個子表，其余的都是父表。 我已經通過Hive閱讀了這個日志，但我現在仍然堅持如何轉動/轉置它。
好吧，到目前為止我已嘗試過以下內容：
- Spark
DataFramePivot。 沒辦法，因為它需要聚合。 - Pandas
DataFramePivot。 它抱怨索引重復。 相同的信息可以出現在不同的日志中，因此唯一唯一的是文件名。 - Hive中的SQL CASE方法。 它不會生成線性信息。 有許多NULL。
加入。 嘗試使用自身連接，使用文件名作為連接列，但它會生成笛卡爾結果。
RowNumber是由dense_rank生成的列，而不是fname。 問題是，對於每個IP，它連接到每個描述，而不僅僅是相同的描述。 因此對於2個IP，它為每個Mask創建4行，8行，依此類推。select coalesce(hn.value, "No_Name") as hostname, d.value as decription, g.value as gateway,i.value as "index", p.value as IP, mc.value as MAC, m.value as Mask, n.value as "Name", t.value as "Type" from net_asclogs_p hn left join net_asclogs_p d on hn.fname=d.fname and d.rownumber= 1 left join net_asclogs_p g on hn.fname=g.fname and g.rownumber=2 left join net_asclogs_p i on hn.fname=i.fname and i.rownumber=4 left join net_asclogs_p p on hn.fname=p.fname and p.rownumber=5 left join net_asclogs_p mc on hn.fname=mc.fname and mc.rownumber=6 left join net_asclogs_p m on hn.fname=m.fname and m.rownumber=7 left join net_asclogs_p n on hn.fname=n.fname and n.rownumber=8 left join net_asclogs_p t on hn.fname=t.fname and t.rownumber=9 where hn.rownumber=3;試過Brickhouse的收集，但它只帶來了最后的記錄，而不是全部。
嘗試了RegexSerde，但我確定我沒有在這里得到一些東西，因為所有字段都是null：
CREATE EXTERNAL TABLE IF NOT EXISTS asclogs1 (host string, index string) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe' WITH SERDEPROPERTIES ( "input.regex" = "Host name\\\\s{2,}\\\\]\\\\s(\\\\w+)|Index\\\\s{2,}\\\\]\\\\s(\\\\w+).*", "output.format.string" = "%1$s %2$s" ) STORED AS TEXTFILE LOCATION 'hdfs:///asclogs/'
好吧，我沒有想法。 最后一種方法是用Java編寫自定義類。 還有其他選擇嗎？
1 個解決方案
解決方案1
0 2017-06-15 18:19:02
create external table log (key string,val string)
row format serde 'org.apache.hadoop.hive.serde2.RegexSerDe'
with serdeproperties ("input.regex" = "\\s*\\[(.*?)\\s*(?:\\]|$)\\s*(.*)")
;
select * from log
;
+--------------------+---------------------------------------------------------------------------------+
| key | val |
+--------------------+---------------------------------------------------------------------------------+
| Windows user | Pâmela |
| Host name | DV6000 |
| Local time | 14:25:07 |
| System time | 17:25:07 |
| ASCWebBrowser info | 1.1.1 |
| Last Write Time | 07/19/2016 14:01 |
| HD Info | Volume name: , Serial: 1713925408, File System: NTFS, Max Component Length: 255 |
| Network Info | |
| Index | 48 |
| Type | 1 |
| Description | TAP-Win32 Adapter OAS #6 |
| Name | {343D77F2-B3CE-414B-AE01-E248D3FC85F6} |
| Ip address | 169.254.92.162 |
| MAC Address | 00-FF-34-3D-77-F2 |
| Gateway | 0.0.0.0 |
| Mask | 255.255.0.0 |
| (null) | (null) |
| Index | 38 |
| Type | 1 |
| Description | TAP-Windows Adapter V9 #6 |
| Name | {C81FC3F7-19F9-44DD-9470-4982F48A141D} |
| Ip address | 169.254.96.118 |
| MAC Address | 00-FF-C8-1F-C3-F7 |
| Gateway | 0.0.0.0 |
| Mask | 255.255.0.0 |
| (null) | (null) |
| Index | 36 |
| Type | 1 |
| Description | TAP-Win32 Adapter OAS #5 |
| Name | {72115AC7-4EE2-4CB3-A8D2- |
| (null) | (null) |
+--------------------+---------------------------------------------------------------------------------+
select max (Windows_user) as Windows_user
,max (Host_name) as Host_name
,max (Local_time) as Local_time
,max (System_time) as System_time
,max (ASCWebBrowser_info) as ASCWebBrowser_info
,max (Last_Write_Time) as Last_Write_Time
,max (HD_Info) as HD_Info
,collect_list
(
case when nwi_seq > 0 then
named_struct
(
'Index' ,Index
,'Type' ,Type
,'Description' ,Description
,'Name' ,Name
,'Ip_address' ,Ip_address
,'MAC_Address' ,MAC_Address
,'Gateway' ,Gateway
,'Mask' ,Mask
)
end
) as Network_Info
from (select ifn
,log_seq
,nwi_seq
,max (case when nwi_seq = 0 and key = 'Windows user' then val end) as Windows_user
,max (case when nwi_seq = 0 and key = 'Host name' then val end) as Host_name
,max (case when nwi_seq = 0 and key = 'Local time' then val end) as Local_time
,max (case when nwi_seq = 0 and key = 'System time' then val end) as System_time
,max (case when nwi_seq = 0 and key = 'ASCWebBrowser info' then val end) as ASCWebBrowser_info
,max (case when nwi_seq = 0 and key = 'Last Write Time' then val end) as Last_Write_Time
,max (case when nwi_seq = 0 and key = 'HD Info' then val end) as HD_Info
,max (case when nwi_seq > 0 and key = 'Index' then val end) as Index
,max (case when nwi_seq > 0 and key = 'Type' then val end) as Type
,max (case when nwi_seq > 0 and key = 'Description' then val end) as Description
,max (case when nwi_seq > 0 and key = 'Name' then val end) as Name
,max (case when nwi_seq > 0 and key = 'Ip address ' then val end) as Ip_address
,max (case when nwi_seq > 0 and key = 'MAC Address' then val end) as MAC_Address
,max (case when nwi_seq > 0 and key = 'Gateway' then val end) as Gateway
,max (case when nwi_seq > 0 and key = 'Mask' then val end) as Mask
from (select key
,val
,ifn
,log_seq
,count(case when key = 'Index' then 1 end) over
(
partition by ifn,log_seq
order by boif
) as nwi_seq
from (select key
,val
,input__file__name as ifn
,block__offset__inside__file as boif
,count(case when key = 'Windows user' then 1 end) over
(
partition by input__file__name
order by block__offset__inside__file
) as log_seq
from log
) l
) l
group by ifn
,log_seq
,nwi_seq
) l
group by ifn
,log_seq
;
+--------------+-----------+------------+-------------+--------------------+-------------------+---------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| windows_user | host_name | local_time | system_time | ascwebbrowser_info | last_write_time | hd_info | network_info |
+--------------+-----------+------------+-------------+--------------------+-------------------+---------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Pâmela | DV6000 | 14:25:07 | 17:25:07 | 1.1.1 | 07/19/2016 14:01 | Volume name: , Serial: 1713925408, File System: NTFS, Max Component Length: 255 | [{"index":"48","type":"1","description":"TAP-Win32 Adapter OAS #6","name":"{343D77F2-B3CE-414B-AE01-E248D3FC85F6}","ip_address":null,"mac_address":"00-FF-34-3D-77-F2","gateway":"0.0.0.0","mask":"255.255.0.0"},{"index":"38","type":"1","description":"TAP-Windows Adapter V9 #6","name":"{C81FC3F7-19F9-44DD-9470-4982F48A141D}","ip_address":null,"mac_address":"00-FF-C8-1F-C3-F7","gateway":"0.0.0.0","mask":"255.255.0.0"},{"index":"36","type":"1","description":"TAP-Win32 Adapter OAS #5","name":"{72115AC7-4EE2-4CB3-A8D2-","ip_address":null,"mac_address":null,"gateway":null,"mask":null}] |
+--------------+-----------+------------+-------------+--------------------+-------------------+---------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Pyspark 避免 Pivot 轉換為 Dataframe - Pivot 替代
[英]Pyspark Avoid Pivot Transformation To Dataframe - Pivot Alternative
聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.