如何從日志文件中提取唯一事件並對其進行計數？

Question

我有一個shoutcast連接日志文件，想要找出使用的客戶端和頻率。 日志文件非常龐大（大約100mb），包含過去3年的條目。 日志條目看起來像這樣（IP已被隨機化！）：

<03/23/13@15:46:25> [dest: 1.187.2.99] starting stream (UID: 25477)[L: 2]{A: Internet%20Explorer%207}(P: 1)
<03/23/13@15:46:34> [dest: 1.187.2.99] connection closed (9 seconds) (UID: 25477)[L: 1]{Bytes: 403705}(P: 1)
<03/23/13@16:24:36> [dest: 1.194.2.16] starting stream (UID: 25478)[L: 2]{A: WMPlayer/10.0.0.364}(P: 1)
<03/23/13@16:40:56> [dest: 1.194.2.16] connection closed (981 seconds) (UID: 25478)[L: 1]{Bytes: 15938209}(P: 1)
<03/23/13@16:41:29> [dest: 1.158.2.39] starting stream (UID: 25479)[L: 2]{A: WinampMPEG/5.50}(P: 1)
<03/23/13@16:41:40> [dest: 1.158.2.39] connection closed (11 seconds) (UID: 25479)[L: 1]{Bytes: 432719}(P: 1)
<03/23/13@17:51:29> [dest: 1.142.2.225] starting stream (UID: 25480)[L: 2]{A: WinampMPEG/5.50}(P: 1)
<03/23/13@18:07:48> [dest: 1.142.2.225] connection closed (979 seconds) (UID: 25480)[L: 1]{Bytes: 15919475}(P: 1)
<03/23/13@18:18:48> [dest: 1.232.2.215] starting stream (UID: 25481)[L: 2]{A: TapinRadio}(P: 1)
<03/23/13@18:19:07> [dest: 1.232.2.215] connection closed (19 seconds) (UID: 25481)[L: 1]{Bytes: 417192}(P: 1)
<03/23/13@18:34:45> [dest: 1.187.2.99] starting stream (UID: 25482)[L: 2]{A: Internet%20Explorer%207}(P: 1)
<03/23/13@18:34:46> [dest: 1.187.2.99] connection closed (2 seconds) (UID: 25482)[L: 1]{Bytes: 282751}(P: 1)

我想提取每個獨特的客戶端，並計算這種客戶端的使用頻率。 對於上面的日志，結果應如下所示：

Internet%20Explorer%207   2
WMPlayer/10.0.0.364       1
WinampMPEG/5.50           2
TapinRadio                1

首先，我簡單地過濾了所有無用的條目。 （抱歉用貓 。）

cat shoutcast.log | grep "starting stream" > filtered.txt

結果如下：

<03/23/13@15:46:25> [dest: 1.187.2.99] starting stream (UID: 25477)[L: 2]{A: Internet%20Explorer%207}(P: 1)
<03/23/13@16:24:36> [dest: 1.194.2.16] starting stream (UID: 25478)[L: 2]{A: WMPlayer/10.0.0.364}(P: 1)
<03/23/13@16:41:29> [dest: 1.158.2.39] starting stream (UID: 25479)[L: 2]{A: WinampMPEG/5.50}(P: 1)
<03/23/13@17:51:29> [dest: 1.142.2.225] starting stream (UID: 25480)[L: 2]{A: WinampMPEG/5.50}(P: 1)
<03/23/13@18:18:48> [dest: 1.232.2.215] starting stream (UID: 25481)[L: 2]{A: TapinRadio}(P: 1)
<03/23/13@18:34:45> [dest: 1.187.2.99] starting stream (UID: 25482)[L: 2]{A: Internet%20Explorer%207}(P: 1)

但現在呢？ 我有點迷失，如何訪問{A: }括號中的信息？

Answer 1

嘗試這個awk行：

 awk -F'{A: |}' '/starting/{a[$2]++}END{for(x in a)print x" : "a[x]}' input

測試您的數據：

kent$  cat ff
<03/23/13@15:46:25> [dest: 1.187.2.99] starting stream (UID: 25477)[L: 2]{A: Internet%20Explorer%207}(P: 1)
<03/23/13@15:46:34> [dest: 1.187.2.99] connection closed (9 seconds) (UID: 25477)[L: 1]{Bytes: 403705}(P: 1)
<03/23/13@16:24:36> [dest: 1.194.2.16] starting stream (UID: 25478)[L: 2]{A: WMPlayer/10.0.0.364}(P: 1)
<03/23/13@16:40:56> [dest: 1.194.2.16] connection closed (981 seconds) (UID: 25478)[L: 1]{Bytes: 15938209}(P: 1)
<03/23/13@16:41:29> [dest: 1.158.2.39] starting stream (UID: 25479)[L: 2]{A: WinampMPEG/5.50}(P: 1)
<03/23/13@16:41:40> [dest: 1.158.2.39] connection closed (11 seconds) (UID: 25479)[L: 1]{Bytes: 432719}(P: 1)
<03/23/13@17:51:29> [dest: 1.142.2.225] starting stream (UID: 25480)[L: 2]{A: WinampMPEG/5.50}(P: 1)
<03/23/13@18:07:48> [dest: 1.142.2.225] connection closed (979 seconds) (UID: 25480)[L: 1]{Bytes: 15919475}(P: 1)
<03/23/13@18:18:48> [dest: 1.232.2.215] starting stream (UID: 25481)[L: 2]{A: TapinRadio}(P: 1)
<03/23/13@18:19:07> [dest: 1.232.2.215] connection closed (19 seconds) (UID: 25481)[L: 1]{Bytes: 417192}(P: 1)
<03/23/13@18:34:45> [dest: 1.187.2.99] starting stream (UID: 25482)[L: 2]{A: Internet%20Explorer%207}(P: 1)
<03/23/13@18:34:46> [dest: 1.187.2.99] connection closed (2 seconds) (UID: 25482)[L: 1]{Bytes: 282751}(P: 1)

kent$  awk -F'{A: |}' '/starting/{a[$2]++}END{for(x in a)print x" : "a[x]}' ff
WMPlayer/10.0.0.364 : 1
TapinRadio : 1
WinampMPEG/5.50 : 2
Internet%20Explorer%207 : 2