堆棧內存溢出
  簡體   English   中英

如何使用 python regex 解析此日志並使用 Pandas 導出到 excel（可選）？

[英]How to parse this log using python regex and export to excel with pandas (optional)?

 原文  2019-05-11 12:59:58  5   1    python/ regex/ pandas/ re

問題描述

我有以下格式的日志文件。 對於每一行，我需要捕獲第三列，例如0102b69880c4b330 、相應的消息DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG及其各自的計數（請參閱輸出）。 我認為使用正則表達式可以讓我更輕松地解決問題。

說明：

案例 1 ：ID 0102b69880c4b330 3 次（第 1、2、3 行）。 所以ID的計數為3，對應的消息DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG也出現了3次，所以計數為3。

案例 2 ：現在第 4 行和第 5 行的 ID 0102b69880c4e3b2有兩條不同的消息JMS DO_METHOD TRACE LAUNCH, DO_METHOD TRACE LAUNCH ，ID 計數為 2，但它們的消息計數應分別為 1、1。

情況 3 ：第 10 行到最后一行中的 ID 0102b6988000000c具有消息DM_WORKFLOW_E_PROCESS_AUTO_TASK 。 ID 計數為 3，消息計數為 3。但這里我需要獲取此錯誤消息旁邊的流程任務 id 和工作流 id。

我在輸出中使用了[Ignore for this]來解釋我不需要 ID。

最后，我還需要維護DM_WORKFLOW_E_PROCESS_AUTO_TASK的總數。

Input:
2019-05-05T00:05:11.507245  12090[12090]    0102b69880c4b330    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info: Attempting to status Index Agent Instance host-address_9200_IndexAgent
2019-05-05T00:05:11.759829  12090[12090]    0102b69880c4b330    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info : Response from HTTP_POST command: HTTP/1.1 200 OK Status: 0 , Time Taken: 0 seconds.
2019-05-05T00:05:11.759898  12090[12090]    0102b69880c4b330    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info : HTTP_POST with args -command status -docbase SubWayX -user dm_fulltext_index_user -ticket ****** -instance host-address_9200_IndexAgent -details false to Index Agent host-address_9200_IndexAgent is successful.
2019-05-05T01:40:53.148751  20135[20135]    0102b69880c4e3b2    JMS DO_METHOD TRACE LAUNCH: do_method launch: successful: user: Xie Xiaoke, session id: 0102b69880c4e3b2, JMS id: 0802b69880003535, method: D2LifecycleChangeStateMethod, host:host-address.net, port:9082, path:/DmMethods/servlet/DoMethod 
2019-05-05T01:40:53.148877  20135[20135]    0102b69880c4e3b2    DO_METHOD TRACE LAUNCH: method launch: successful, user: Xie Xiaoke, session id: 0102b69880c4e3b2, method: D2LifecycleChangeStateMethod
2019-05-07T05:42:21.171087  22484[22484]    0102b6988000000b    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  "Workflow Agent failed to process task 4a02b698800aad04 of workflow 4d02b6988000f709. The task is using method 'D2WFLifeCycleMethod'. Activity: 'Demote to Draft with new Version'. Check the Java Method Server log for errors."
2019-05-05T05:24:48.483966  17114[17114]    0102b69880c4fb1e    JMS DO_METHOD TRACE LAUNCH: user: dmadmin, session id: 0102b69880c4fb1e, JMS id: 0802b69880003535, method: D2LifecycleChangeStateMethod, host:host-address.net, port:9082, path:/DmMethods/servlet/DoMethod, arguments:-method_verb com.emc.d2.api.methods.D2Method -class_name com.emc.d2.api.methods.D2LifecycleChangeStateMethod -__dm_docbase__ SubWayX -__dm_server_config__ host-address_SubWayX -docbase_name SubWayX -user_name dmadmin -method_return_id "0802b6988167b46e" -locale en
2019-05-05T05:24:50.362650  17114[17114]    0102b69880c4fb1e    JMS DO_METHOD TRACE LAUNCH: do_method launch: successful: user: dmadmin, session id: 0102b69880c4fb1e, JMS id: 0802b69880003535, method: D2LifecycleChangeStateMethod, host:host-address.net, port:9082, path:/DmMethods/servlet/DoMethod 
2019-05-05T05:24:50.362702  17114[17114]    0102b69880c4fb1e    DO_METHOD TRACE LAUNCH: method launch: successful, user: dmadmin, session id: 0102b69880c4fb1e, method: D2LifecycleChangeStateMethod
2019-05-05T05:44:35.410674  12791[12791]    0102b6988000000c    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  "Workflow Agent failed to process task 4a02b698800a977c of workflow 4d02b698800107e9. The task is using method 'D2WFLifeCycleMethod'. Activity: 'validate entry conditions for Effective'. Method timed out within 60 secs."
2019-05-05T05:50:31.383668  12791[12791]    0102b6988000000c    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  "Workflow Agent failed to process task 4a02b698800a9782 of workflow 4d02b6988001081e. The task is using method 'D2WFLifeCycleMethod'. Activity: 'validate entry conditions for Effective'. Method timed out within 60 secs."
2019-05-05T05:53:49.978053  12791[12791]    0102b6988000000c    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  "Workflow Agent failed to process task 4a02b698800a9784 of workflow 4d02b6988001081c. The task is using method 'D2WFLifeCycleMethod'. Activity: 'validate entry conditions for Effective'. Method timed out within 60 secs."
2019-05-05T00:50:11.761273  2591[2591]  0102b69880c4ccde    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info: Attempting to status Index Agent Instance phchbs-sp220333_9200_IndexAgent
2019-05-05T00:50:12.015521  2591[2591]  0102b69880c4ccde    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info : Response from HTTP_POST command: HTTP/1.1 200 OK Status: 0 , Time Taken: 1 seconds.
2019-05-05T00:50:12.015563  2591[2591]  0102b69880c4ccde    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info : HTTP_POST with args -command status -docbase SubWayX -user dm_fulltext_index_user -ticket ****** -instance phchbs-sp220333_9200_IndexAgent -details false to Index Agent phchbs-sp220333_9200_IndexAgent is successful.


I need to get the below output:

Output:
ID:                 Count:          Message:                                            Corresponding Message Count         Task ID:                                                Workflow ID
0102b69880c4b330     3              DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG                  3                                   [Ignore for this]                                       [Ignore for this]
0102b69880c4e3b2     2              JMS DO_METHOD TRACE LAUNCH, DO_METHOD TRACE LAUNCH  1, 1                                [Ignore for this]                                       [Ignore for this]
0102b6988000000b     1              DM_WORKFLOW_E_PROCESS_AUTO_TASK                     1                                   4a02b698800aad04                                        4d02b6988000f709
0102b69880c4fb1e     3              JMS DO_METHOD TRACE LAUNCH, DO_METHOD TRACE LAUNCH  2, 1                                [Ignore for this]                                       [Ignore for this]
0102b6988000000c     3              DM_WORKFLOW_E_PROCESS_AUTO_TASK                     3                                   4a02b698800a977c, 4a02b698800a9782, 4a02b698800a9784    4d02b698800107e9, 4d02b6988001081e, 4d02b6988001081c

我嘗試過的測試程序如下。 我在 ID 列之后沒有正確使用正則表達式，我只是選擇了包含 [] 中的值的值，但它跳過了不包含的值。 它也不會選擇流程任務 ID 和工作流 ID。 你能幫我修改我的代碼以獲得正確的計數、任務 ID 和工作流 ID 嗎？

import re
import collections

regexp = re.compile(
        r'(?P<date>[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}\.[0-9]{6}\s*)'+
        '(?P<un_num>[0-9]{3,5}\[[0-9]{3,5}\]\s*)'+
        '(?P<id>[a-z0-9]{16}\s*)'+
        '(?P<message>\[(.*?)\])'
        )
ls = ["2019-05-05T00:05:11.507245   12090[12090]    0102b69880c4b330    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info: Attempting to status Index Agent Instance host-address_9200_IndexAgent",
      "2019-05-05T00:05:11.759829   12090[12090]    0102b69880c4b330    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info : Response from HTTP_POST command: HTTP/1.1 200 OK Status: 0 , Time Taken: 0 seconds.",
      "2019-05-05T00:05:11.759898   12090[12090]    0102b69880c4b330    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info : HTTP_POST with args -command status -docbase SubWayX -user dm_fulltext_index_user -ticket ****** -instance host-address_9200_IndexAgent -details false to Index Agent host-address_9200_IndexAgent is successful.",
      "2019-05-05T01:40:53.148751   20135[20135]    0102b69880c4e3b2    JMS DO_METHOD TRACE LAUNCH: do_method launch: successful: user: Xie Xiaoke, session id: 0102b69880c4e3b2, JMS id: 0802b69880003535, method: D2LifecycleChangeStateMethod, host:host-address.net, port:9082, path:/DmMethods/servlet/DoMethod",
      "2019-05-05T01:40:53.148877   20135[20135]    0102b69880c4e3b2    DO_METHOD TRACE LAUNCH: method launch: successful, user: Xie Xiaoke, session id: 0102b69880c4e3b2, method: D2LifecycleChangeStateMethod",
      "2019-05-07T05:42:21.171087   22484[22484]    0102b6988000000b    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  'Workflow Agent failed to process task 4a02b698800aad04 of workflow 4d02b6988000f709. The task is using method 'D2WFLifeCycleMethod'. Activity: 'Demote to Draft with new Version'. Check the Java Method Server log for errors.'",
      "2019-05-05T05:44:35.410674   12791[12791]    0102b6988000000c    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  'Workflow Agent failed to process task 4a02b698800a977c of workflow 4d02b698800107e9. The task is using method 'D2WFLifeCycleMethod'. Activity: 'validate entry conditions for Effective'. Method timed out within 60 secs.'",
      "2019-05-05T05:50:31.383668   12791[12791]    0102b6988000000c    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  'Workflow Agent failed to process task 4a02b698800a9782 of workflow 4d02b6988001081e. The task is using method 'D2WFLifeCycleMethod'. Activity: 'validate entry conditions for Effective'. Method timed out within 60 secs.'",
      "2019-05-05T05:53:49.978053   12791[12791]    0102b6988000000c    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  'Workflow Agent failed to process task 4a02b698800a9784 of workflow 4d02b6988001081c. The task is using method 'D2WFLifeCycleMethod'. Activity: 'validate entry conditions for Effective'. Method timed out within 60 secs.'"
      ]

id_counter = collections.Counter()
message_counter = collections.Counter()

print("started......!!!!!")
for i in range(len(ls)):
    x = regexp.match(ls[i])
    y = re.search(regexp, ls[i])
    if x is None or y is None:
        print("None")
        continue
    print("-----------------")
    print(y.group('date'))
    print(y.group('un_num'))
    print(y.group('id'))
    id_counter.update([y.group('id')])
    print(y.group('message'))
    message_counter.update([y.group('message')])

print("end....!!!")

print(id_counter)
print(message_counter)

def print_counts(cdict):
    for key, values in enumerate(cdict.items()):
        print(key, values)

print_counts(id_counter)
print_counts(message_counter)

這個輸出是：

started......!!!!!
-----------------
2019-05-05T00:05:11.507245      
12090[12090]    
0102b69880c4b330        
[DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG]
-----------------
2019-05-05T00:05:11.759829      
12090[12090]    
0102b69880c4b330        
[DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG]
-----------------
2019-05-05T00:05:11.759898      
12090[12090]    
0102b69880c4b330        
[DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG]
None
None
-----------------
2019-05-07T05:42:21.171087      
22484[22484]    
0102b6988000000b    
[DM_WORKFLOW_E_PROCESS_AUTO_TASK]
-----------------
2019-05-05T05:44:35.410674      
12791[12791]    
0102b6988000000c        
[DM_WORKFLOW_E_PROCESS_AUTO_TASK]
-----------------
2019-05-05T05:50:31.383668      
12791[12791]    
0102b6988000000c        
[DM_WORKFLOW_E_PROCESS_AUTO_TASK]
-----------------
2019-05-05T05:53:49.978053      
12791[12791]    
0102b6988000000c        
[DM_WORKFLOW_E_PROCESS_AUTO_TASK]
end....!!!
Counter({'0102b69880c4b330\t': 3, '0102b6988000000c\t': 3, '0102b6988000000b    ': 1})
Counter({'[DM_WORKFLOW_E_PROCESS_AUTO_TASK]': 4, '[DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG]': 3})
0 ('0102b69880c4b330\t', 3)
1 ('0102b6988000000b    ', 1)
2 ('0102b6988000000c\t', 3)
0 ('[DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG]', 3)
1 ('[DM_WORKFLOW_E_PROCESS_AUTO_TASK]', 4)

1 個解決方案

解決方案1
 1 已采納  2019-05-11 14:52:38

從輸入數據作為文本開始：

txt = """
2019-05-05T00:05:11.507245  12090[12090]    0102b69880c4b330    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info: Attempting to status Index Agent Instance host-address_9200_IndexAgent
2019-05-05T00:05:11.759829  12090[12090]    0102b69880c4b330    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info : Response from HTTP_POST command: HTTP/1.1 200 OK Status: 0 , Time Taken: 0 seconds.
2019-05-05T00:05:11.759898  12090[12090]    0102b69880c4b330    [DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG] info : HTTP_POST with args -command status -docbase SubWayX -user dm_fulltext_index_user -ticket ****** -instance host-address_9200_IndexAgent -details false to Index Agent host-address_9200_IndexAgent is successful.
2019-05-05T01:40:53.148751  20135[20135]    0102b69880c4e3b2    JMS DO_METHOD TRACE LAUNCH: do_method launch: successful: user: Xie Xiaoke, session id: 0102b69880c4e3b2, JMS id: 0802b69880003535, method: D2LifecycleChangeStateMethod, host:host-address.net, port:9082, path:/DmMethods/servlet/DoMethod 
2019-05-05T01:40:53.148877  20135[20135]    0102b69880c4e3b2    DO_METHOD TRACE LAUNCH: method launch: successful, user: Xie Xiaoke, session id: 0102b69880c4e3b2, method: D2LifecycleChangeStateMethod
2019-05-07T05:42:21.171087  22484[22484]    0102b6988000000b    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  "Workflow Agent failed to process task 4a02b698800aad04 of workflow 4d02b6988000f709. The task is using method 'D2WFLifeCycleMethod'. Activity: 'Demote to Draft with new Version'. Check the Java Method Server log for errors."
2019-05-05T05:24:48.483966  17114[17114]    0102b69880c4fb1e    JMS DO_METHOD TRACE LAUNCH: user: dmadmin, session id: 0102b69880c4fb1e, JMS id: 0802b69880003535, method: D2LifecycleChangeStateMethod, host:host-address.net, port:9082, path:/DmMethods/servlet/DoMethod, arguments:-method_verb com.emc.d2.api.methods.D2Method -class_name com.emc.d2.api.methods.D2LifecycleChangeStateMethod -__dm_docbase__ SubWayX -__dm_server_config__ host-address_SubWayX -docbase_name SubWayX -user_name dmadmin -method_return_id "0802b6988167b46e" -locale en
2019-05-05T05:24:50.362650  17114[17114]    0102b69880c4fb1e    JMS DO_METHOD TRACE LAUNCH: do_method launch: successful: user: dmadmin, session id: 0102b69880c4fb1e, JMS id: 0802b69880003535, method: D2LifecycleChangeStateMethod, host:host-address.net, port:9082, path:/DmMethods/servlet/DoMethod 
2019-05-05T05:24:50.362702  17114[17114]    0102b69880c4fb1e    DO_METHOD TRACE LAUNCH: method launch: successful, user: dmadmin, session id: 0102b69880c4fb1e, method: D2LifecycleChangeStateMethod
2019-05-05T05:44:35.410674  12791[12791]    0102b6988000000c    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  "Workflow Agent failed to process task 4a02b698800a977c of workflow 4d02b698800107e9. The task is using method 'D2WFLifeCycleMethod'. Activity: 'validate entry conditions for Effective'. Method timed out within 60 secs."
2019-05-05T05:50:31.383668  12791[12791]    0102b6988000000c    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  "Workflow Agent failed to process task 4a02b698800a9782 of workflow 4d02b6988001081e. The task is using method 'D2WFLifeCycleMethod'. Activity: 'validate entry conditions for Effective'. Method timed out within 60 secs."
2019-05-05T05:53:49.978053  12791[12791]    0102b6988000000c    [DM_WORKFLOW_E_PROCESS_AUTO_TASK]error:  "Workflow Agent failed to process task 4a02b698800a9784 of workflow 4d02b6988001081c. The task is using method 'D2WFLifeCycleMethod'. Activity: 'validate entry conditions for Effective'. Method timed out within 60 secs."
"""

我們可以做一些預處理，先拆分成行，丟棄空行：

lines = [line for line in txt.split('\n') if line.strip()]

然后提取我們感興趣的塊，但只是對數據進行粗略（且非常快速）的拆分

parts = [(line[44:60], line[64:].split(':', 1))  for line in lines]

更新：由於您的新數據不是固定寬度，我們需要一些其他的預處理方式，例如：

# parts = [(line[44:60], line[64:].split(':', 1))  for line in lines]
import re
lines = [re.sub(r'\s+', ' ', line) for line in lines]   # squash all multiple spaces to a single space
parts = [line.split() for line in lines]  # split on whitespace
parts = [(line[2], ' '.join(line[3:]).split(':', 1)) for line in parts]  # this is similar to the original line

請記住，這部分只是為了讓下面的 InputData 類中的最終處理更容易。

然后我們為我們感興趣的輸入數據創建一個數據結構，它可以將我們擁有的預處理數據分成幾部分：

class InputData(object):
    def __init__(self, idtag, (msg, details)):  # py3 is more awkward here (*)
        self.idtag = idtag
        self.error_task = None
        self.error_workflow = None
        msg = msg.strip()
        if msg.endswith('] info'):
            self.msg = msg[1:-len('] info')]
        elif msg.endswith('error'):
            self.msg = msg[1:-len(']error')]
            self.error_task = details.split(' task ', 1)[1].split(' ', 1)[0]
            self.error_workflow = details.split(' workflow ', 1)[1].split('.', 1)[0]
        else:
            self.msg = msg

    def __repr__(self):
        return repr(self.__dict__)  # this is a great trick for making debugging easier

(*) 對於 py3 你需要（不知道他們為什么改變這個......？）

def __init__(self, idtag, tmp):
    msg, details = tmp

現在我們可以將這個類應用於預處理的輸入：

input_data = [InputData(*part) for part in parts]

如果我們打印出到目前為止我們所擁有的：

for d in input_data:
    print d

結果是：

{'error_workflow': None, 'error_task': None, 'idtag': '0102b69880c4b330', 'msg': 'DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG'}
{'error_workflow': None, 'error_task': None, 'idtag': '0102b69880c4b330', 'msg': 'DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG'}
{'error_workflow': None, 'error_task': None, 'idtag': '0102b69880c4b330', 'msg': 'DM_FT_INDEX_T_INIT_INDEX_AGENT_MSG'}
{'error_workflow': None, 'error_task': None, 'idtag': '0102b69880c4e3b2', 'msg': 'JMS DO_METHOD TRACE LAUNCH'}
{'error_workflow': None, 'error_task': None, 'idtag': '0102b69880c4e3b2', 'msg': 'DO_METHOD TRACE LAUNCH'}
{'error_workflow': '4d02b6988000f709', 'error_task': '4a02b698800aad04', 'idtag': '0102b6988000000b', 'msg': 'DM_WORKFLOW_E_PROCESS_AUTO_TASK'}
...

現在我們創建一個類來表示我們想要的輸出數據：

from collections import defaultdict

class OutputData(object):
    def __init__(self):   # I'm using this class in a defaultdict, so the __init__ method can't take any arguments
        self.idtag = None
        self.idtag_count = 0
        self.messages = defaultdict(int)
        self.errors = []
        self.workflows = []

    def add(self, indata):
        "Adds indata to this object."
        self.idtag = indata.idtag
        self.idtag_count += 1
        self.messages[indata.msg] += 1        
        if indata.error_task:
            self.errors.append(indata.error_task)
            self.workflows.append(indata.error_workflow)

並將輸入數據輸入其中：

output_data = defaultdict(OutputData)

for indata in input_data:
    output_data[indata.idtag].add(indata)

最后，我們可以輸出所需格式的輸出數據：

fmt = '%-20s %-6s %-55s %-15s %-60s %s'

print fmt % ('ID:', 'Count:', 'Message:', 'msg counts', 'taskid', 'workflowid')
for outdata in output_data.values():
    print fmt % (
        outdata.idtag,
        outdata.idtag_count,
        ', '.join(outdata.messages.keys()),
        ', '.join(str(outdata.messages[k]) for k in outdata.messages.keys()),
        ', '.join(outdata.errors),
        ', '.join(outdata.workflows)
    )

這種類型的結構，即：預處理文本，提取有趣的輸入數據，將輸入數據轉換為輸出數據，最后對輸出數據進行序列化/格式化； 適用於所有此類問題，並且可以更輕松地在將來進行調試和修改。

暫無
暫無

聲明:本站的技術帖子網頁，遵循CC BY-SA 4.0協議，如果您需要轉載，請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.

相關問題 Python使用正則表達式解析日志 使用Regex在Python中解析日志文本 如何使用python argparse解析可選參數？ 如何使用 Pandas 將字典導出到 excel 如何使用Pandas導出QTableWidget數據到Excel？ 如何使用 Python 字符串方法或正則表達式解析復雜的文本文件並導出為表格形式 使用 python pandas 解析日志文件 如何使用Python解析WordPress CSV導出 如何使用Pandas將python Web抓取數據導出到現有excel文件中的特定工作表？ Python：Excel 文件 (xlsx) 以變量為文件路徑導出，使用 Pandas
相關標簽
 
粵ICP備18138465號  © 2020-2024 STACKOOM.COM