将列表解析为另一个函数参数 - Python

Question

I have a .log file where i want to check if there are errors/warnings in it:

    2018-03-05 10:55:54,636 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M740.aswc.arxml is well-formed
    2018-03-05 10:55:55,193 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M740.aswc.arxml is valid with the AUTOSAR4.2.2-STRICT schema
    2018-03-05 10:55:55,227 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M741.aswc.arxml is well-formed
    2018-03-05 10:55:55,795 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M741.aswc.arxml is valid with the AUTOSAR4.2.2-STRICT schema
    2018-03-05 10:55:55,831 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M742.aswc.arxml is well-formed
    2018-03-05 10:55:56,403 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M742.aswc.arxml is valid with the AUTOSAR4.2.2-STRICT schema
    2018-03-05 10:55:56,438 WARNING ASWC_M740_MSI is without connector
2018-03-05 10:55:56,438 ERROR ASWC_M741_MSI is without connector
    2018-03-05 10:55:56,438 WARNING PRP_CS_VehicleSPeed is without connector

Until now, i've managed to write the next function but without success:

def checkLog(path, level, message):
    """
    path = used for defining the file to be checked
    level = criticity level :INFO, WARNING, ERROR
    message = string to be matched
    """
    datafile = open(path)
    line_file = datafile.readline()
    while line_file != "":
        for text in message:
            if level + " " + text in line_file:
                return True
            line_file = datafile.readline()
    return False

checkLog("C:\\test\Abu\TRS.ABU.GEN.003_1\output\\result.log", "WARNING", ["PRP_CS_VehicleSPeed", "ASWC_M740_MSI", "ASWC_M741_MSI"])

Where i'm wrong?

Answer 1

The second readline() is inside the for loop that iterates over the possible messages to be matched and so the code moves on to the next line before all messages have been checked.

Try moving it to the outer scope:

def checkLog(path, level, message):
    datafile = open(path)
    line_file = datafile.readline()
    while line_file != "":
        for text in message:
            if level + " " + text in line_file:
                return True
        line_file = datafile.readline()
    return False

Your code could be better written like this:

def checkLog(path, level, message):
    with open(path) as datafile:
        for line in datafile:
            for text in message:
                if (level + " " + text) in line:
                    return True
    return False

This avoids the calls to readline() instead iterating over the file object, which simplifies the code. Also it opens the file using a context manager (the with statement) which will ensure the the file is properly closed.

Answer 2

I recommend you use pandas for this. Here is an illustrative example.

Setup

import pandas as pd, numpy as np
from io import StringIO

mystr = StringIO("""2018-03-05 10:55:54,636 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M740.aswc.arxml is well-formed
2018-03-05 10:55:55,193 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M740.aswc.arxml is valid with the AUTOSAR4.2.2-STRICT schema
2018-03-05 10:55:55,227 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M741.aswc.arxml is well-formed
2018-03-05 10:55:55,795 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M741.aswc.arxml is valid with the AUTOSAR4.2.2-STRICT schema
2018-03-05 10:55:55,831 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M742.aswc.arxml is well-formed
2018-03-05 10:55:56,403 INFO The file: C:/test/Abu/TRS.ABU.GEN.003_1/input\ASWC_M742.aswc.arxml is valid with the AUTOSAR4.2.2-STRICT schema
2018-03-05 10:55:56,438 WARNING ASWC_M740_MSI is without connector
2018-03-05 10:55:56,438 ERROR ASWC_M741_MSI is without connector
2018-03-05 10:55:56,438 WARNING PRP_CS_VehicleSPeed is without connector
""")

df = pd.read_csv(mystr, sep=',', header=None, names=['Timestamp', 'Message'])

Solution

df['Message_Error'] = df.loc[df['Message'].str.contains('WARNING|ERROR'), 'Message'].apply(lambda x: x.split(' ')[:3])
df['Message_Error'] = df['Message_Error'].apply(lambda x: x if isinstance(x, list) else [])
df = df.join(pd.DataFrame(df['Message_Error'].values.tolist()))

#              Timestamp                                            Message  \
# 0  2018-03-05 10:55:54  636 INFO The file: C:/test/Abu/TRS.ABU.GEN.003...   
# 1  2018-03-05 10:55:55  193 INFO The file: C:/test/Abu/TRS.ABU.GEN.003...   
# 2  2018-03-05 10:55:55  227 INFO The file: C:/test/Abu/TRS.ABU.GEN.003...   
# 3  2018-03-05 10:55:55  795 INFO The file: C:/test/Abu/TRS.ABU.GEN.003...   
# 4  2018-03-05 10:55:55  831 INFO The file: C:/test/Abu/TRS.ABU.GEN.003...   
# 5  2018-03-05 10:55:56  403 INFO The file: C:/test/Abu/TRS.ABU.GEN.003...   
# 6  2018-03-05 10:55:56     438 WARNING ASWC_M740_MSI is without connector   
# 7  2018-03-05 10:55:56       438 ERROR ASWC_M741_MSI is without connector   
# 8  2018-03-05 10:55:56  438 WARNING PRP_CS_VehicleSPeed is without con...   

#                          Message_Error     0        1                    2  
# 0                                   []  None     None                 None  
# 1                                   []  None     None                 None  
# 2                                   []  None     None                 None  
# 3                                   []  None     None                 None  
# 4                                   []  None     None                 None  
# 5                                   []  None     None                 None  
# 6        [438, WARNING, ASWC_M740_MSI]   438  WARNING        ASWC_M740_MSI  
# 7          [438, ERROR, ASWC_M741_MSI]   438    ERROR        ASWC_M741_MSI  
# 8  [438, WARNING, PRP_CS_VehicleSPeed]   438  WARNING  PRP_CS_VehicleSPeed 

Example query

q= {'PRP_CS_VehicleSPeed', 'ASWC_M740_MSI', 'ASWC_M741_MSI'}))
mask = (df[1] == 'WARNING') & df[2].isin(q)

df_mask = df[mask]

#              Timestamp                                            Message  \
# 6  2018-03-05 10:55:56     438 WARNING ASWC_M740_MSI is without connector   
# 8  2018-03-05 10:55:56  438 WARNING PRP_CS_VehicleSPeed is without con...   

#                          Message_Error    0        1                    2  
# 6        [438, WARNING, ASWC_M740_MSI]  438  WARNING        ASWC_M740_MSI  
# 8  [438, WARNING, PRP_CS_VehicleSPeed]  438  WARNING  PRP_CS_VehicleSPeed 

Answer 3

I think you need to unindent this line line_file = datafile.readline() . What you currently doing is checking if the first line contains first message, if not jump to second line and check if it contains second message. So it is not checking if every line contains one of those three messages.

Answer 4

You are creating a file pointer but not iterating over it, due to which you are unable to parse whole file.

I would suggest on using context guard with

with open(path, 'r') as datafile:
    all_lines = datafile.readlines()
    for line in all_lines:
        if line:
           # rest of your logic