从列表中提取时间值并添加到新列表或数组

Question

我有一个脚本，它读取包含数百个此类日志的日志文件，并查找具有“开、关或开关”类型的日志。 然后我将每个日志输出到它自己的列表中。 我试图找到一种方法将 Out 和 In 时间提取到单独的列表/数组中，然后减去两次以找到每个单独日志的持续时间。 这是输出的日志的样子：

['2020-01-31T12:04:57.976Z 1234 Out: [2020-01-31T00:30:20.150Z] Id: {"Id":"4-f-4-9-6a"', '"Type":"Switch"', '"In":"2020-01-31T00:30:20.140Z"']

这是我当前的代码：

logfile = '/path/to/my/logfile'

with open(logfile, 'r') as f:
    text = f.read()
    words = ["On", "Off", "Switch"]
    text2 = text.split('\n')
    for l in text.split('\n'):
        if (words[0] in l or words[1] in l or words[2] in l):
            log = l.split(',')[0:3]

我被困在如何仅针对日志中的 Out 和 In 时间值并将它们放入数组中并转换为时间值以找到持续时间。

脚本前的初始日志：“进入”时间之后的所有内容对于我正在寻找的内容都是无用的，所以我只输出了前三个索引

2020-01-31T12:04:57.976Z 1234 Out: [2020-01-31T00:30:20.150Z] Id: {"Id":"4-f-4-9-6a","Type":"Switch,"In":"2020-01-31T00:30:20.140Z","Path":"interface","message":"interface changed status from unknown to normal","severity":"INFORMATIONAL","display":true,"json_map":"{\"severity\":null,\"eventId\":\"65e-64d9-45-ab62-8ef98ac5e60d\",\"componentPath\":\"interface_css\",\"displayToGui\":false,\"originalState\":\"unknown\",\"closed\":false,\"eventType\":\"InterfaceStateChange\",\"time\":\"2019-04-18T07:04:32.747Z\",\"json_map\":null,\"message\":\"interface_css changed status from unknown to normal\",\"newState\":\"normal\",\"info\":\"Event created with current status\"}","closed":false,"info":"Event created with current status","originalState":"unknown","newState":"normal"}

Answer 1

下面是一个可能的解决方案。 wordmatch行有点黑客，直到我找到更清晰的东西：它只是一个单行，如果其中一个单词匹配，则创建一个空的或 1 个元素的True集。 （未经测试）

import re

logfile = '/path/to/my/logfile'

words = ["On", "Off", "Switch"]
dateformat = r'\d{4}\-\d{2}\-\d{2}T\d{2}:\d{2}:\d{2}\.\d+[Zz]?'
pattern = fr'Out:\s*\[(?P<out>{dateformat})\].*In":\s*\"(?P<in>{dateformat})\"'
regex = re.compile(pattern)
with open(logfile, 'r') as f:
    for line in f:
        wordmatch = set(filter(None, (word in s for word in words)))
        if wordmatch:
            match = regex.search(line)
            if match:
                intime = match.group('in')
                outtime = match.group('out')
                # whatever to store these strings, e.g., append to list or insert in a dict.

如前所述，您的日志示例非常笨拙，因此这适用于示例行，但可能不适用于每一行。 根据需要进行调整。

我也没有包括（如果需要）转换为datetime.datetime对象。 为此，请通读datetime模块文档，特别是datetime.strptime 。 （或者，您可能希望将结果存储在 Pandas 表中。在这种情况下，请阅读有关如何将字符串转换为实际日期时间对象的 Pandas 文档。）

您也不需要自己在换行符上阅读 nad split ： for line in f将为您做到这一点（假设f确实是一个文件句柄）。

Answer 2

正则表达式可能是要走的路（快速、高效等）......但是......

您可以采用一种非常简单（如果非常低效）的方法来清理数据：

• 将所有这些连接成一个字符串
• 替换阻碍简单解析的东西
• 明智地拆分并过滤拆分

像这样：

data = ['2020-01-31T12:04:57.976Z 1234 Out: [2020-01-31T00:30:20.150Z] Id: {"Id":"4-f-4-9-6a"', '"Type":"Switch"', '"In":"2020-01-31T00:30:20.140Z"']

all_text = " ".join(data)


# this is inefficient and will create throwaway intermediate strings - if you are
# in a hurry or operate on 100s of MB of data, this is NOT the way to go, unless
# you have time

# iterate pairs of ("bad thing", "what to replace it with") (or list of bad things)
for thing in [ (": ",":"), (list('[]{}"'),"") ]:
    whatt = thing[0]
    withh = thing[1]

    # if list, do so for each bad thing
    if isinstance(whatt, list):
        for p in whatt:
            # replace it
            all_text = all_text.replace(p,withh)
    else:
        all_text = all_text.replace(whatt,withh)

# format is now far better suited to splitting/filtering
cleaned = [a for a in all_text.split(" ") 
           if any(a.startswith(prefix) or "Switch" in a 
                  for prefix in {"In:","Switch:","Out:"})]

print(cleaned)

输出：

['Out:2020-01-31T00:30:20.150Z', 'Type:Switch', 'In:2020-01-31T00:30:20.140Z']

---

清理数据后，您的数据将如下所示：

2020-01-31T12:04:57.976Z 1234 Out:2020-01-31T00:30:20.150Z Id:Id:4-f-4-9-6a Type:Switch In:2020-01-31T00:30:20.140Z

您可以将干净列表转换为字典以便于查找：

d = dict( part.split(":",1) for part in cleaned)

print(d)

将产生：

{'In': '2020-01-31T00:30:20.140Z', 
 'Type': 'Switch', 
 'Out': '2020-01-31T00:30:20.150Z'}

您可以使用 datetime 模块从您的值中解析时间，如0 0帖子中所示。