将文本文件解析为表格数据以进行处理

Question

当前的问题是使用python以表格形式解析特定数据。一小部分数据如下所示

Statistics indicator:0x222235

number of records = 3 

records[0]

value one = 2

value two = 5

blocks = 2

block[0] {

some irrelevant data....

value three = 4 bytes

}

block[1]{

some irrelevant data...

value three = 6 bytes

}

records[1]

value one = 3

value two = 5

blocks = 1

block[0] {

some irrelevant data....

value three = 4 bytes

}

records[2]

value one = 7

value two = 6

blocks = 2

block[0] {

some irrelevant data....

value three = 3 bytes

}

block[1]{

some irrelevant data...

value three = 4 bytes

}

Statistics indicator:0x135256

number of records = 2 

records[0]

value one = 4

value two = 8

blocks = 1

block[0] {

some irrelevant data....

value three = 6 bytes

}

records[1]

value one = 3

value two = 5

blocks = 1

block[0] {

some irrelevant data....

value three = 3 bytes

}

如图所示，数据具有特定的模式。 它在每个特定数据块的开头都有统计指示器。它具有记录数字段以指示数据块具有的记录数。在每个记录中，``值一''和值二``是不同的。 但是，取决于每个记录具有的块数，有几个“值三”，这些值由“块”字段指示。 这里的问题是将以下数据以表格形式排列，并添加与特定记录相对应的所有值三。

决赛桌应如下所示：

值一值二值三

2          5        10 

3          5         4

7          6         7

4          8         6

3          5         3

我正在考虑的方法是先找到“统计指标”，如果找到“统计指标”，我将寻找记录和与每个记录相对应的块的数量，以遍历这些块，然后将值三与对应于相似值一和值二的值相加。 。

这是我尝试提取值一，值二和值三的代码。 我还没有研究求和值三。

import re
import pandas as pd
val_dict = { 'value_one':re.compile(r'value one = (?P<value_one>.*)\n'),
           'value_two':re.compile(r'value two = (?P<value_two>.*)\n'),
           'value_three':re.compile(r'value three = (?P<value_three>.*)\n')}

def _parse_line(line):


    for key, val in val_dict.items():
        match = val.search(line)
        if match:
            return key, match
# if there are no matches
    return None, None


def parse_file(filepath):
    data = []  
    with open(filepath, 'r') as file_object:
        row = {}                                # prepare an empty row
        for line in file_object:
            key, match = _parse_line(line)
            # search for keys in the line
            if key == 'value_one':
                value_one = match.group('value_one')
                value_one = int(value_one)
                if 'value one' in row:          # we always have a full row
                    data.append(row)            # append it to the data liest
                    row = {}                    # and reset it
                row['value one'] = value_one    # we have a match: store the value in row

            if key == 'value_two':
                value_two = match.group('value_two')
                value_two = int(value_two)
                if 'value two' in row:
                    data.append(row)
                    row = {}
                row['value two'] = value_two

            if key == 'value_three':
                value_three = match.group('value_three')
                value_three = int(value_three)
                if 'value three' in row:
                    data.append(row)
                    row = {}
                row['value three'] = value_three

        if row != {}:                      # do not forget the last row
            data.append(row)
        data = pd.DataFrame(data)
        return data
if __name__ == '__main__':
    filepath = 'test3.txt'
    data = parse_file(filepath)

Answer 1

在这里，也许我们不想使用正则表达式。 但是，如果我们愿意，可以将属性名称设置为左边界，并收集所需的数字，也许使用类似以下的表达式：

value\s+(one|two|three)\s+=\s+([0-9]+)

然后，我们剩下的问题就可以编写脚本了。 如果需要的话，我们还可以在表达式中添加更多边界。

演示版

测试

# coding=utf8
# the above tag defines encoding for this document and is for Python 2.x compatibility

import re

regex = r"value\s+(one|two|three)\s+=\s+([0-9]+)"

test_str = ("Statistics indicator:0x222235\n\n"
    "number of records = 3\n\n"
    "records[0]\n\n"
    "value one = 2\n\n"
    "value two = 5\n\n"
    "blocks = 2\n\n"
    "block[0] {\n\n"
    "some irrelevant data....\n\n"
    "value three = 4 bytes\n\n"
    "}\n\n"
    "block[1]{\n\n"
    "some irrelevant data...\n\n"
    "value three = 6 bytes\n\n"
    "}\n\n"
    "records[1]\n\n"
    "value one = 3\n\n"
    "value two = 5\n\n"
    "blocks = 1\n\n"
    "block[0] {\n\n"
    "some irrelevant data....\n\n"
    "value three = 4 bytes\n\n"
    "}\n\n"
    "records[2]\n\n"
    "value one = 7\n\n"
    "value two = 6\n\n"
    "blocks = 2\n\n"
    "block[0] {\n\n"
    "some irrelevant data....\n\n"
    "value three = 3 bytes\n\n"
    "}\n\n"
    "block[1]{\n\n"
    "some irrelevant data...\n\n"
    "value three = 4 bytes\n\n"
    "}\n\n"
    "Statistics indicator:0x135256\n\n"
    "number of records = 2\n\n"
    "records[0]\n\n"
    "value one = 4\n\n"
    "value two = 8\n\n"
    "blocks = 1\n\n"
    "block[0] {\n\n"
    "some irrelevant data....\n\n"
    "value three = 6 bytes\n\n"
    "}\n\n"
    "records[1]\n\n"
    "value one = 3\n\n"
    "value two = 5\n\n"
    "blocks = 1\n\n"
    "block[0] {\n\n"
    "some irrelevant data....\n\n"
    "value three = 3 bytes\n\n"
    "}")

matches = re.finditer(regex, test_str, re.MULTILINE)

for matchNum, match in enumerate(matches, start=1):

    print ("Match {matchNum} was found at {start}-{end}: {match}".format(matchNum = matchNum, start = match.start(), end = match.end(), match = match.group()))

    for groupNum in range(0, len(match.groups())):
        groupNum = groupNum + 1

        print ("Group {groupNum} found at {start}-{end}: {group}".format(groupNum = groupNum, start = match.start(groupNum), end = match.end(groupNum), group = match.group(groupNum)))

# Note: for Python 2.7 compatibility, use ur"" to prefix the regex and u"" to prefix the test string and substitution.

演示版

 const regex = /value\\s+(one|two|three)\\s+=\\s+([0-9]+)/gm; const str = `Statistics indicator:0x222235 number of records = 3 records[0] value one = 2 value two = 5 blocks = 2 block[0] { some irrelevant data.... value three = 4 bytes } block[1]{ some irrelevant data... value three = 6 bytes } records[1] value one = 3 value two = 5 blocks = 1 block[0] { some irrelevant data.... value three = 4 bytes } records[2] value one = 7 value two = 6 blocks = 2 block[0] { some irrelevant data.... value three = 3 bytes } block[1]{ some irrelevant data... value three = 4 bytes } Statistics indicator:0x135256 number of records = 2 records[0] value one = 4 value two = 8 blocks = 1 block[0] { some irrelevant data.... value three = 6 bytes } records[1] value one = 3 value two = 5 blocks = 1 block[0] { some irrelevant data.... value three = 3 bytes }`; let m; while ((m = regex.exec(str)) !== null) { // This is necessary to avoid infinite loops with zero-width matches if (m.index === regex.lastIndex) { regex.lastIndex++; } // The result can be accessed through the `m`-variable. m.forEach((match, groupIndex) => { console.log(`Found match, group ${groupIndex}: ${match}`); }); } 

正则表达式

如果不需要此表达式，则可以在regex101.com中对其进行修改或更改。

RegEx电路

jex.im可视化正则表达式：