将非csv文本文件解析为数据框

Question

I have a text file where all columns are merged into a single column and 'rows' are separated by two long rows of '-'. It looks like this:

Hash: some_hash_id
Author: some_author
Message: Message about the update


Reviewers: jimbo

Reviewed By: jimbo

Test Plan: Auto-generated

@bypass-lint
Commit Date: 2019-06-30 20:12:38
Modified path: path/to/my/file.php
Modified path: some/other/path/to/my/file.php
Modified path: path/to/other/file.php
-------------------------------------------------------
-------------------------------------------------------
Hash: some_other_hash_id
Author: different_author
Message: Auto generated message



Reviewers: broseph

Reviewed By: broseph

Test Plan: Auto-generated by Sam

@bypass-lint
Commit Date: 2019-06-30 18:09:12
Modified path: my/super/file.php
Modified path: totally/awesome/file.php
Modified path: file/path.json
-------------------------------------------------------
-------------------------------------------------------
Hash: hash_id_4
Author: new_author
Message: Auto DB big update



Reviewers: foo

Reviewed By: foo

Test Plan: Auto-generated by Tom

@bypass-lint
Commit Date: 2019-06-30 11:08:59
Modified path: big/scripts/file.json

expected output for this example is is a dataframe with just 3 rows. dataframe columns: Hash (str), Author (str), Message (str), Reviewers (str), Reviewed By (str), Test Plan (str), Commit Date (timestamp), Modified path (array(str))

Answer 1

Load the whole file content into a variable named txt .

Then, to generate a DataFrame, it is enough to run a single (although quite complex) instruction:

pd.DataFrame([ collections.OrderedDict(
    { m.group('key').strip(): re.sub(r'\n', ' ', m.group('val').strip())
        for m in re.finditer(
            r'^(?P<key>[^:\n]+):\s*(?P<val>.+?(?:\n[^:\n]+)*)$', chunk, re.M)})
    for chunk in re.split(r'(?:\n\-+)+\n', txt) ])

Start reading of the code from the last line. It splits txt into chunks, on a sequence of lines containing only - chars.

Then finditer takes over, dividing each chunk into key and value capturing groups.

The next step is a dictionary comprehension, stripping / substituting each key and value and creating an OrderedDict (import collections ).

All these dictionaries are enclosed in a list comprehension.

And the last step is to create a DataFrame.

To avoid multi-line items, in each value (the piece of text after the colon) newlines were replaced with a space (you are free to change it).

Answer 2

Here's one implementation. Loop through each line and when the line contains : split the line as columnname:columnval and add columnname as key and columnval as value to a temp dictionary. use if statements to detect when you encountered special keys Hash (for the start of new row), Modified path (add it to an array) and Commit Date (converting it to datetime)

import pandas as pd
from datetime import datetime

test_path = '/home/kkawabat/.PyCharmCE2018.1/config/scratches/test.txt'
with open(test_path, 'r') as ofile:
    lines = ofile.readlines()
row_list = []
cur_row_dict = {}
for line in lines:
    line_split = line.split(':', 1)
    if len(line_split) == 2:
        colname, colval = line_split[0].strip(), line_split[1].strip()
        if colname == 'Hash': #assuming Hash is always the first element
            if len(cur_row_dict) != 0:
                row_list.append(cur_row_dict)
                cur_row_dict = {}
        elif colname == 'Commit Date':
            cur_row_dict[colname] = datetime.strptime(colval, '%Y-%m-%d %H:%M:%S')
        elif colname == 'Modified path':
            if colname not in cur_row_dict:
                cur_row_dict[colname] = [colval]
            else:
                cur_row_dict[colname].append(colval)
        else:
            cur_row_dict[colname] = colval
row_list.append(cur_row_dict)

df = pd.DataFrame(row_list)
print(df)