在Python中将杂乱的数据文件清理为更具可读性的格式？

Question

I have a text file (heavily modified for this example) which has some data that I want to extract and do some calculations with it. However the text file is extremely messy, so I'm trying to clean it up and write it out to new files first.

Here is the .txt file I'm working with: http://textuploader.com/5elql

I am trying to extract the data which is under the titles (called “Important title”). The only possible way to do that is to first locate a string which always occurs in the file, and its called “DATASET” because all the mess above and below the important data will cover an arbitrary number of lines, difficult to remove manually. Once that's done I want to store the data in separate files so that it is easier to analyse like this:

http://textuploader.com/5elqw

The file names will be concatenated with the title + the date.

Here is what I have tried so far

with open("example.txt") as file:
    for line in file:
        if line.startswith('DATASET:'):
            fileTitle = line[9:]
        if line.startswith("DATE:"):
            fileDate = line[:]
            print(fileTitle+fileDate)

OUTPUT

IMPORTANT TITLE 1
DATE: 12/30/2015

IMPORTANT TITLE 2
DATE: 01/03/2016

So it appears my loop manages to locate the lines where the titles inside the file are and print them out. But this is where I run out of steam. I have no idea on how to extract the data under those titles from there onwards. I have tried using file.readlines() but it outputs all the mess that is in between Important Title 1 and Important Title 2.

Any advice on how I can read all the data under the titles and output them into separate files? Thanks for your time.

Answer 1

You could use regex.

import re

pattern = r"(\s+X\s+Y\s*)|(\s*\d+\s+\d+\s*)"
prog = re.compile(pattern)

with open("example.txt") as file:
cur_filename = ''
content = ""
for line in file:
    if line.startswith('DATASET:'):
        fileTitle = line[9:]
    elif line.startswith("DATE:"):
        fileDate = line[6:]
        cur_filename = (fileTitle.strip() + fileDate.strip()).replace('/', '-')
        print(cur_filename)
        content_title = fileTitle + line
    elif prog.match(line):
        content += line
    elif cur_filename and content:
        with open(cur_filename, 'w') as fp:
            fp.write(content_title)
            fp.write(content)
        cur_filename = ''
        content = ''

Answer 2

I don't know exactly how you want to store your data but assuming you want a dictionary you could use regex to check if the incoming line matched the pattern, then because fileTitle isn't global you could use that as the key and add the values. I also added rstrip('\\r\\n') to remove the newline characters after fileTitle.

import re

#if you don't want to store the X and Y, just use re.compile('\d\s+\d+')
p = re.compile('(\d\s+\d+)|(X\s+Y)')
data={}
with open("input.txt") as file:
    for line in file:
        if line.startswith('DATASET:'):
            fileTitle = line[9:].rstrip('\r\n')
        if line.startswith("DATE:"):
            fileDate = line[:]
            print(fileTitle+fileDate)
        if p.match(line):
            if fileTitle not in data:
                data[fileTitle]=[]
            line=line.rstrip('\r\n')
            data[fileTitle].append(line.split('\t'))
            if len(data[fileTitle][len(data[fileTitle])-1]) == 3:
                data[fileTitle][len(data[fileTitle])-1].pop()

print data

Answer 3

Yet another regex solution:

sep = '*************************\n'

pattern = r'DATASET[^%]*'
good_stuff = re.compile(pattern)
pattern = r'^DATASET: (.*?)$'
title = re.compile(pattern, flags = re.MULTILINE)
pattern = r'^DATE: (.*?)$'
date = re.compile(pattern, flags = re.MULTILINE)

with open(r'foo.txt') as f:
    data = f.read()
for match in good_stuff.finditer(data):
    data = match.group()
    important_title = title.search(data).group(1)
    important_date = date.search(data).group(1)
    important_date = important_date.replace(r'/', '-')
    fname = important_title + important_date + '.txt'
    print(sep, fname)
    print(data)
    ##with open(fname, 'w') as f:
    ##    f.write(data)