读取数据文件空白区域之间的行并写入新文件

Question

I have the BIG data text file for example:

#01textline1
1 2 3 4 5 6
2 3 5 6 7 3
3 5 6 7 6 4
4 6 7 8 9 9

1 2 3 6 4 7
3 5 7 7 8 4
4 6 6 7 8 5

3 4 5 6 7 8
4 6 7 8 8 9
..
..

I want to extract data between empty lines and write it in new files. It is hard to know how many empty lines are in file (means you also dont know how many new files you will be writing ; thus it seems very hard to write new files since u dont know how many new files will you be writing. Can anyone guide me? Thank you. I hope my question is clear.

Answer 1

Unless your file is very large, split all into individual sections using re, splitting on 2 or more whitespace chars

import re
with open("in.txt") as f:
    lines = re.split("\s{2,}",f.read())
    print lines
['#01textline1\n1 2 3 4 5 6\n2 3 5 6 7 3\n3 5 6 7 6 4\n4 6 7 8 9 9', '1 2 3 6 4 7\n3 5 7 7 8 4\n4 6 6 7 8 5', '3 4 5 6 7 8\n4 6 7 8 8 9']

Just iterate over lines and write your new files each iteration

Answer 2

Reading files is not data-mining . Please choose more appropriate tags...

Splitting a file on empty lines is trivial:

num = 0
out = open("file-0", "w")

for line in open("file"):
    if line == "\n":
      num = num + 1
      out.close()
      out = open("file-"+num, "w")
      continue
    out.write(line)

out.close()

As this approach is reading just one line at a time , file size does not matter. It should process data as fast as your disk can handle it, with near-constant memory usage.

Perl would have had a neat trick, because you can set the input record separator to two newlines via $/="\\n\\n"; and then process the data one record at a time as usual... I could not find something similar in python; but the hack with "split on empty lines" is not bad either.

Answer 3

Here is a start:

with open('in_file') as input_file:
    processing = False
    i = 0
    for line in input_file:
        if line.strip() and not processing:
            out_file = open('output - {}'.format(i), 'w')
            out_file.write(line)
            processing = True
            i += 1
        elif line.strip():
            out_file.write(line)
        else:
            processing = False
            out_file.close()

This code keeps track of whether a file is being currently written to, with the processing flag. It resets the flag when it sees a blank line. The code also creates a new file upon seeing an empty line.

Hope it helps.