如何使用Python有效地读取带有自定义换行符的大文件？

Question

We have a huge .csv file but it doesn't seem to really be a csv.

The line endings are \\tl\\n .
The text between this newline character sometimes has "real" newline characters. We don't want to split on those.

We currently do it using awk .

awk_code = r'BEGIN{ RS="""(\tl\n)"""; FS="\t"} { print "\42"$1"\42,\42"$2"\42,\42\42\42"$3"\42\42\42,\n";}'
bash_command_awk = f"awk '{awk_code}' {input_file_path} > {output_path}"
awk_command_output = subprocess.check_output(bash_command_awk,stderr=subprocess.STDOUT, shell=True)

I'm trying to find an efficient way of doing it directly in Python and tried passing a custom newline into the .open() command.

def process_without_putting_file_in_RAM(file_to_process):
    with file_to_process.open(encoding="utf-8", newline="\tl\n") as csv_file:
        for line in csv.reader(csv_file):

However, I quickly learned newline arg only accepts one of the default characters.

How can I efficiently process this file containing the weird line ending?

Answer 1

Here's a function which can handle multi-character newline between chunks correctly

def line_splitter(file, newline, chunk_size=4096):
    tail = ''
    while True:
        chunk = file.read(chunk_size)
        if not chunk:
            if tail:
                yield tail
            break
        lines = (tail + chunk).split(newline)
        tail = lines.pop(0)
        if lines:
            yield tail
            tail = lines.pop()
            yield from lines

another version which, although it doesn't make copies of whole chunks didn't prove faster. It will be marginally faster for large chunks. Do not use chunk_size less than newline size :)

def line_splitter(file, newline, chunk_size=4096):
    tail = ''
    while True:
        chunk = file.read(chunk_size)
        if not chunk:
            if tail:
                yield tail
            break
        lines = chunk.split(newline)
        tail = (tail + lines[0]).split(newline)
        if len(tail) > 1:
            lines[0] = tail[1]
        else:
            del lines[0]
        tail = tail[0]
        if lines:
            yield tail
            tail = lines.pop()
            yield from lines

The caller should be like:

with longabstract_file.open() as f:
    for line in line_splitter(f, "\tl\n"):
        if line: # ignore blank lines
            print(line)

Answer 2

Assuming your csv is a comma or space delimited, not tab, what you were looking for is lineterminator flag , but there's no need for that since it's automatically assumed '\\n' is a line break. From the doc:

Note: The reader is hard-coded to recognise either '\\r' or '\\n' as end-of-line, and ignores lineterminator . This behavior may change in the future.

so what you can do is add string method .replace() to get rid of '\\tl' like this

def process_without_putting_file_in_RAM(file_to_process):
    with file_to_process.open(encoding="utf-8") as csv_file:
        for line in csv.reader(csv_file, delimiter=","):
            print(line[-1].replace('\tl', ''))

Answer 3

Why not use pandas . Specifically pandas.read_csv using lineterminator and chunksize parameters:

import pandas as pd

batch_size = 10000
new_line_str = '\tl\n'
iterator_df = pd.read_csv(file_to_process, chunksize=batch_size, lineterminator=new_line_str)
for chunk in iterator_df:
    # process chunk here