阅读csv文件的特定行

Question

Hllo guys, so i have a huge CSV file (500K of lines), i want to process the file simultaneously with 4 processes (so each one will read aprox. 100K of lines) what is the best way to do it using multi proccessing?

what i have up til now:

def csv_handler(path, procceses = 5):

    test_arr = []
    with open(path) as fd:
        reader = DictReader(fd)

        for row in reader:
            test_arr.append(row)

    current_line = 0
    equal_length = len(test_arr) / 5

    for i in range(5):
        process1 = multiprocessing.Process(target=get_data, args=(test_arr[current_line: current_line + equal_length],))
        current_line = current_line + equal_length

i know it's a bad udea to do that with one reading line, but i don't find another option.. i would be happy to get some ideas to how to do it in a better way!

Answer 1

CSV is a pretty tricky format to split the reads up with, and other file formats may be more ideal.

The basic problem is that as lines may be different lengths, you can't know where to start reading a particular lines easily to " fseek " to it. You would have to scan through the file counting newlines, which is basically, reading it.

But you can get pretty close which sounds like it is enough for your needs. Say for two parts, take the file size, divide that by 2.

• The first part you start at zero, and stop after completing the record at file_size / 2 .
• The second part, you seek to file_size / 2 , look for the next new line, and start there.

This way while the Python processes won't all get exactly the same amount it will be pretty close, and avoids too much inter-process message passing or multi-threading and with CPython probably the global interpreter lock.

---

Of course all the normal things for optimising either file IO, or Python code still apply (depending on where your bottleneck lies. You need to measure this .).