Python GIL-固有的文件I / O保护？

Question

Tomorrow I need to write a threaded program that reads all lines from some a src file, does some network bound I/O process, and then writes the output to a dest file. My worry is that two (or more) threads might try to write to disk at the same time and, therefore, I will have jumbled/corrupted data.

Is this something I even need to worry about? I've never multithreaded anything in my life before. Will global interpreter lock (GIL) effectively protect me in this scenario? (The reason I'm doing this is because a single threaded implementation of this will take 30 days).

Many thanks

Answer 1

Does your program really need to do the I/O multithreaded? If you are reading / writing to a single file, I would do that part of the program single threaded - it will be faster and you are avoiding locking problems all around. Not to mention the fact that disc i/o is so slow that parallen processes will not really help you. I see two possible scenarios for you: 1. Single file: Do the i/o in one thread, start N processes to do the heavy computations. 2. N files: Start one thread which manages a queue, which then starts N processes to do the work.

See https://docs.python.org/2/library/multiprocessing.html for "pool" which might be really helpful in your case.

As a word of advice: multithreading is damned hard. Many things can and will go wrong in different ways when you switch from single-threading to multi-threading. So start off with a simple single threaded model and build your way up and ensure that each step still does the correct thing.