threads with readline

Question

I have a huge file with hundreds of thousands of lines. I need to run the same process on each line. My plan was to make several threads to speed up the process. Whenever I multithreaded before I used treading and Queue modules. However I cannot figure out how to apply a queue. What I really need to do is read the file line by line, as the file is too large to do the whole thing. I thought that maybe I could just add one thing to the queue at a time with .put(), then immediately pass it to the thread, but it seems like if I did this the threads could conflict. Any suggestions?

Answer 1

How much processing is there per line.

If not a lot then you might slow things down with multiple threads contending for the device the file is on? You might want to split the file beforehand and put the components on different devices? Then it's a simple matter of firing up a process per file or per group of files.

I'd use the split, xargs -P unix commands for this