迭代一个文件，但每次迭代需要5行

Question

I've created 5 threads to process the lines. I send one line as argument to each thread.

The output is just that I need. But then it stop by error.

The code:

#!usr/bin/env python3
# -*- coding: UTF-8 -*-

import threading

# Create class myThread as subclass of Thread
class MyThread(threading.Thread):
    def __init__(self, num, myArg):
        threading.Thread.__init__(self)
        self.num = num
        self.myArg = myArg

# Overrides run() method to defines the thread goes to do.
    def run(self):
        print ("I'm thread number: ", self.num)
        print(self.myArg)


myFile = open('file_01.txt', mode='r')

for myLine in myFile:
    for h in range(1, 6):    # create 5 instances of the thread
        t = MyThread(h, myLine)
        t.start()
        myLine = myFile.__next__()

myFile.close()

The error:

Traceback (most recent call last):
  File "/sajime/PycharmProjects/Learning/iterarFichero.py", line 25, in <module>
    myLine = myFile.__next__()
StopIteration

The 'file_01.txt' content is a simple 'Lorem ipsum dolor sit amet,...' stuff.

The bug isn't in the multi-threadinig class nor calls, It comes in the iteration of the file, but, why?

For those who is asking why I need this: The script must process the lines to load data in web forms, and take a lot of time (lags in the server). I realized that if I divide the tasks is more faster. (I don't know if there is a better method to do it)

Answer 1

Try this:

for count, myLine in enumerate(myFile):
    t = MyThread(count % 5 + 1, myLine)
    t.start()

Answer 2

with myLine = myFile.__next__() , you advance the myFile Iterator. When the iterator is fully consumed, it throws that StopIteration Exception as a signal.

You can catch that, and simply break the loop, since you know you're done.

Unfortunately, there is a logic error in your program, too: you advance the iterator after every thread start, but also in the outer loop. That means that after you've started all Threads, the next line will be read into myLine which gets immediately overwritten by the outer loop.

To avoid that (and to have fewer code), you can replace the whole inner and outer loop with something like

[MyThread(i%5+1, myLine).start() for i, myLine in enumerate(myFile)]

Answer 3

from itertools import cycle, izip
for h, myLine in izip(cycle(range(1,6)), myFile):
    t = MyThread(h, myLine)
    t.start()

Does this do what you want?

Answer 4

It's because you're calling the 'next' line twice in each loop.

The for loop in your code iterates through the lines by calling next each time. Then you're calling it again within your loop.

Pull out this:

myLine = myFile.__next__()

To make the final loop:

h=0
for myLine in myFile:
    t = MyThread((h % 6), myLine)
    t.start()
    h+=1

The % does an integer division to always ensure it fits into the thread indexes!