Python在同一资源上执行不同的操作

Question

I'm trying to analyze a large python file with different tasks on it. I've already read and preprocessed the file and it's now in memory. The thing is, the tasks I have, they pretty much have to go through the whole list of the records. It's something similar like:

resourceList = [..] #list of records from the file (say, 2GB)
def taskA():
    for i in resourceList:
        #doSthA()

def taskB():
    for i in resourceList:
        #doSthB()

If I do taskA() then taskB() it's going through the 2GB file twice and it's really slow. Is that a way that taskA and taskB can do their job simultaneously at the same time so that I don't have to go through the task twice?

I read about something which involved python threads and Queue, is that the only (and right) way to do it? If so, what if the "resourceList" is a generator instead of a list?

Thanks!

Answer 1

I'd implement this using threading (because I find this problem easier to reason about when each task is a separate thread, and threading over multiprocessing so that data can be shared), then passing each function a queue which can be iterated over:

import threading
from Queue import Queue

class IterableQueue(Queue): 
    _sentinel = object()

    def __iter__(self):
        return iter(self.get, self._sentinel)

    def close(self):
        self.put(self._sentinel)

def taskA(resources):
    for resource in resources:
        do_stuff()

def taskB(resources):
    for resource in resources:
        do_stuff()

def start_thread(task):
    queue = IterableQueue(maxsize=1)
    thread = threading.Thread(target=task, args=(queue, ))
    thread.start()
    return (thread, queue)

threads = [
    start_thread(taskA),
    start_thread(taskB),
]

resource_list = [...]

for resource in resource_list:
    for _, queue in threads:
        queue.put(resource)

for thread, _ in threads:
    thread.join()