需要Python线程帮助

Question

I am snowblind from looking at examples that don't exactly fit my case ... or maybe they do. So if there are great examples out there for this, I have not been able to interpret them with about 3 weeks of Python experience

I have a script that queries a database, gathers a list of downloadable movies, then downloads them to your chosen directory, one by one. And I would like to make it download 4 or 5 at once, as it takes an age to do it.

Here is a simplified version what I have attempted to do, myapp is my database application. However it appears to just run sequential, even though it says it's starting both threads

listOfIDs are the ids of some containers that may or may not have movies, then versionS returns the movie file names.

import threading
import myapp_api

listOfIDs = (14809, 14808, 14807, 14806, 14805, 14804, 14803)
for ID in listOfIDs:
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])

ipath = ('/Users/me/Desktop/scripts/downloads/')

exitFlag = 0

class myThread (threading.Thread):
    def __init__(self, threadID, name, counter):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
        self.counter = counter
    def run(self):
        print "Starting " + self.name
        for ID in listOfIDs:
            print "\nID= " + str(ID) + "\n"
            downLoad(ID)
        print "Exiting " + self.name

def downLoad(ID):
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])
    path = ipath + (str(versionS).split("'")[5])
    result = myapp.download_attachment(attachment=versionS['uploaded_movie'], file_path=path)
    print "Thread Name = " + threadName


# Create new threads
thread1 = myThread(1, "Thread-1", 1)
thread2 = myThread(2, "Thread-2", 2)


# Start new Threads
thread1.start()
thread2.start()

print "Exiting Main Thread"

Ok, so I altered the code to take on the suggestion by ShadowRanger and it is still just downloading one at a time, have I stuffed this up somewhere? ... code looks like this now.

import threading
import myapp_api
from collections import deque

listOfIDs = (14809, 14808, 14807, 14806, 14805, 14804, 14803)
for ID in listOfIDs:
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])

ipath = ('/Users/me/Desktop/scripts/downloads/')

def downLoad(ID):
    path = ipath + (str(versionS).split("'")[5])
    result = myapp.download_attachment(attachment=versionS['uploaded_movie'], file_path=path)

with closing(multiprocessing.Pool(4)) as pool:
    deque(pool.imap_unordered(downLoad, listOfIDs), maxlen=0)

And finally, All the advice by ShadowRanger was spot on, the error was in something I had done incorrectly (I think I was iterating over the listOfIDs early, and only passing the last one out to the function) ... here is the final working version.

   import threading
    import myapp_api
    from collections import deque

listOfIDs = (14809, 14808, 14807, 14806, 14805, 14804, 14803)

ipath = ('/Users/me/Desktop/scripts/downloads/')

def downLoad(ID):
    versionS = myapp.find_one('Version', [['id', 'is', ID]], ['uploaded_movie'])
    path = ipath + (str(versionS).split("'")[5])
    result = myapp.download_attachment(attachment=versionS['uploaded_movie'], file_path=path)


with closing(multiprocessing.Pool(4)) as pool:
    deque(pool.imap_unordered(downLoad, listOfIDs), maxlen=0)

Answer 1

I don't see how the threads are splitting work. Looks like they both download the same set of things.

If the goal is to download a bunch of files based on known IDs, multiprocessing has a .dummy module that can act like multiprocessing but implemented using threads, which gets you an easy thread pool:

import multiprocessing.dummy as multiprocessing
from contextlib import closing

with closing(multiprocessing.Pool(4)) as pool: # Pick your favorite number of workers
    pool.map(downLoad, listOfIDs)