如何在python 3中创建和启动任意数量的线程

Question

I'm making a script to download images from a web page and I'm trying to make it multithreaded so it's a lot faster.

In the downloading function I had to set two arguments because when I set one (queue) I get this error:

TypeError: downloading() takes 1 positional arguments but 21* were given

** queue has 21 links

Code:

count = 0
queue = {"some urls", , , }
done = set()
path = 'foldername'


def downloading(queue, name):
    for imgs in queue:
        if imgs not in done:
            done.add(imgs)
            urllib.request.urlretrieve(imgs, path + '/' + imgs.split('/')[-1])
            global count
            count += 1
            print(str(count) + ' ' + name)
            print('Done:  ' + imgs.split('/')[-1])


def threads(queue):
    print('Start Downloading ...')
    th1 = Thread(target=downloading, args=(queue, "Thread 1"))
    th1.start()
    th2 = Thread(target=downloading, args=(queue, "Thread 2"))
    th2.start()
    th3 = Thread(target=downloading, args=(queue, "Thread 3"))
    th3.start()
    th4 = Thread(target=downloading, args=(queue, "Thread 4"))
    th4.start()
    th5 = Thread(target=downloading, args=(queue, "Thread 5"))
    th5.start()
    th6 = Thread(target=downloading, args=(queue, "Thread 6"))
    th6.start()
    th7 = Thread(target=downloading, args=(queue, "Thread 7"))
    th7.start()
    th8 = Thread(target=downloading, args=(queue, "Thread 8"))
    th8.start()
    th9 = Thread(target=downloading, args=(queue, "Thread 9"))
    th9.start()
    th10 = Thread(target=downloading, args=(queue, "Thread 10"))
    th10.start()

Answer 1

Use a simple for loop on the number of threads you want. Of course you should save them somehow to probably join them in the end:

def threads(queue):
    num_of_threads = 10
    print('Start Downloading ...')
    threads = []
    for i in range(1, num_of_threads+1):
        th = Thread(target=downloading, args=(queue, "Thread {}".format(i)))
        threads.append(th)
        th.start()
    return threads

---

As suggested, a pool will be helpful in this use-case since it will optimize the run to your system:

from multiprocessing.dummy import Pool as ThreadPool

def downloading(img):
    urllib.request.urlretrieve(img, path + '/' + img.split('/')[-1])
    global count
    count += 1
    print('Done:  ' + img.split('/')[-1])

def threads(queue):
    pool = ThreadPool()
    pool.map(downloading, queue)
    pool.close()
    pool.join()

Note that in this way you should change the downloading function to receive one argument that is a single image. The map function sends each item of an iterable (second argument) to a function (first argument). This is also why the done set is not necessary since each image will be processed exactly once.