如何在python 3中創建和啟動任意數量的線程

Question

我正在制作一個腳本來從網頁上下載圖像，並且試圖使其成為多線程，因此速度要快得多。

在下載功能中，我必須設置兩個參數，因為設置一個（隊列）時會出現此錯誤：

TypeError: downloading() takes 1 positional arguments but 21* were given

** queue has 21 links

碼：

count = 0
queue = {"some urls", , , }
done = set()
path = 'foldername'


def downloading(queue, name):
    for imgs in queue:
        if imgs not in done:
            done.add(imgs)
            urllib.request.urlretrieve(imgs, path + '/' + imgs.split('/')[-1])
            global count
            count += 1
            print(str(count) + ' ' + name)
            print('Done:  ' + imgs.split('/')[-1])


def threads(queue):
    print('Start Downloading ...')
    th1 = Thread(target=downloading, args=(queue, "Thread 1"))
    th1.start()
    th2 = Thread(target=downloading, args=(queue, "Thread 2"))
    th2.start()
    th3 = Thread(target=downloading, args=(queue, "Thread 3"))
    th3.start()
    th4 = Thread(target=downloading, args=(queue, "Thread 4"))
    th4.start()
    th5 = Thread(target=downloading, args=(queue, "Thread 5"))
    th5.start()
    th6 = Thread(target=downloading, args=(queue, "Thread 6"))
    th6.start()
    th7 = Thread(target=downloading, args=(queue, "Thread 7"))
    th7.start()
    th8 = Thread(target=downloading, args=(queue, "Thread 8"))
    th8.start()
    th9 = Thread(target=downloading, args=(queue, "Thread 9"))
    th9.start()
    th10 = Thread(target=downloading, args=(queue, "Thread 10"))
    th10.start()

Answer 1

對所需的線程數使用簡單的for循環。 當然，您應該以某種方式保存他們，以便最終join他們：

def threads(queue):
    num_of_threads = 10
    print('Start Downloading ...')
    threads = []
    for i in range(1, num_of_threads+1):
        th = Thread(target=downloading, args=(queue, "Thread {}".format(i)))
        threads.append(th)
        th.start()
    return threads

---

如建議的那樣， pool將在此用例中有所幫助，因為它將優化對系統的運行：

from multiprocessing.dummy import Pool as ThreadPool

def downloading(img):
    urllib.request.urlretrieve(img, path + '/' + img.split('/')[-1])
    global count
    count += 1
    print('Done:  ' + img.split('/')[-1])

def threads(queue):
    pool = ThreadPool()
    pool.map(downloading, queue)
    pool.close()
    pool.join()

請注意，通過這種方式，您應該更改downloading功能以接收一個參數（即單個圖像）。 map函數將可迭代項（第二個參數）的每個項目發送到函數（第一個參數）。 這也是為什么done集沒有必要的原因，因為每個圖像將被精確地處理一次。