Python-多線程時限制線程數

Question

下面的代碼。

            import sys
            import urllib2
            import threading
            import time

            urls = ["http://www.google.com", "http://www.apple.com"]

            def fetch_url(url):
                html = urllib2.urlopen(url).read()
                print html

            f = open("Output.txt", "w")
            e = open("ErrorUsers.txt", "w")

            threads = [threading.Thread(target=fetch_url, args=(url,)) for url in urls]
            for thread in threads:
                try:
                    thread.daemon = True
                    thread.start()
                except urllib2.HTTPError, e:
                    e.write(url + "\n")
                except urllib2.URLError, e:
                    e.write(url + "\n")
            for thread in threads:
                thread.join()

本質上，我需要從大量站點中獲取數據（我們正在談論數以萬計的站點）。 我將上面的代碼用作基礎，效果很好，但是，在創建了750個線程之后，代碼凍結了。 我想知道如何限制活動線程的數量，或在完成線程后喜歡關閉線程。

Answer 1

您可以像在多處理中實現的那樣使用線程池。在以下代碼段中，任何時候最多可以激活100個線程。

from multiprocessing.pool import ThreadPool

urls = ["http://www.google.com", "http://www.apple.com"]

def fetch_url(url):
    html = urllib2.urlopen(url).read()
    print html

pool = ThreadPool(100)
pool.map(fetch_url, urls)
pool.close()
pool.join()

Answer 2

            from multiprocessing.pool import ThreadPool
            import urllib2


            urls = ["http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com"]

            def fetch_url(url):
                html = urllib2.urlopen(url).read()
                print html[1:10]
                pool.TerminateProcess()

            pool = ThreadPool(100)
            pool.map(fetch_url, urls)
            pool.close()
            pool.join()