Python-多线程时限制线程数

Question

下面的代码。

            import sys
            import urllib2
            import threading
            import time

            urls = ["http://www.google.com", "http://www.apple.com"]

            def fetch_url(url):
                html = urllib2.urlopen(url).read()
                print html

            f = open("Output.txt", "w")
            e = open("ErrorUsers.txt", "w")

            threads = [threading.Thread(target=fetch_url, args=(url,)) for url in urls]
            for thread in threads:
                try:
                    thread.daemon = True
                    thread.start()
                except urllib2.HTTPError, e:
                    e.write(url + "\n")
                except urllib2.URLError, e:
                    e.write(url + "\n")
            for thread in threads:
                thread.join()

本质上，我需要从大量站点中获取数据（我们正在谈论数以万计的站点）。 我将上面的代码用作基础，效果很好，但是，在创建了750个线程之后，代码冻结了。 我想知道如何限制活动线程的数量，或在完成线程后喜欢关闭线程。

Answer 1

您可以像在多处理中实现的那样使用线程池。在以下代码段中，任何时候最多可以激活100个线程。

from multiprocessing.pool import ThreadPool

urls = ["http://www.google.com", "http://www.apple.com"]

def fetch_url(url):
    html = urllib2.urlopen(url).read()
    print html

pool = ThreadPool(100)
pool.map(fetch_url, urls)
pool.close()
pool.join()

Answer 2

            from multiprocessing.pool import ThreadPool
            import urllib2


            urls = ["http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com","http://www.google.com", "http://www.apple.com"]

            def fetch_url(url):
                html = urllib2.urlopen(url).read()
                print html[1:10]
                pool.TerminateProcess()

            pool = ThreadPool(100)
            pool.map(fetch_url, urls)
            pool.close()
            pool.join()