Python该脚本可以是多线程的吗？

Question

It would be great if someone could help me with multi-threading this script and write the output to a text file. I am really new at coding, so please help me out.

#!/usr/bin/python

from tornado import ioloop, httpclient
from BeautifulSoup import BeautifulSoup
from mechanize import Browser
import requests
import urllib2
import socket
import sys

def handle_request(response):
    print response.code

global i

i = 0
i -= 1
if i == 0:
    http_client = httpclient.AsyncHTTPClient()
for url in open('urls.txt'):
    try:
        br = Browser()
        br.set_handle_robots(False)
        res = br.open(url, None, 2.5)
        data = res.get_data()
        soup = BeautifulSoup(data)
        title = soup.find('title')
        if soup.title != None:
            print url, title.renderContents(), '\n'
        i += 1
    except urllib2.URLError, e:
        print "Oops, timed out?", '\n'
    except socket.error,e:
        print "Oops, timed out?", '\n'
    except socket.timeout:
        print "Oops, timed out?", '\n'
print 'Processing of list completed, Cheers!!'
sys.exit()
try:
    ioloop.IOLoop.instance().start()
except KeyboardInterrupt:
    ioloop.IOLoop.instance().stop()

I am trying to grep the HTTP title of a list of hosts.

Answer 1

The basic idea you have already implemented is an non-blocking HTTP client.

def handle_request(response):
    if response.error:
        print "Error:", response.error
    else:
        print response.body

for url in ["http://google.com", "http://twitter.com"]:
    http_client = httpclient.AsyncHTTPClient()
    http_client.fetch(url, handle_request)

You could loop over your urls and the callback will be called as soon the respone for a specific url becomes availible.

I wouldn't mix up mechanize, ioloop,... if not necessary.

---

Apart from that, I recommend grequests . It is a lightweight tool which satisfies your requirements.

import grequests
from bs4 import BeautifulSoup

urls = ['http://google.com', 'http://www.python.org/']

rs = (grequests.get(u) for u in urls)
res = grequests.map(rs)

for r in res:
    soup = BeautifulSoup(r.text)
    print "%s: %s" % (r.url, soup.title.text)