python2.7 + multiprocessing + selenium：异常时重启进程

Question

我似乎对使用多处理的python脚本有疑问。 它实际上要做的是获取ID代码列表，并启动使用Selenium和PhantomJS作为驱动程序的进程，以导航到包含该ID代码的URL，将数据提取到单个csv文件中，然后在所有进程完成后编译另一个csv文件。 一切运行良好，但有时其中一个进程将返回一个异常，内容为：

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
    self.run()
  File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
    self._target(*self._args, **self._kwargs)
  File "modtest.py", line 11, in worker
    do_work(item)
  File "/home/mdrouin/Dropbox/Work/Dev/Python/WynInvScrape/items.py", line 14, in do_work
    driver = webdriver.PhantomJS()
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 50, in __init__
    self.service.start()
  File "/usr/lib/python2.7/site-packages/selenium/webdriver/phantomjs/service.py", line 72, in start
    raise WebDriverException("Can not connect to GhostDriver")

我尝试过以某种方式在出现异常的情况下重新启动过程，但是无论如何，似乎正在发生的事情是一旦过程完成，程序挂起且无法继续运行，或为此执行任何操作。 我本质上想在进程崩溃时重新启动正在搜索的ID号，并在所有进程完成后继续进行。 这是代码的极简缩版：

from selenium import webdriver
from time import sleep
from bs4 import BeautifulSoup as bs
import multiprocessing
import datetime, time, csv, glob


num_procs = 8

def do_work(rsrt):

        driver = webdriver.PhantomJS()

        try:
            driver.get('http://www.example.com/get.php?resort=' + rsrt)

            rows = []

            for row in soup.find_all('tr'):
                if row.find('input', {'name': 'booksubmit'}):
                    wyncheckin = row.find('td', {'class': 'searchAvailDate'}).string
                    wynnights = row.find('td', {'class': 'searchAvailNights'}).string
                    wynroom = row.find('td', {'class': 'searchAvailUnitType'}).string
                    rows.append([wynresort, wyncheckin, wynroom])


            driver.quit()

            with open('/home/mdrouin/Dropbox/Work/Dev/Python/WynInvScrape/availability/'+rsrt+'.csv', 'wb') as f:
                writer = csv.writer(f)
                writer.writerows(row for row in rows if row)

            print 'Process ' + rsrt + ' End: ' + str(time.strftime('%c'))


        except:
            driver.quit()



def worker():
    for item in iter( q.get, None ):
        do_work(item)
        q.task_done()
    q.task_done()


q = multiprocessing.JoinableQueue()

procs = []

for i in range(num_procs):
    procs.append( multiprocessing.Process(target=worker) )
    procs[-1].daemon = True
    procs[-1].start()

source = ['0017', '0113', '0020', '0013', '0038', '1028', '0115', '0105', '0041', '0037', '0043', '2026', '0165', '0164',
        '0033', '0126', '0116', '0103', '9135', '0185', '0206', '0053', '0062', '1020', '0019', '0042', '2028', '0213',
        '0211', '0163', '0073', '2020', '0214', '2140', '0084', '0193', '0095', '0064', '0196', '0028', '0068', '0074']

for item in source:
    q.put(item)

q.join()

for p in procs:
    q.put( None )

q.join()

for p in procs:
    p.join()

print "Finished"
print 'Writting core output: ' + str(time.strftime('%c'))
with open('availability.csv', 'wb') as outfile:
    for csvfile in glob.glob('/home/mdrouin/Dropbox/Work/Dev/Python/WynInvScrape/availability/*.csv'):
        for line in open(csvfile, 'r'):
            outfile.write(line)

print 'Process End: ' + str(time.strftime('%c'))

Answer 1

解决此类问题的方法之一是反复调用自身，类似于以下内容：

def do_work(rsrt):
    if failed:
        return do_work(rsrt)

当然，它会一直运行到解析为止，因此您可能需要传递一个计数器，如果它高于某个值，则返回false。