Scrapy：TypeError：__ init __（）缺少1個必需的位置參數：'settings'

Question

我有一個Scrapy中間件：

class ProxyMiddleware(object):
    def __init__(self, settings):
        self.proxy_file = settings.get('PROXY_FILE')
        fin = open(self.proxy_file)
        self.proxy_list = list()
        for line in fin.readlines():
            parts = line.strip().split()
            proxy = parts[2] + '://' + parts[0] + ':' + parts[1]
            self.proxy_list.append(proxy)

    def process_request(self, request, spider):
        request.meta['proxy'] = random.choice(self.proxy_list)

但運行時出錯，設置是什么？

Answer 1

如果需要settings對象來初始化中間件，則需要定義from_crawler()類方法， 否則scrapy會在不使用參數的情況下初始化中間件 。

看看內置的中間件獲取靈感，例如HttpErrorMiddleware

class HttpErrorMiddleware(object):

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def __init__(self, settings):
        self.handle_httpstatus_all = settings.getbool('HTTPERROR_ALLOW_ALL')
        self.handle_httpstatus_list = settings.getlist('HTTPERROR_ALLOWED_CODES')

在你的情況下，它將是這樣的：

class ProxyMiddleware(object):

    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def __init__(self, settings):
        self.proxy_file = settings.get('PROXY_FILE')
        fin = open(self.proxy_file)
        self.proxy_list = list()
        for line in fin.readlines():
            parts = line.strip().split()
            proxy = parts[2] + '://' + parts[0] + ':' + parts[1]
            self.proxy_list.append(proxy)

    def process_request(self, request, spider):
        request.meta['proxy'] = random.choice(self.proxy_list)