Python Scrapy 在 40k 请求后卡住

Question

My scrapy spider is getting stuck after 40k requests.

I am new to scrapy. Searching around, I wonder if the problem has to do with using the default parse method name and start_urls .

I am using custom_settings to speed things up. If a URL doesn't resolve within a few seconds, move on and don't retry.

Spider code:

import scrapy, re, pandas as pd, os, logging
from scrapy.spiders import CrawlSpider, Rule
from ..items import SfscrapeItem
from scrapy.utils.log import configure_logging 

dirname = os.path.dirname(__file__)
filename = os.path.join(dirname, '../resources/scraping_urls_1.1.csv')
df = pd.read_csv(filename, index_col=0)

keywords = ['canidae', 'felidae', 'cat', 'cattle', 'dog', 'donkey', 'goat', 'guinea pig', 'horse', 'pig', 'rabbit']

class sfSpider(scrapy.Spider):
    name='sfspider'

    custom_settings = {
        'DNS_TIMEOUT': 10,
        'DOWNLOAD_TIMEOUT':10,
        'RETRY_ENABLED': False,
        'REDIRECT_MAX_TIMES': 2,
    }

    configure_logging(install_root_handler=False)
    logging.basicConfig(
        filename='log.txt',
        format='%(levelname)s: %(message)s',
        level=logging.INFO
    )

    #just the first 75k for now...
    start_urls = df.url.to_list()[:75000] 

    def parse(self, response):

        response_body = response.body.decode('utf-8')

        url = response.url
        domain = url.split('/')[2]

        item = SfscrapeItem()
        item['url'] = url
        item['domain'] = domain
        item['status'] = response.status
        item['matches'] = [str(len(re.findall(keyword, response_body, re.IGNORECASE))) for keyword in keywords]
        
        yield item

Here is the log at the point when it is stuck

INFO: Crawled 40940 pages (at 565 pages/min), scraped 16473 items (at 243 items/min)
INFO: Crawled 40940 pages (at 0 pages/min), scraped 16473 items (at 0 items/min)
INFO: Crawled 40940 pages (at 0 pages/min), scraped 16473 items (at 0 items/min)
INFO: Crawled 40940 pages (at 0 pages/min), scraped 16473 items (at 0 items/min)
INFO: Crawled 40940 pages (at 0 pages/min), scraped 16473 items (at 0 items/min)
INFO: Crawled 40940 pages (at 0 pages/min), scraped 16473 items (at 0 items/min)

And here is the output of stats.get_stats() and prefs() after it is stuck.

Answer 1

It looks like You started to receive DNSLookupError
as you have this downloader/exception_type_count/twisted.internet.DNSLookupError: 5109 in stats on minutes with ...pages (at 0 pages/min), scraped x items (at 0 items/min)

LogStats extention (module that print INFO: Crawled X pages (at X pages/min), scraped X items (at X items/min) to log) count only received responses (it doesn't count received exceptions).