time.sleep（）函数在Scrapy递归网络爬虫中不起作用

Question

I am using Python.org version 2.7 64 bit on Windows Vista 64 bit. I have some recursive webscraping code that is being caught by anti scraping measures on a site I am looking at:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import Selector
from scrapy.item import Item
from scrapy.spider import BaseSpider
from scrapy import log
from scrapy.cmdline import execute
from scrapy.utils.markup import remove_tags
import time


class ExampleSpider(CrawlSpider):
    name = "goal3"
    allowed_domains = ["whoscored.com"]
    start_urls = ["http://www.whoscored.com/"]
    rules = [Rule(SgmlLinkExtractor(allow=()), 
                  follow=True),
             Rule(SgmlLinkExtractor(allow=()), callback='parse_item')
    ]
    def parse_item(self,response):
        self.log('A response from %s just arrived!' % response.url)
        scripts = response.selector.xpath("normalize-space(//title)")
        for scripts in scripts:
            body = response.xpath('//p').extract()
            body2 = "".join(body)
            print remove_tags(body2).encode('utf-8')  
            time.sleep(5)

execute(['scrapy','crawl','goal3'])

In order to stop this from happening, I have tried adding a basic 'time.sleep()' function to slow down the rate at which submissions are made. However, when running the code via Command Prompt, this function does not seem to be having any kind of effect. The code continues to run at the same speed and thus all the requests come back as HTTP 403.

Can anyone see why this might not be working?

Thanks

Answer 1

Don't reinvent the wheel. DOWNLOAD_DELAY setting is what you are looking for:

The amount of time (in secs) that the downloader should wait before downloading consecutive pages from the same website. This can be used to throttle the crawling speed to avoid hitting servers too hard.

There are other techniques like rotating User Agents, IP addresses, see more at Avoid Getting Banned section.

Also, make sure you know what are the Terms of Use of the web-site. Make sure they don't state against web-crawling and whether the site provides API or not.