需要限制刮板以仅每4秒从URL的python列表中访问网站-scraperapi，scrapy，python

Question

Scraping a python list of web domains, would like to put a 4 second delay between each scrape in order to comply with robots.txt. Would like each iteration to run asynchronously, so the loop will continue running every 4 seconds, irrespective of whether the scrape for that particular page has finished.

I have tried implementing asyncio gather, coroutine and was beginning to attempt throttling. However my solutions were getting very complex and I believe there must be a simpler way, or that I am missing something here. In one of my past versions, I just put a sleep(4) inside the for in loop, though to my updated understanding this is bad as it sleeps the entire interpreter and other loops won't run asynchronously at that time?

import requests
import csv
csvFile = open('test.csv', 'w+')

urls = [
    'domain1', 'domain2', 'domain3'...
];

YOURAPIKEY = <KEY>; 
from bs4 import BeautifulSoup

writer = csv.writer(csvFile)
writer.writerow(('Scraped text', 'other info 1', 'other info 2'))

lastI = len(urls) - 1

for i, a in enumerate(urls):
  payload = {'api_key': YOURAPIKEY, 'url': a}
  r = requests.get('http://api.scraperapi.com', params=payload)
  soup = BeautifulSoup(r.text, 'html.parser')
  def parse(self, response):
    scraper_url = 'http://api.scraperapi.com/?api_key=YOURAPIKEY&url=' + a
    yield scrapy.Request(scraper_url, self.parse)

  price_cells = soup.select('.step > b.whb:first-child')
  lastF = len(price_cells) - 1
  for f, price_cell in enumerate(price_cells):
    writer.writerow((price_cell.text.rstrip(), '...', '...'))
    print(price_cell.text.rstrip())

    if (i == lastI and f == lastF):
      print('closing now')
      csvFile.close()

No errors with the above code that I can tell. Just want each loop to keep running at 4s intervals and the results coming back from the fetch to be saved to the excel document ad hoc.

Answer 1

In scrapy the appropriate setting in the setting.py file would be:

DOWNLOAD_DELAY

The amount of time (in secs) that the downloader should wait before downloading consecutive pages from the same website. This can be used to throttle the crawling speed to avoid hitting servers too hard. Decimal numbers are supported.

DOWNLOAD_DELAY = 4 # 4s of delay

https://doc.scrapy.org/en/latest/topics/settings.html