Scrapy如何減慢數據請求

Question

當我想獲得更少的數據時沒有問題，但是當我想獲得更多的數據時，我遇到了錯誤 429。我查看了 scrapy 文檔，但沒有幫助。 我認為問題是速度。 因為在 6 秒內響應計數為 210，我不知道如何減慢它。 順便說一句，我試過DOWNLOAD_DELAY = [1]但也沒有用。 這是代碼：

class WanikaniSpider(scrapy.Spider):
name = 'japandict'
allowed_domains = ['www.wanikani.com']         
url = ('https://www.wanikani.com/kanji/')
start_urls = []
kanjis = ["愛", "暗", "位", "偉", "易", "違", "育", "因", "引", "泳", "越", "園", "演", "煙", "遠", "押", "橫", "王", "化", "加", "科", "果", "過", "解", "回", "皆", "絵", "害", "格", "確", "覚", "掛", "割", "活", "寒", "完", "官", "感", "慣", "観", "関", "顔", "願", "危", "喜", "寄", "幾", "期", "機", "規", "記", "疑", "議", "客", "吸", "求", "球", "給", "居", "許", "供", "共", "恐", "局", "曲", "勤", "苦", "具", "偶", "靴", "君", "係", "形", "景", "経", "警", "迎", "欠", "決", "件", "権", "険", "原", "現", "限", "呼", "互", "御", "誤", "交", "候", "光", "向", "好", "幸", "更", "構", "港", "降", "號", "合", "刻", "告", "込", "困", "婚", "差", "座", "最", "妻", "才", "歳", "済", "際", "在", "罪", "財", "昨", "察", "殺", "雑", "參", "散", "産", "賛", "殘", "市", "師", "指", "支", "資", "歯", "似", "次", "治", "示", "耳", "辭", "式", "識", "失", "実", "若", "取", "守", "種", "酒"]
liste=[]
for kanji in kanjis:
    liste.append(kanji)
    nurl = url + kanji
    start_urls.append(nurl)
file =  open("n3kanji.txt","w",encoding="utf-8")
file1 = open("n3onyomi.txt","w",encoding="utf-8")
file2 = open("n3kunyomi.txt","w",encoding="utf-8") 
file3 = open("n3meanings.txt","w",encoding="utf-8")   
       
def parse(self, response):
    print(response.url)
    kanjiicon = response.xpath('//*[@id="main"]/body/div[1]/div[3]/div/div/header/h1/span/text()').getall()
    meanings = response.xpath('//*[@id="meaning"]/div[1]/p/text()').getall()
    reading = response.xpath('//*[@id="reading"]/div') 
    for onkun in reading:
        onyomi= onkun.xpath('//*[@id="reading"]/div/div[1]/p/text()').getall()
        kunyomi= onkun.xpath('//*[@id="reading"]/div/div[2]/p/text()').getall()                
    for x in kanjiicon:
        yield{'kanjiicon': x.strip()}
        self.file.write(x + "\n")
        self.file.close
    for y in onyomi:
        yield{'onyomi': y.strip()}
        self.file1.write(y + "\n" +"\r")
        self.file1.close
    for z in kunyomi:
        yield{'kunyomi': z.strip()}
        self.file2.write(z + "\n" + "\r")
        self.file2.close
    for m in meanings:
        yield{'meanings': m.strip()}
        self.file3.write(m + "\n")
        self.file3.close`

謝謝你的幫助。

Answer 1

您可以通過在蜘蛛上或在項目的主settings.py文件中設置自定義設置來使用多種方式減慢蜘蛛的速度。

一些設置包括 CONCURRENT_REQUESTS、DOWNLOAD_DELAY、CONCURRENT_REQUESTS_PER_DOMAIN、CONCURRENT_REQUESTS_PER_IP、AUTOTHROTTLE_ENABLED

例如：

class WanikaniSpider(scrapy.Spider):
    name = 'japandict'
    allowed_domains = ['www.wanikani.com']         
    url = ('https://www.wanikani.com/kanji/')
    start_urls = []
    custom_settings = {
        CONCURRENT_REQUESTS: 1,
        DOWNLOAD_DELAY: 10,
        CONCURRENT_REQUESTS_PER_DOMAIN: 1,
        AUTOTHROTTLE_ENABLED: True,
        AUTOTHROTTLE_START_DELAY: 3,
        AUTOTHROTTLE_TARGET_CONCURRENCY: 1
    }