如何在 Scrapy 中使用多个相同格式的 URL 开始抓取

Question

My Scrapy spider needs to start with URLs of the following format:

https://catalog.loc.gov/vwebv/search?searchArg={$variable}&searchCode=GKEY%5E*&searchType=1&limitTo=none&fromYear=&toYear=&limitTo=LOCA%3Dall&limitTo=PLAC%3Dall&limitTo=TYPE%3Dall&limitTo=LANG%3Dall&recCount=1000'

where $variable is a parameter that can be fed with as many values as possible (possibly even 1000 possible values).

How do I implement this?

Answer 1

You could overwrite the start_requests method to something like:

def start_requests(self):
    base_url = 'https://catalog.loc.gov/vwebv/search?...'
    variables = [...]
    for variable in variables:
        url = base_url.format(variable)
        yield Request(url)