[英]Python - how to add response from scrapy.request from yield into an array
[英]How to determine if the generator returned from `yield scrapy.Request` has any data?
在Scrapy 教程中,蜘蛛从class="next"
中提取下一页链接并抓取它们 -
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'http://quotes.toscrape.com/page/1/',
]
def parse(self, response):
for quote in response.css('div.quote'):
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('span small::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
}
next_page = response.css('li.next a::attr(href)').get()
if next_page is not None:
yield response.follow(next_page, callback=self.parse)
就我而言,我在从网络服务器下载的文件中找不到下一页链接,但我知道格式是response.url
与/page/[page number]/
连接。 不产生报价的请求页面仍会返回response
,例如 - 未找到报价! . 由于下一页的数量通常少于 20,我可以通过将蜘蛛的最后 3 行替换为 - 来遍历所有可能的 url
for page_num in range(2, 20):
yield response.follow(f"/page/{page_num}/", callback=self.parse)
然而,这迫使蜘蛛请求不产生报价的页面(例如http://quotes.toscrape.com/page/11到 20)。 在请求不产生引号的第一页后,如何调整我的蜘蛛以终止page_num
循环? (如http://quotes.toscrape.com/page/11 )
伪代码 -
page_num = 2
while (quotes are yielded from the response):
yield response.follow(f"/page/{page_num}/", callback=self.parse)
page_num += 1
您可以使用response.css('..')
结果作为下一页的条件。
在这种情况下,您的代码将如下所示:
import scrapy
class QuotesSpider(scrapy.Spider):
name = "quotes"
start_urls = [
'http://quotes.toscrape.com/page/1/',
]
def parse(self, response):
page_num = get_pagenumber_from_url(response.url)
quotes_sel = response.css('div.quote'):
# quotes_sel - will be SelectorList if page have item data
# quotes_sel - will be None if page doesn't have item data
for quote in quotes_sel:
yield {
'text': quote.css('span.text::text').get(),
'author': quote.css('span small::text').get(),
'tags': quote.css('div.tags a.tag::text').getall(),
}
if quotes_sel:
next_page_url = f"/page/{str(page_num+1)}"
yield response.follow(next_page_url , callback=self.parse)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.