[英]how to pass an argument into a scrapy spider and init it from within python
[英]Python Scrapy | How to pass the response to the main function from the spider
我曾尝试广泛搜索解决方案,但可能没有使用正确的关键字。 I am aware that I can use the shell to work with CSS and XPath selectors immediately, but I would like to know if this is possible to do in the IDE environment outside of the spider class, namely in another cell.
示例代码:
class ExampleSpider(scrapy.Spider):
name = "exampleSpider"
start_urls = ["https://www.example.com"]
def parse(self, response):
URL = "www.example.com/1/"
yield response
然后我希望能够在另一个单元格中使用此响应和选择器:
table_rows = response.xpath("//div[@class='example']/table/tr") # produces error
print(table_rows.xpath("td[4]//text()")[0] .get()
它产生错误: NameError: name 'response' is not defined
任何帮助/指导将不胜感激。
如果我理解正确,您希望蜘蛛返回响应并在主脚本中解析它?
主要.py:
from scrapy.crawler import CrawlerProcess, CrawlerRunner
from scrapy.utils.project import get_project_settings
from scrapy.signalmanager import dispatcher
from scrapy import signals
def spider_output(spider):
output = []
def get_output(item):
output.append(item)
dispatcher.connect(get_output, signal=signals.item_scraped)
settings = get_project_settings()
settings['USER_AGENT'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36'
process = CrawlerProcess(settings)
process.crawl(spider)
process.start()
return output
if __name__ == "__main__":
spider = "exampleSpider"
response = spider_output(spider)
response = response[0]['response']
title = response.xpath('//h3//text()').get()
price = response.xpath('//div[@class="card-body"]/h4/text()').get()
print(f"Title: {title}")
print(f"Price: {price}")
我们启动蜘蛛并将生成的项目附加到output
。 由于output
只有一个值,我们不必循环,只需取第一个值response[0]
。 然后我们想从键response
中获取值,所以response = response[0]['response']
。
蜘蛛.py:
import scrapy
class ExampleSpider(scrapy.Spider):
name = "exampleSpider"
start_urls = ['https://scrapingclub.com/exercise/detail_basic/']
def parse(self, response):
yield {'response': response}
在这里,我们返回一个带有响应的项目。
步骤是:main->spider_output->spider->将响应项返回到spider_output->将项目附加到output列表->将output返回给main->从Z78E6221F6393D14CE6DZ中获取响应->解析响应。
Output:
Title: Long-sleeved Jersey Top
Price: $12.99
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.