Scrapy从蜘蛛中的mongodb获取数据

Question

我创建了一个spider ，从列表页面抓取网站products 。 有什么方法可以连接到我的spider mongodb 。 获取存储的urls列表，并抓取这些网址

谢谢..

Answer 1

您可以在spider本身中从mongodb import urls 。

from pymongo import MongoClient()
import scrapy

class Myspider(scrapy.Spider):

    def __init__(self):
        self.db = MongoClient() #you can add db-url and port as parameter to MongoClient(), localhost by default
        self.urls = self.db.db_name.collection.find() #use appropriate finding criteria here according to the structure of data resides in that collection

    def parse(self, response):
        # other codes
        for url in self.urls: # self.urls refers to the url's fetched from db
            #do operations with the urls