Scrapy從蜘蛛中的mongodb獲取數據

Question

我創建了一個spider ，從列表頁面抓取網站products 。 有什么方法可以連接到我的spider mongodb 。 獲取存儲的urls列表，並抓取這些網址

謝謝..

Answer 1

您可以在spider本身中從mongodb import urls 。

from pymongo import MongoClient()
import scrapy

class Myspider(scrapy.Spider):

    def __init__(self):
        self.db = MongoClient() #you can add db-url and port as parameter to MongoClient(), localhost by default
        self.urls = self.db.db_name.collection.find() #use appropriate finding criteria here according to the structure of data resides in that collection

    def parse(self, response):
        # other codes
        for url in self.urls: # self.urls refers to the url's fetched from db
            #do operations with the urls