Scrapy：无法安排

Question

I want to run a spider every couple of minutes. I put the following script in my project that I want to call for this purpose.

import schedule, os
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

def job():
      process = CrawlerProcess(get_project_settings())
      process.crawl('amazon_spider')
      process.start() # error: twisted.internet.error.ReactorNotRestartable 
      #process.start(stop_after_crawl=False)  #process get stuck

while True:
     schedule.run_pending()
     schedule.every().minutes.do(job)

With this appoach the process get the following error:

twisted.internet.error.ReactorNotRestartable or stuck if I put process.start(stop_after_crawl=False)

From a previous stackoverflow posted I also try this :

from twisted.internet import reactor
from amazon.spiders.amazon_spider import AmazonSpider
from scrapy.crawler import CrawlerRunner

def run_crawl():

    runner = CrawlerRunner({
    'USER_AGENT': 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)',
    })
     deferred = runner.crawl(AmazonSpider)
     deferred.addCallback(reactor.callLater, 10, run_crawl)

     return deferred

     run_crawl()
     reactor.run()   

The process get stuck again in the middle of the parse function . I really don't know what to try next . If you have an idea please let me know. Thank you in advance .... ( By the way , it is not a duplicate , since the posts on the same subject didn#t solve my problem.

Answer 1

I use apscheduler

pip install apscheduler

then

# -*- coding: utf-8 -*-
from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings
from apscheduler.schedulers.twisted import TwistedScheduler

from Demo.spiders.baidu import YourSpider

process = CrawlerProcess(get_project_settings())
scheduler = TwistedScheduler()
scheduler.add_job(process.crawl, 'interval', args=[YourSpider], seconds=10)
scheduler.start()
process.start(False)