Schedule scrapy spiders to run every N minutes

Question

I need help to schedule my spiders runs every N minutes. Early I see most people used reactor.callLater and reactor.run for this purpose but it seems there is no possibilities to use these functionalities now. How can I schedule it programmaticaly?

def run_crawl():
    """
    Run a spider within Twisted. Once it completes,
    wait 5 seconds and run another spider.
    """
    runner = CrawlerRunner(get_project_settings())
    runner.crawl(SpiderA)
    runner.crawl(SpiderB)
    deferred = runner.join()
    deferred.addCallback(reactor.callLater, 5, run_crawl)
    return deferred

run_crawl()
reactor.run()

At this moment my crawler scheduled by Windows task scheduler, but I want to schedule it programmatically.

Answer 1

You could give a try with an external module named schedule :
schedule github link

Tell me if it doesn't fit your needs.