通过HTTP请求运行Python Scrapy脚本

Question

I'm looking for an example to run scrapy script via HTTP request. I'm planing to send url as a parameter that i need to crawl, via GET or POST method. How can i do that.

Answer 1

You should use scrapyd .

Link to the GitHub project page .

Once you are using scrapyd you can use this api to scedule a crawl.

Answer 2

Try something like that.

from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy import log, signals
from testspiders.spiders.followall import FollowAllSpider
from scrapy.utils.project import get_project_settings

spider = FollowAllSpider(domain='url.com')
settings = get_project_settings()
crawler = Crawler(settings)
crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
crawler.configure()
crawler.crawl(spider)
crawler.start()
log.start()
reactor.run()