Can't stop Scrapy from inside pipeline.py

Question

So I am writing a validator for my Scrapy data and want the spider to stop crawling if the data is in an incorrect format. I am doing this in Pipeline.py.

I have already tried calling CloseSpider, close_spider and crawler._signal_shutdown(9,0) (which have been used in other tutorials but for some reason don't work in pipeline.py). I am aware that the spider does not finish straight away but all the above methods seem to yield some sort of error. Is there just a straight forward way to kill the crawler?

Answer 1

Your scraper still working because of its schedule some amount of request and CloseSpider was created for a graceful shutdown. It means that all request that is in progress will be canceled or done before crawler will be closed. Do you call close_spider() in this way

Answer 2

只需尝试下面的代码即可消除Spider的过程：

raise SystemExit