how to use scrapy-redis pipeline?

Question

I am using scrapy-redis now, and I am ok with it, and I am success to crawl in different computer by using the same redis server. But I don't understand how to use the scrapy-redis pipeline properly. In my understanding, I think I need another script than the spiders to deal with the item in the redis pipeline list, then I can do stuffs like store them into the database. Do I understand right, do I have to write another script, which is somehow dependent from the spider?

Answer 1

The pipeline is a different script, yes. In the settings file you can enable the pipeline. A pipeline can be used to store the crawled results in any database you want.