抓狂地管理多个蜘蛛

Question

I am creating an aggregator and I started with scrapy as my initial tool set. First I only had a few spiders, but as the project grows it seems like I may have hundreds or even a thousand different spiders as i scrape more and more sites. What is the best way to manage these spiders as some websites only need to be crawled once, some on a more regular basis? Is scrapy still a good tool when dealing with so many sites or would you recommend some other technology.

Answer 1

You can check out the project scrapely , that is from the creators of scrapy. But, as far as I know, it is not suitable for parsing sites containing javascript (more precisely, if the parsed data is not generated by javascript).