如何隐藏攻击性爬虫？

Question

I'm planning to crawl a specific site. I have 3000 specific pages that I want to crawl once every few months. I've created a crawler, but I don't want to be banned from the site.

Is there a way to reduce the aggressiveness of the crawler or hide it in some way so not to be "noticed" or cause issues for the provider/website that I'm crawling?

A delay is possible, but if I set it to random 10-30 second delay per page then it will take forever.

ANy tips or guidelines to make an acceptable crawler?

Answer 1

One more solution is to use PROXY server provider ( like this one for example ) and rotate IP address every X requests. This particular provider has an API to retrieve IPs on the fly. cURL can be used for this purpose easily if speaking about PHP .

This technique works in most cases, but it requires a bit more planning and tuning. Anyway you will face some limitations. It can be as time issue as well as the number of requests per period what is almost the same issue as time ones. Or you will need more proxy servers to satisfy your time requirements.

And read attentively TOS of providers. This particular provider doesn't allow you to be banned by Google and some other sites. Otherwise your account will be banned also.

Answer 2

"Acceptable" is a relative term. Some site owners have enough processing power and bandwidth that they don't think scanning 3000 pages per hour is "aggressive". Some site owners struggle for bandwidth or processing power and can't keep up with 3000 page reads per day.

If you want to read pages and get current contents, then you must read the pages. There's no shortcut to that.