如何限制非谷歌搜索引擎机器人抓取率，以便他们不会超过外部API请求限制？

Question

I'm building for a client an amazon-affiliate website that uses Amazon Product API to fetch data from their catalogue.

Amazon have a 1 per sec request limit.

Google allow configuring the googlebot's crawl rate via their Webmasters Tools, so there is no issue with them.

I need advice about how to go about treating other search engine crawl bots. What would be a good way to avoid as much possible, exceeding Amazon's API rate limit due to bot crawling?

Answer 1

PHP

If you want to follow PHP approach, follow my answer on php redirect url with og metatag (open graph)

robots.txt

I would go with robots.txt file as it's fairly simple and saves time. Generally, all bots respect, and abide the rules in this file. Create a file named robots.txt (type: text/plain) with the following rules

User-agent: * 
Disallow: /path/to/dir/

Asterisk * is a wildcard indicating every user agent.

Disallow: /path/to/dir/

Disallow rule defines the paths you want bots not to crawl. You can have multiple lines for different user-agents.

User-agent: Googlebot
Disallow: /path1/

User-agent: Facebookhit
Disallow: /path2/

Above will allow to access to /path2/ to Googlebot but not Facebookhit , and vice versa. You can read more here