谷歌抓取杀死地方 api 配额

Question

We have a few API's on our site, one is a places API. When google spider crawls our site it will hit the quota for our places API.

I have reset the API over and over and its getting very tiring!

I also set my site to run 3 different API projects with the same APIs (google places) and used logic to make it use up one, switch to the next ect ect however; Even after now having 450,000 calls per day, by noon google search spider has killed all 3 API's!!!

this now makes it so that my users can no longer use any section that uses the places API, this is a HUGE problem!!! i am not being charged for the google hitting google API calls, however it is destroying the users experience at my site and will not be tolerated!

Please help right away!

I imagine it rests within googles hands to fix this bug within their system, there is really nothing i can personally do as you have read above that i have done everything i can for my users experience when visiting my site.

Answer 1

这不是他们系统中的错误，如果您有成千上万个唯一的 URL 都进行 API 调用，并且您没有阻止使用robots.txt （ 请参阅此处）抓取它们，那么这就是您网站中的错误。

Answer 2

I ended up solves this in a work around way, for anyone else having this issue here is what i did.

1) i have 3 API projects set up, each can make 150,000 calls a day

2) i have logic set up to look and see if the page is being accessed from a sprider like google bot

3) if the session is coming from a spider, the 3rd API key is set to be null

4)the system trys to use each API one by one, if the first result set is empty it tries number 2, then if 2 is empty it tries number 3

5) because the 3rd API key is set to null if a spider, this allows for those 150,000 calls to be set aside for a user, but now we have to stop crawl from crawling blank content

6) in the logic block that switches from trying API 1, then 2, then 3 I made php rewrite my robots.txt file, if API 1 is usable i set this:

file_put_contents('robots.txt', 'User-agent: * Disallow: ');

same for API 2, if API 3 is being used then i rewrite the robots.txt to become:

file_put_contents('robots.txt', 'User-agent: * Disallow: /');

this now has set aside 150,000 calls for users, the spiders can not use these 150,000 calls, and at the point that the other 300,000 calls have been exhausted, the site can no longer be crawled for the rest of the day by any spiders.

problem solved! told you id fix it myself if they couldnt.

oh and another note, because its google using google API's im not being charged for the 300,000 calls that google kills, i only get charged for that real users use up......pure perfection!