Web Scraping and http too many requests answer
Question
在执行Web抓取项目时，要提取我需要的所有信息，我需要发出许多请求，但是服务器在大约50个请求后以429状态代码回答，该状态代码指示错误“太多请求”，如何处理呢？
2 answers
solution1
0 2013-07-10 06:59:39
例如，也许您应该尝试使用HideMyIP，并每5或10分钟更改一次IP
solution2
0 ACCPTED 2013-07-10 16:52:26
My guess is you'll either need to slow down the frequency of your requests, or anonymize. We have a couple of articles on our site that will explain your options if you decide to go the anonymization route:
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
C# parallel HTTP GET request "Too many requests" error
.NET web service with many requests
Too many requests exception on HttpWebRequest.GetResponse()
Azure SignalR Error: (429) Too Many Requests
HttpClient Error too many requests rate limit
C# (429) Too Many Requests
Prevent calling a web service too many times
Get a custom answer, an object for an HTTP Post request in a Web API
Send multiple http requests to a web service
Beginning to understand web sockets and http requests (and STOMP)
Related Tags