Pytrends：请求失败：Google 返回了代码为 429 的响应

Question

I'm using Pytrends to extract Google trends data, like:

from pytrends.request import TrendReq
pytrend = TrendReq()
pytrend.build_payload(kw_list=['bitcoin'], cat=0, timeframe=from_date+' '+today_date)

And it returns an error:

ResponseError: The request failed: Google returned a response with code 429.

I made it yesterday and for some reason it doesn't work now! The source code from github failed too:

pytrends = TrendReq(hl='en-US', tz=360, proxies = {'https': 'https://34.203.233.13:80'})

How can I fix this? Thanks a lot!

Answer 1

This one took a while but it turned out the library just needed an update. You can check out a few of the approaches I posted here, both of which resulted in Status 429 Responses:

https://github.com/GeneralMills/pytrends/issues/243

Ultimately, I was able to get it working again by running the following command from my bash prompt:

Run:

pip install --upgrade --user git+https://github.com/GeneralMills/pytrends

For the latest version.

Hope that works for you too.

EDIT:

If you can't upgrade from source you may have some luck with:

pip install pytrends --upgrade

Also, make sure you're running git as an administrator if on Windows.

Answer 2

I had the same problem even after updating the module with pip install --upgrade --user git+https://github.com/GeneralMills/pytrends and restart python.

But, the issue was solved via the below method:

Instead of

pytrends = TrendReq(hl='en-US', tz=360, timeout=(10,25), proxies=['https://34.203.233.13:80',], retries=2, backoff_factor=0.1, requests_args={'verify':False})

Just ran:

pytrend = TrendReq()

Hope this can be helpful!

Answer 3

TLDR; I solved the problem with a custom patch

Explanation

The problem comes from the Google bot recognition system. As other similar systems do, it stops serving too frequent requests coming from suspicious clients. Some of the features used to recognize trustworthy clients are the presence of specific headers generated by the javascript code present on the web pages. Unfortunately, the python requests library does not provide such a level of camouflage against those bot recognition systems since javascript code is not even executed. So the idea behind my patch is to leverage the headers generated by my browser interacting with google trends. Those headers are generated by the browser meanwhile I am logged in using my Google account, in other words, those headers are linked with my google account, so for them, I am trustworthy.

Solution

I solved in the following way:

1. First of all you must use google trends from your web browser meanwhile you are logged in with your Google Account;
2. In order to track the actual HTTP GET made: (I am using Chromium) Go into "More Tools" -> "Developers Tools" -> "Network" tab.
3. Visit the Google Trend page and perform a search for a trend; it will trigger a lot of HTTP requests on the left sidebar of the "Network" tab;
4. Identify the GET request (in my case it was /trends/explore?q=topic&geo=US) and right-click on it and select Copy -> Copy as cURL;
5. Then go to this page and paste the cURL script on the left side and copy the "headers" dictionary you can find inside the python script generated on the right side of the page;
6. Then go to your code and subclass the TrendReq class, so you can pass the custom header just copied:

from pytrends.request import TrendReq as UTrendReq
GET_METHOD='get'

import requests

headers = {
...
}


class TrendReq(UTrendReq):
    def _get_data(self, url, method=GET_METHOD, trim_chars=0, **kwargs):
        return super()._get_data(url, method=GET_METHOD, trim_chars=trim_chars, headers=headers, **kwargs)

7. Remove any "import TrendReq" from your code since now it will use this you just created;
8. Retry again;
9. If in any future the error message comes back: repeat the procedure. You need to update the header dictionary with fresh values and it might happen you will have to solve a captcha mechanism.

Answer 4

通过 pip install 运行升级命令后，您应该重新启动 python 内核并重新加载 pytrend 库。