我可以使用Python从Google搜索中抓取所有URL结果而不会被阻止吗？

Question

I realize that versions of this question have been asked and I spent several hours the other day trying a number of strategies.

What I would like to is use python to scrape all of the URLs from a google search that I can use in a separate script to do text analysis of a large corpus (news sites mainly). This seems relatively straightforward, but none of the attempts I've tried have worked properly.

This is as close as I got:

from google import search

for url in search('site:cbc.ca "kinder morgan" and "trans mountain" and protest*', stop=100):
    print(url)

This returned about 300 URLs before I got kicked. An actual search using these parameters provides about 1000 results and I'd like all of them.

First: is this possible? Second: does anyone have any suggestions to do this? I basically just want a txt file of all the URLs that I can use in another script.

Answer 1

It seems that this package uses screen scraping to retrieve search results from google, so it doesn't play well with Google's Terms of Service which could be the reason why you've been blocked.

The relevant clause in Google's Terms of Service :

Don't misuse our Services. For example, don't interfere with our Services or try to access them using a method other than the interface and the instructions that we provide. You may use our Services only as permitted by law, including applicable export and re-export control laws and regulations. We may suspend or stop providing our Services to you if you do not comply with our terms or policies or if we are investigating suspected misconduct.

I haven't been able to find a definite number, but it seems like their limit for the number of search queries a day is rather strict too - at 100 search queries / day on their JSON Custom Search API documentation here .

Nonetheless, there's no harm trying out other alternatives to see if they work better:

1. BeautifulSoup
2. Scrapy
3. ParseHub - this one is not in code, but is a useful piece of software with good documentation. Link to their tutorial on how to scrape a list of URLs .