在 python 中抓取特定日期范围的 Google 搜索结果

Question

I am trying to build a system that allows me to pull Google search results of specific date range to my local machine. I tried the following code:

import requests, bs4

res =
requests.get('https://www.google.com/search?q=nepal+OR+नेपाल&tbs=cdr:1,cd_min:4/1/2020,cd_max:4/8/2020')
res.raise_for_status()

soup = bs4.BeautifulSoup(res.text, 'html.parser') 
linkElements = soup.select('div')

 open('output.html', 'w').write(res.text)

It generates the result as shown in image shown in the following image. No date range

It is showing a normal search result for 'nepal OR नेपाल' but I need the search results for a date range lets say April 1, 2020 to April 8, 2020 as shown in the image

With some research, I came to know that this can be done by the use of APIs such as SerpAPI but that will cost me 50$ per month.

Is there any other way I could do this?

Answer 1

I would suggest you try out selenium unlike requests it can interact with the page once it's there and you can execute JavaScript commands to set a date range.