I've been trying to figure out how to iterate pages to scrape multiple news articles.
This is the page I want to scrape: (and its following pages) https://www.startribune.com/search/?page=1&q=China%20COVID-19&refresh=true
I tried out the below code, but it doesn't return a correct result:
def scrape(url):
user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko'}
urls = [f"{url}{x}" for x in range(1,10)]
params = {
'q': 'China%20COVID-19'
}
for page in urls:
response = requests.get(url=page,
headers=user_agent,
params=params)
print(page)
print(scrape('https://www.startribune.com/search/'))
Please suggest improvements or solutions!
The results I expect are:
https://www.startribune.com/search/?page=1&q=China%20COVID-19&refresh=true
https://www.startribune.com/search/?page=2&q=China%20COVID-19&refresh=true
...
https://www.startribune.com/search/?page=9&q=China%20COVID-19&refresh=true
As mentioned in the comments, make sure the params
are complete:
def scrape(url):
user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko'}
params = {
'q': 'China%20COVID-19',
'refresh': 'true',
}
for page_no in range(1, 10):
params['page'] = page_no
response = requests.get(url=url,
headers=user_agent,
params=params)
print(response.request.url)
# https://www.startribune.com/search/?q=China%2520COVID-19&refresh=true&page=1
scrape('https://www.startribune.com/search/')
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.