如何提取显示在 chrome 的开发人员工具上的 href 属性，而不是 BeautifulSoup 的 output

Question

I'm trying to scrape a website to compile and summarize news, using Python's request and bs4. The links (href) that I'm trying to access appear on Chrome's developer tools with this path:

"/html/body/div/div/div/main/article/div/div/section/div/section/div/div[3]/ul/li[1]/a"

I tried everything to extract them, but I realized that Python's html output doesn't go down to that level. It stays at:

"/html/body/div/div/div/main/article/div/div/section/div/section"

I'm using the following code:

import requests
from bs4 import BeautifulSoup
url = 'https://www.gp.com/news'
response = requests.get(url)
soup = BeautifulSoup(response.content, 'html.parser')
for url in soup.find_all('a'):
    print(url.get('href'))

I'd really appreciate any help you can give me because I'm completely out of ideas. Also, I'm completely new to programing, so would appreciate your answers to be dumbed down as much as possible.

Thanks a lot in advance!

Answer 1

The requests module doesn't render javascript -- you have to use requests-html ( https://github.com/psf/requests-html ). You can see the difference if you open the page in a browser and then look at the source (typically CTRL-U). It will be different than what you can view using Developer Tools because the latter includes content that is rendered by javascript.