美丽的汤和findAll（）过程

Question

I am attempting to scrape data from a site using the following code. The site required the decode method and I followed a @royatirek solution. My problem is that container_a ends up containing nothing. I use a similar method on few other sites and it works. But on this and a couple of other sites my container_a variable remains an empty list. Cheers

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup as soup
my_url = 'http://www.news.com.au/sport/afl-round-3-teams-full-lineups-and- 
the-best-supercoach-advice/news-story/dfbe9e0e68d445e07c9522a138a2b824'
req = Request(my_url, headers={'User-Agent': 'Mozilla/5.0'})
web_byte = urlopen(req).read()
webpage = web_byte.decode('utf-8')
page_soup = soup(web_byte, "html.parser")
container_a = page_soup.findAll("div",{"class":"fyre-comment-wrapper"})

Answer 1

The content you want to parse is being dynamically loaded by JavaScript and therefore requests won't do the job for you. You could use selenium and ChromeDriver or any other driver for that:

from selenium import webdriver
from bs4 import BeautifulSoup

driver = webdriver.Chrome()
driver.get("http://www.news.com.au/sport/afl-round-3-teams-full-lineups-and-the-best-supercoach-advice/news-story/dfbe9e0e68d445e07c9522a138a2b824")

You can then proceed with the use of bs4 as you wish by accessing the page source using .page_source :

page_soup = BeautifulSoup(driver.page_source, "html.parser")
container_a = page_soup.findAll("div",{"class":"fyre-comment-wrapper"})