在Python刮网页无法找到文本

Question

I am trying to scrape this webpage using BeautifulSoup in Canopy 1.6.1. What I want to be able to return is the "College Dominator" rating as well as the "Breakout Age" rating. I think that the reason this isn't working is because it's behind Javascript, but I don't know how to find that information from the scraped data. Please help!

Answer 1

The page you provided is rendered within the browser with JavaScript (Angular). The actual sent HTML doesn't include the information about "College Dominator". Thus you'll have to render it before you can parse it, I'd recommend using a library more suited for parsing client side rendered pages. Requests-HTML is one such option. With that library you could achieve your results like this:

r = session.get("https://www.playerprofiler.com/nfl/larry-fitzgerald/")
r.html.render()
college_dominator = r.html.search("College Dominator {percentage}% ({rank}th)")
# {"rank": 96, "percentage": 51.3}