美丽的汤不会返回结果

Question

Here is my code. it doesn't return any errors but it also doesn't return any results.

import requests
from bs4 import BeautifulSoup

googtrends = requests.get("https://www.google.com/trends/")
soup = BeautifulSoup(googtrends.content)
links = soup.find_all("a", {"class": "trending-story ng-isolate-scope"})

print links

I haven't solved this yet, I started working on something else instead but I'm going to first try it with selenium and the try using selenium with either phantom js or zombie js and if that still doesn't work I'll use pytrends but I just checked them out and you need a gmail account with, which I have but I would rather try getting it to work without an api first.

I will post back here when I get it working

Answer 1

Yes this page is being rendered by JS dynamically- let's have a try even changing the request header( it fails and likewise assures that JS is the cause! )

Testing code-

import requests
from bs4 import BeautifulSoup


my_headers={"Host": "www.google.com",
"User-Agent": "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language": "en-US,am;q=0.7,zh-HK;q=0.3",
"Accept-Encoding": "gzip, deflate",
"Cookie": "PREF=ID=1111111111111111:FF=0:LD=en:TM=1439993585:LM=1444815129:V=1:S=Zjbb3gK_m_n69Hqv; NID=72=F6UyD0Fr18smDLJe1NzTReJn_5pwZz-PtXM4orYW43oRk2D3vjb0Sy6Bs_Do4J_EjeOulugs_x2P1BZneufegpNxzv7rkY9BPHcfdx9vGOHtJqv2r46UuFI2f5nIZ1Cu4RcT9yS5fZ1SUhel5fHTLbyZWhX-yiPXvZCiQoW4FjZd-3Bwxq8yrpdgmPmf4ufvFNlmTd3y; OGP=-5061451:; OGPC=5061713-3:",
"Connection": "keep-alive"}


googtrends = requests.get("https://www.google.com/trends/",headers=my_headers)
my_content = googtrends.text.encode('utf-8')
soup = BeautifulSoup(my_content,'html.parser')
links = soup.find_all("a", {"class": "trending-story ng-isolate-scope"},href=True)

#Lets try if we are getting correct content from the site
# That site contains "Apple Inc.‬, ‪App Store‬‬" so let's check it in the got response

print 'Apple Inc.‬, ‪App Store‬‬' in my_content

# It prints false so website is being rendered by JS even header change does not affect

So try webdriver like selenium in Firefox, Chrome, PhantomJS etc that executes JS dynamically. Even better try API.