使用 BeautifulSoup 抓取 Google 搜索结果

Question

My goal is to web scrape Google search results using BeautifulSoup. I am using Anaconda Python and use Ipython as the IDE console. Why don't I get an ouptput when run the following command?

def google_scrape(query):
    address = "http://www.google.com/search?q=%s&num=100&hl=en&start=0" % (urllib.quote_plus(query))
    request = urllib2.Request(address, None, {'User-Agent':'Mosilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11'})
    urlfile = urllib2.urlopen(request)
    page = urlfile.read()
    soup = BeautifulSoup(page)

    linkdictionary = {}

    for li in soup.findAll('li', attrs={'class':'g'}):
        sLink = li.find('a')
        print sLink['href']
        sSpan = li.find('span', attrs={'class':'st'})
        print sSpan

    return linkdictionary

if __name__ == '__main__':
    links = google_scrape('english')

Answer 1

You are never adding anything to linkedDictionary

def google_scrape(query):
    address = "http://www.google.com/search?q=%s&num=100&hl=en&start=0" % (urllib.quote_plus(query))
    request = urllib2.Request(address, None, {'User-Agent':'Mosilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.57 Safari/536.11'})
    urlfile = urllib2.urlopen(request)
    page = urlfile.read()
    soup = BeautifulSoup(page)

    linkdictionary = {}

    for li in soup.findAll('li', attrs={'class':'g'}):
        sLink = li.find('a')
        sSpan = li.find('span', attrs={'class':'st'})

        linkeDictionary['href'] = sLink['href']
        linkedDictionary['sSpan'] = sSpan

    return linkdictionary

if __name__ == '__main__':
    links = google_scrape('english')

Answer 2

The problem as Cody Bouche mentioned is that nothing has been adding to the dict() . In my opinion, you'll have hard times updating your dict if you haven't change {} (dict) to [] (array).

Appending to array is much simpler ( note: I could be wrong here, it's just a personal opinion from previous experience ).

To make it work in a simple maner, you need to change dict to array {} --> [] and then use .append({}) to append to list()

Code and example in the online IDE :

def google_scrape(query):
    html = requests.get(f'https://www.google.com/search?q={query}', headers=headers).text
    soup = BeautifulSoup(html, 'lxml')

    data = []

    for container in soup.findAll('div', class_='tF2Cxc'):
        title = container.select_one('.DKV0Md').text
        link = container.find('a')['href']

        data.append({
          'title': title,
          'link': link,
        })
        print(f'{title}\n{link}')

    print(json.dumps(data, indent=2))

google_scrape('english')

# part of the outputs:
'''
English language - Wikipedia
https://en.wikipedia.org/wiki/English_language
[
  {
    "title": "English language - Wikipedia",
    "link": "https://en.wikipedia.org/wiki/English_language"
  },
]
'''

If you still want to append to dict() then this is one of the ways of approaching this ( only part of the for loop shown ):

for container in soup.findAll('div', class_='tF2Cxc'):

    data_dict = {}

    title = container.select_one('.DKV0Md').text
    link = container.find('a')['href']
    # creates title key and assigns title value
    data_dict['title'] = title
    # creates link key and assigns link value
    data_dict['link'] = link

    print(json.dumps(data_dict, indent = 2))

# part of the output:
'''
{
  "title": "Minecraft Official Site | Minecraft",
  "link": "https://www.minecraft.net/en-us/"
}
'''

---

To get a dict output right away you can do the same thing using Google Search Engine Results API from SerpApi. It's a paid API with a free trial of 5,000 searches.

Essentially, it's doing the same thing as the code above, but you don't to figure out how to do certain things or trying to understand how to scrape certain element, it's already done for the end-user with a JSON output so the only thing that needs to be done is to iterate over a JSON and get the desired output.

Code to integrate:

from serpapi import GoogleSearch
import json

params = {
  "api_key": "YOUR_API_KEY",
  "engine": "google",
  "q": "minecraft",
}

search = GoogleSearch(params)
results = search.get_dict()

for result in results['organic_results']:
  print(json.dumps(result, indent = 2, ensure_ascii = False))

# part of the json output:
'''
{
  "position": 1,
  "title": "Minecraft - Aplikasi di Google Play",
  "link": "https://play.google.com/store/apps/details?id=com.mojang.minecraftpe&hl=in&gl=US",
  "displayed_link": "https://play.google.com › store › apps › details › id=co...",
  "rich_snippet": {
    "top": {
      "detected_extensions": {
        "skor": 46,
        "suara": 4144655,
        "us": 749
      },
      "extensions": [
        "Skor: 4,6",
        "‎4.144.655 suara",
        "‎US$7,49",
        "‎Android",
        "‎Game"
    ]
  }
}
'''

Disclaimer, I work for SerpApi.