Python - 在 Google 中搜索并收集标题和描述

Question

I need to search in Google a few words from a CSV file, and to collect from the google search - URL, google description and title.

I managed to write a script to search the words, collect only the URLs and store the results in a CSV file. I can't figure out how to collect - Title and google description. Also, I need to return - "missing results" if the word I search is not found.

from bs4 import BeautifulSoup
from googlesearch import search
import pandas as pd

keywords = pd.read_csv('keywords.csv', header=0, index_col=None)

#print(keywords['keyword'])

df = pd.DataFrame(columns=['keyword', 'url'])

for i in keywords['keyword']:
    print('Search results for keyword: ', i)
    count = 0
    for j in search(i, tld="co.in", num=10, stop=3, pause=2, lang='en'):
        count = count + 1
        print('URL number ',count, ': ', j)
        df = df.append({'keyword': i, 'url': j}, ignore_index=True)

df.to_csv(r'final_dataset.csv', index=False)

Answer 1

• specify user-agent in order not to be blocked by Google while making a request.What is my user-agent?
• try SelectorGadget Chrome extension to visually grab CSS selectors by clicking on the desired element.

Code and full example in the online IDE (note: the following code is not appending to .csv , instead it will overwrite existing file):

import pandas as pd
from bs4 import BeautifulSoup
import requests, lxml

headers = {
    'User-agent':
    "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.102 Safari/537.36 Edge/18.19582"
}

keywords = pd.read_csv('keywords.csv', header=0, index_col=None)

# collected data
data = []

for query in keywords['keywords']:
  html = requests.get(f'https://www.google.com/search?q={query}', headers=headers)
  soup = BeautifulSoup(html.text, 'lxml')

  for result in soup.select('.tF2Cxc'):
    title = result.select_one('.DKV0Md').text
    link = result.select_one('.yuRUbf a')['href']
    displayed_link = result.select_one('.TbwUpd.NJjxre').text

    try:
      snippet = result.select_one('#rso .lyLwlc').text
    except: snippet = None

    print(f'{title}\n{link}\n{displayed_link}\n{snippet}\n')

    # appending all data to array as dict()
    data.append({
      'title': title,
      'link': link,
      'displayed link': displayed_link,
      'snippet': snippet
    })

  # create dataframe and save it as .csv
  df = pd.DataFrame(data)
  df.to_csv('bs4_final.csv', index=False)

Alternatively, you can do it as well by using Google Organic Results API from SerpAPI as αԋɱҽԃ αмєяιcαη mentioned in the comments. It's a paid API with a free plan.

The main difference is that you only need to iterate over JSON string without doing any additional things.

Code to integrate:

from serpapi import GoogleSearch
import os
import pandas as pd

keywords = pd.read_csv('keywords.csv', header=0, index_col=None)

for query in keywords['keywords']:
    params = {
      "api_key": os.getenv("API_KEY"),
      "engine": "google",
      "q": query,
      "hl": "en",
    }

    search = GoogleSearch(params)
    results = search.get_dict()

    data = []

    for result in results['organic_results']:
        title = result['title']
        link = result['link']
        displayed_link = result['displayed_link']
        snippet = result['snippet']
                
        print(f"{title}\n{link}\n{displayed_link}\n{snippet}\n")

        data.append({
          'title': title,
          'link': link,
          'displayed link': displayed_link,
          'snippet': snippet
        })

    df = pd.DataFrame(data)
    df.to_csv('serpapi_final.csv', index=False)

PS - I wrote a blog post with visuals ( gifs ) about how to scrape Google Organic Search results.

Disclaimer, I work for SerpApi.