How do I scrap text from "People also ask" section from google search using Selenium and Python

Question

Selenium is really important here. So, I want to create a program that can help me scrap stuff from google like the snippets etc while also giving me the ability to automate the browser for some other tasks. And here's what I've done.

from selenium import webdriver as webd
import time
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By

driver = webd.Firefox()

driver.get('https://google.com/')
driver.find_element(By.NAME, 'q').send_keys('cars')
driver.find_element(By.NAME, 'q').send_keys(Keys.ENTER)
ques = driver.find_element(By.CLASS_NAME, "ULSxyf")
print(ques)

time.sleep(8)
driver.close()

Although the first few lines works fine. I'm unable to open the People also ask section with selenium no matter what I do. I've used the class name of the object by inspecting it or used the id etc etc. And after searching for awhile I haven't really found anything much that would help this specific case scenario. I need to know how exactly to do this or why my method isn't working, if anybody has any idea. I'd be glad if you let me know. Thanks!

I'm a total beginner in selenium so if you can't give me a straight answer but feel a article or tutorial would be better, that would help as well.

EDIT: I want to open the questions from People also ask section and extract the answers from in there along with the questions themselves.

Answer 1

To extract the texts from the questions under People also ask column you have to induce WebDriverWait for visibility_of_all_elements_located() and you can use either of the following locator strategies :

• Code Block:

   driver.get("https://google.com/") WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.NAME, "q"))).send_keys("cars" + Keys.RETURN) print([my_elem.get_attribute("data-q") for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//span[text()='People also ask']//following::div[@data-q]")))])

• Console Output:

   ['Which is the most popular car?', 'What are top 10 cars?', 'Which type of car is best?', 'Which car is very cheapest?']

• Note : You have to add the following imports :

   from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC