如何使用 selenium 和 beautiful soup 抓取隐藏的 class 数据

Question

I'm trying to scrape java script enabled web page content. I need to extract data in the table of that website. However each row of the table has button (arrow) by which we get additional information of that row.

I need to extract that additional description of each row. By inspecting it is observed that the contents of those arrow of each row belong to same class. However the class is hidden in source code. It can be observed only while inspecting. The data I'm trying to sparse is from the webpage .

I have used selenium and beautiful soup. I'm able to scrape data of table but not content of those arrows in the table. My python is returning me an empty list for the class of that arrow. But working for the classs of normal table data.

from bs4 import BeautifulSoup
from selenium import webdriver

browser = webdriver.Firefox()
browser.get('https://projects.sfchronicle.com/2020/layoff-tracker/')
html_source = browser.page_source  
soup = BeautifulSoup(html_source,'html.parser')
data = soup.find_all('div',class_="sc-fzoLsD jxXBhc rdt_ExpanderRow")
print(data.text)

Answer 1

The content you are interested in is generated when you click a button, so you would want to locate the button. A million ways you could do this but I would suggest something like:

element = driver.find_elements(By.XPATH, '//button')

for your specific case you could also use:

element = driver.find_elements(By.CSS_SELECTOR, 'button[class|="sc"]')

Once you get the button element, we can then do:

element.click()

Parsing the page after this should get you the javascript generated content you are looking for

Answer 2

To print hidden data, you can use this example:

import re
import json
import requests
from bs4 import BeautifulSoup


url = 'https://projects.sfchronicle.com/2020/layoff-tracker/'
soup = BeautifulSoup(requests.get(url).content, 'html.parser')
data_url = 'https://projects.sfchronicle.com' + soup.select_one('link[href*="commons-"]')['href']

data = re.findall(r'n\.exports=JSON\.parse\(\'(.*?)\'\)', requests.get(data_url).text)[1]
data = json.loads(data.replace(r"\'", "'"))

# uncomment this to see all data:
# print(json.dumps(data, indent=4))

for d in data[4:]:
    print('{:<50}{:<10}{:<30}{:<30}{:<30}{:<30}{:<30}'.format(*d.values()))

Prints:

Company                                           Layoffs   City                          County                        Month                         Industry                      Company description           
Tesla (Temporary layoffs. Factory reopened)       11083     Fremont                       Alameda County                April                         Industrial                    Car maker                     
Bon Appetit Management Co.                        3015      San Francisco                 San Francisco County          April                         Food                          Food supplier                 
GSW Arena LLC-Chase Center                        1720      San Francisco                 San Francisco County          May                           Sports                        Arena vendors                 
YMCA of Silicon Valley                            1657      Santa Clara                   Santa Clara County            May                           Sports                        Gym                           
Nutanix Inc. (Temporary furlough of 2 weeks)      1434      San Jose                      Santa Clara County            April                         Tech                          Cloud computing               
TeamSanJose                                       1304      San Jose                      Santa Clara County            April                         Travel                        Tourism bureau                
San Francisco Giants                              1200      San Francisco                 San Francisco County          April                         Sports                        Stadium vendors               
Lyft                                              982       San Francisco                 San Francisco County          April                         Tech                          Ride hailing                  
YMCA of San Francisco                             959       San Francisco                 San Francisco County          May                           Sports                        Gym                           
Hilton San Francisco Union Square                 923       San Francisco                 San Francisco County          April                         Travel                        Hotel                         
Six Flags Discovery Kingdom                       911       Vallejo                       Solano County                 June                          Entertainment                 Amusement park                
San Francisco Marriott Marquis                    808       San Francisco                 San Francisco County          April                         Travel                        Hotel                         
Aramark                                           777       Oakland                       Alameda County                April                         Food                          Food supplier                 
The Palace Hotel                                  774       San Francisco                 San Francisco County          April                         Travel                        Hotel                         
Back of the House Inc                             743       San Francisco                 San Francisco County          April                         Food                          Restaurant                    
DPR Construction                                  715       Redwood City                  San Mateo County              April                         Real estate                   Construction                  

...and so on.