使用 BeautifulSoup 抓取 OSHA 网站

Question

I'm looking for help with two main things: (1) scraping a web page and (2) turning the scraped data into a pandas dataframe (mostly so I can output as .csv, but just creating a pandas df is enough for now). Here is what I have done so far for both:

(1) Scraping the web site:

• I am trying to scrape this page: https://www.osha.gov/pls/imis/establishment.inspection_detail?id=1285328.015&id=1284178.015&id=1283809.015&id=1283549.015&id=1282631.015 . My end goal is to create a dataframe that would ideally contain only the information I am looking for (ie I'd be able to select only the parts of the site that I am interested in for my df); it's OK if I have to pull in all the data for now.
• As you can see from the URL as well as the ID hyperlinks underneath "Quick Link Reference" at the top of the page, there are five distinct records on this page. I would like each of these IDs/records to be treated as an individual row in my pandas df.

EDIT : Thanks to a helpful comment, I'm including an example of what I would ultimately want in the table below. The first row represents column headers/names and the second row represents the first inspection.

inspection_id   open_date   inspection_type close_conference    close_case  violations_serious_initial  
1285328.015     12/28/2017    referral        12/28/2017       06/21/2018         2

Mostly relying on BeautifulSoup4, I've tried a few different options to get at the page elements I'm interested in:

# This is meant to give you the first instance of Case Status, which in the case of this page is "CLOSED".

case_status_template = html_soup.head.find('div', {"id" : "maincontain"}, 
class_ = "container").div.find('table', class_ = "table-bordered").find('strong').text

# I wasn't able to get the remaining Case Statuses with find_next_sibling or find_all, so I used a different method:

for table in html_soup.find_all('table', class_= "table-bordered"):
    print(table.text)

# This gave me the output I needed (i.e. the Case Status for all five records on the page), 
# but didn't give me the structure I wanted and didn't really allow me to connect to the other data on the page.

# I was also able to get to the same place with another page element, Inspection Details.
# This is the information reflected on the page after "Inspection: ", directly below Case Status.

insp_details_template = html_soup.head.find('div', {"id" : "maincontain"}, 
class_ = "container").div.find('table', class_ = "table-unbordered")

for div in html_soup.find_all('table', class_ = "table-unbordered"):
    print(div.text)

# Unfortunately, although I could get these two pieces of information to print,
# I realized I would have a hard time getting the rest of the information for each record.
# I also knew that it would be hard to connect/roll all of these up at the record level.

So, I tried a slightly different approach. By focusing instead on a version of that page with a single inspection record, I thought maybe I could just hack it by using this bit of code:

url = 'https://www.osha.gov/pls/imis/establishment.inspection_detail?id=1285328.015'
response = get(url)
html_soup = BeautifulSoup(response.text, 'html.parser')
first_table = html_soup.find('table', class_ = "table-borderedu")
first_table_rows = first_table.find_all('tr')

for tr in first_table_rows:
    td = tr.find_all('td')
    row = [i.text for i in td]
    print(row)

# Then, actually using pandas to get the data into a df and out as a .csv.

dfs_osha = pd.read_html('https://www.osha.gov/pls/imis/establishment.inspection_detail?id=1285328.015',header=1)
for df in dfs_osha:
    print(df)

path = r'~\foo'
dfs_osha = pd.read_html('https://www.osha.gov/pls/imis/establishment.inspection_detail?id=1285328.015',header=1)
for df[1,3] in dfs_osha:
    df.to_csv(os.path.join(path,r'osha_output_table1_012320.csv'))

# This worked better, but didn't actually give me all of the data on the page,
# and wouldn't be replicable for the other four inspection records I'm interested in.

So, finally, I found a pretty handy example here: https://levelup.gitconnected.com/quick-web-scraping-with-python-beautiful-soup-4dde18468f1f . I was trying to work through it, and had gotten as far as coming up with this code:

for elem in all_content_raw_lxml:
    wrappers = elem.find_all('div', class_ = "row-fluid")
    for x in wrappers:
        case_status = x.find('div', class_ = "text-center")
        print(case_status)
        insp_details = x.find('div', class_ = "table-responsive")
        for tr in insp_details:
            td = tr.find_all('td')
            td_row = [i.text for i in td]
            print(td_row)
        violation_items = insp_details.find_next_sibling('div', class_ = "table-responsive")
        for tr in violation_items:
            tr = tr.find_all('tr')
            tr_row = [i.text for i in tr]
            print(tr_row)
        print('---------------')

Unfortunately, I ran into too many bugs with this to be able to use it so I was forced to abandon the project until I got some further guidance. Hopefully the code I've shared so far at least shows the effort I've put in, even if it doesn't do much to get to the final output! Thanks.

Answer 1

For this type of page you don't really need beautifulsoup; pandas is enough.

url = 'your url above'
import pandas as pd
#use pandas to read the tables on the page; there are lots of them...
tables = pd.read_html(url)

#Select from this list of tables only those tables you need:
incident = [] #initialize a list of inspections
for i, table in enumerate(tables): #we need to find the index position of this table in the list; more below       
    if table.shape[1]==5: #all relevant tables have this shape
        case = [] #initialize a list of inspection items you are interested in       
        case.append(table.iat[1,0]) #this is the location in the table of this particular item
        case.append(table.iat[1,2].split(' ')[2]) #the string in the cell needs to be cleaned up a bit...
        case.append(table.iat[9,1])
        case.append(table.iat[12,3])
        case.append(table.iat[13,3])
        case.append(tables[i+2].iat[0,1]) #this particular item is in a table which 2 positions down from the current one; this is where the index position of the current table comes handy
        incident.append(case)        


columns = ["inspection_id",   "open_date",   "inspection_type", "close_conference",    "close_case",  "violations_serious_initial"]
df2 = pd.DataFrame(incident,columns=columns)
df2 

Output (pardon the formatting):

    inspection_id   open_date   inspection_type close_conference    close_case  violations_serious_initial
0   Nr: 1285328.015 12/28/2017  Referral    12/28/2017  06/21/2018  2
1   Nr: 1283809.015 12/18/2017  Complaint   12/18/2017  05/24/2018  5
2   Nr: 1284178.015 12/18/2017  Accident    05/17/2018  09/17/2018  1
3   Nr: 1283549.015 12/13/2017  Referral    12/13/2017  05/22/2018  3
4   Nr: 1282631.015 12/12/2017  Fat/Cat 12/12/2017  11/16/2018  1