从图表中抓取数据

Question

I am working with lobbying data from opensecrets.org, in particular industry data. I want to have a time series of lobby expenditures for each industry going back since the 90's.

I want to web-scrape the data automatically. Urls where the data is have the following format:

https://www.opensecrets.org/lobby/indusclient.php?id=H04&year=2019

which are pretty easy to embed in a loop, the problem is that the data I need is not in an easy format in the webpage. It is inside a bar graph, and when I inspect the graph I do not know how to get the data since it is not in the html code. I am familiar with web-scraping in python when the data is in the html code, but in this case I am not sure how to proceed.

Answer 1

If there is an API, that your best bet as mentioned above. But the data is able to be parsed anyway provided you get the right url/query parameters:

I've managed to iterate through it with the links for you to grab each table. I stored it in a dictionary with the key being the Firm name, and the value being the table/data. You can change it up to anyway you'd like. Maybe just store as json, or save each as csv.

Code:

import requests
from bs4 import BeautifulSoup
import pandas as pd

url = 'https://www.opensecrets.org/lobby/indusclient.php?id=H04&year=2019'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'}

data = requests.get(url, headers=headers)
soup = BeautifulSoup(data.text, 'html.parser')


links = soup.find_all('a', href=True)

root_url = 'https://www.opensecrets.org/lobby/include/IMG_client_year_comp.php?'
links_dict = {}

for each in links:
    if 'clientsum.php?' in each['href']:
        w=1
        firms = each.text
        link = root_url + each['href'].split('?')[-1].split('&')[0].strip() + '&type=c'
        links_dict[firms] = link


all_tables = {}
n=1
tot = len(links_dict)
for firms, link in links_dict.items():

    print ('%s of %s  ---- %s' %(n, tot, firms))
    data = requests.get(link)
    soup = BeautifulSoup(data.text, 'html.parser')

    results = pd.DataFrame()
    graph = soup.find_all('set')

    for each in graph:
        year = each['label']
        total = each['value']

        temp_df = pd.DataFrame([[year, total]], columns=['year','$mil'])
        results = results.append(temp_df,sort=True).reset_index(drop=True)

    all_tables[firms] = results
    n+=1

*Output:**

Not going to print as there are 347 tables, but just so you see the structure: