beautifulsoup 抓取 - 可扩展 header 文本缺失

Question

I was trying to extract data from a Y.Finance website using BeautifulSoup and store everything in a list, In the list, the headers of the expandable lines (Total Revenue. Operating Expense) are missing but the figures are still there? Is there a way to include the headers in the output?

import pandas as pd
from bs4 import BeautifulSoup
import urllib.request as ur

url = 'https://finance.yahoo.com/quote/AAPL/financials?p=AAPL'

read_data = ur.urlopen(url).read() 
soup= BeautifulSoup(read_data,'lxml')

ls= [] # Create empty list
for l in soup.find_all('div'): 
  ls.append(l.string) 


new_ls = list(filter(None,ls))

Current output:

 'Expand All',
 'ttm',
 '9/30/2019',
 '9/30/2018',
 '9/30/2017',
 '9/30/2016',
 '273,857,000',
 '260,174,000',
 '265,595,000',
 '229,234,000',
 '215,639,000',

Expected output:

 'Expand All',
 'ttm',
 '9/30/2019',
 '9/30/2018',
 '9/30/2017',
 '9/30/2016',
 'Total Revenue',
 '273,857,000',
 '260,174,000',
 '265,595,000',
 '229,234,000',
 '215,639,000',

Update: if I extract from "span", figures that are 0 were missing from the output which creates another problem when I construct the data frame later on

for l in soup.select('div.D\(tbr\)'): 
    for n in l.select('span'):
        print(n.text)

Answer 1

I know this is kind of off topic, but it looks like you just want the data from Yahoo finance right? If so, they have a python package already available that would probably be easier to work with then web scraping.

https://pypi.org/project/yahoo-finance/

You can enter a share

apple = Share('AAPL')

And also get a bunch of data by just using the following command

from pprint import pprint
pprint(yahoo.get_historical('2019-08-10', '2020-01-10'))

Answer 2

The following will get you all the data, and then you can filter out what you don't need:

for row in soup.select('div[data-test="fin-row"]'):     
    for r in row:
        for l in r:
            print(l.text)
    print('-------\n')

Output:

Total Revenue
273,857,000
260,174,000
265,595,000
-
215,639,000
-------

Cost of Revenue
169,277,000
161,782,000
163,756,000
-
131,376,000
-------

Gross Profit

etc.

If you also want to get the headers programmatically, try:

head_ind = [55,58,60,62,64,66]
for i in head_ind:
    heads = f'span[data-reactid="{i}"]:not([class])'
    for head in soup.select(heads):
        print(head.text)

Output:

Breakdown
ttm
9/30/2019
9/30/2018
9/30/2017
9/30/2016