如何通过解析包含代码列表的文本文件来刮取Yahoo Finance？

Question

I have a text file containing a list of company tickers.

My intention is to be able to scrape Yahoo Finance to get financial data for each of these tickers.

I do not know how to tell Python to go over each individual line in my text file, assign it to a variable and then scrape that particular Yahoo Finance page and retrieve its data and store it, repeating the process for how many tickers are contained in the text file.

I found some code online to be able to scrape data from Yahoo Finance. Then I tried to add a for loop so that each ticker is added to a symbol variable (see code).

def scrape_table(url):
    page = requests.get(url)
    tree = html.fromstring(page.content)
    table = tree.xpath('//table')
    assert len(table) == 1

    df = pd.read_html(lxml.etree.tostring(table[0], method='html'))[0]

    df = df.set_index(0)
    df = df.dropna()
    df = df.transpose()
    df = df.replace('-', '0')

    # The first column should be a date
    df[df.columns[0]] = pd.to_datetime(df[df.columns[0]])
    cols = list(df.columns)
    cols[0] = 'Date'
    df = df.set_axis(cols, axis='columns', inplace=False)

    numeric_columns = list(df.columns)[1::]
    df[numeric_columns] = df[numeric_columns].astype(np.float64)

    return df

symbol = []

tf = open("SMI_tickers.txt")
for line in tf.readlines():
   symbol.append(print(line))

balance_sheet_url = 'https://finance.yahoo.com/quote/' + symbol + 
'/balance-sheet?p=' + symbol

df_balance_sheet = scrape_table(balance_sheet_url)
df_balance_sheet

The error message I got was

TypeError: can only concatenate str (not "list") to str

My end goal is to be able to import financial data for each of these companies so that I can perform some calculations on each stock.

Answer 1

You were trying to pass a list instead of a string. You need to iterate through all the symbols in your file, which you can do as follows. This code will append the results of the scraping function (which I have not tested) to a list balance_sheets.

balance_sheets=[]
tf = open("SMI_tickers.txt")
for line in tf.readlines():
    symbol.append(line)
    balance_sheet_url = 'https://finance.yahoo.com/quote/' + line + '/balance-sheet?p=' + line
    balance_sheets.append(scrape_table(balance_sheet_url))

Answer 2

I'm assuming your SMI_tickers.txt has a list of ticker symbols with one symbol per line. The erros comes when you build your url, you try to concatenate symbol, which is a list, not a string.

Change your code to below.

tf = open("SMI_tickers.txt")
for line in tf.readlines():
  symbol.append(line)
for symb in symbol:
    balance_sheet_url = 'https://finance.yahoo.com/quote/' + symb + 
          '/balance-sheet?p=' + symb

    df_balance_sheet = scrape_table(balance_sheet_url)
    df_balance_sheet

You don't actually need to read the symbols into a list, you could just use the symbol once you get it from the file without the list.

tf = open("SMI_tickers.txt")
for line in tf.readlines():
    balance_sheet_url = 'https://finance.yahoo.com/quote/' + line+ 
          '/balance-sheet?p=' + line

    df_balance_sheet = scrape_table(balance_sheet_url)
    df_balance_sheet

In both cases df_balance_sheet will be your pandas datafram.

Answer 3

For scraping balance sheet info, amongst anything else you might like, check out the yahoo_fin package http://theautomatic.net/yahoo_fin-documentation/ Makes things a wee bit easier

import pandas as pd
from yahoo_fin.stock_info import get_balance_sheet

stock_list = ["AAPL", "IBM", "AMZN"]

all_data = {ticker: get_balance_sheet(ticker)
                 for ticker in stock_list}

aapl = pd.DataFrame(all_data['AAPL'])
aapl = aapl.set_index('Period Ending')

Hope this helps