urllib.error.HTTPError：HTTP 错误 404：未找到（雅虎财经）

Question

For my computing project, I'm attempting at making a financial forecasting website. One of the elements in the code is a web scraping API. It scrapes data from the income statement of a company on Yahoo Finance.

However, even though the URL is correct, I still keep on getting a 404 error.

My code

import pandas as pd
import urllib.request as ur
from bs4 import BeautifulSoup
import warnings
import ssl


ssl._create_default_https_context = ssl._create_unverified_context
income_url = 'http://uk.finance.yahoo.com/quote/AAPL/financials?p=AAPL'
read_url = ur.urlopen(income_url).read()
income_soup = BeautifulSoup(read_url, 'lxml')

div_list = []
for div in income_soup.find_all('div'):
    div_list.append(div.string)

    if not div.string == div.get('title'):
        div_list.append(div.get('title'))

div_list = [incl for incl in div_list if incl not in
            ('Operating Expenses', 'Non-recurring Events', 'Expand All')]
div_list = list(filter(None, div_list))
div_list = [incl for incl in div_list if not incl.startswith('(function')]
income_list = div_list[13: -5]
income_list.insert(0, 'Breakdown')

income_data = list(zip(*[iter(income_list)]*6))
income_df = pd.DataFrame(income_data)

headers = income_df.iloc[0]
income_df = income_df[1:]
income_df.columns = headers
income_df.set_index('Breakdown', inplace=True, drop=True)

warnings.warn('Amounts are in thousands.')
print(income_df)

I keep getting this error:

urllib.error.HTTPError: HTTP Error 404: Not Found error

How to fix it?

Answer 1

This issue appears to be resolved by ensuring that you pass a User-Agent header.

Using the requests module:

import requests

agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15'
headers = {'User-Agent': agent}
url = 'http://uk.finance.yahoo.com/quote/AAPL/financials?p=AAPL'
response = requests.get(url, headers=headers)
response.raise_for_status()