使用 Beautiful Soup 从华尔街日报抓取债券数据

Question

I'm trying to scrape some data from the WSJ markets page using Beautiful Soup but always get Status Code=404 Page not found message. My code works fine on other websites so what's going wrong?

import requests
from bs4 import BeautifulSoup
url='https://www.wsj.com/market-data/quotes/bond/BX/TMBMKJP-10Y?mod=md_bond_govt_bonds_quote'
page=requests.get(url)
soup=BeautifulSoup(page.content,'html.parser')
print('Page status code',page.status_code)

Answer 1

You need to add headers . You can try it:

headers =  {'User-Agent': 'Mozilla/5.0 (Windows NT x.y; Win64; x64; rv:10.0) Gecko/20100101 Firefox/10.0 '}

url='https://www.wsj.com/market-data/quotes/bond/BX/TMBMKJP-10Y?mod=md_bond_govt_bonds_quote'

page=requests.get(url, headers=headers)