为什么我在抓取实时数据时得到一个空列表？

Question

I am trying toscrape some financial data but i keep getting empty lists whenever i use selenium or beautiful soup

this is the code i used with selenium

driver = webdriver.Chrome(os.getcwd() + '/chromedriver')
driver.get("https://www.amarkets.com/research-education/sentiment-widget/")
time.sleep(3)

real_soup = BeautifulSoup(driver.page_source, 'html.parser')
sentiments = real_soup.findAll("TD")
print(sentiments)

i keep getting an empty list.

Answer 1

Don't scrape it - use the api (which you can find by checking network activity ):

import requests
import pandas as pd

r = requests.get('https://api.prod.amarkets.dev/v1/sentiment-widget-dev/?request=%7B%22Symbols%22%3A%5B%22EURUSD%22%2C%22GBPUSD%22%2C%22USDJPY%22%2C%22AUDUSD%22%2C%22NZDUSD%22%2C%22USDCAD%22%2C%22USDCHF%22%2C%22EURJPY%22%2C%22EURGBP%22%2C%22XAUUSD%22%2C%22XAGUSD%22%2C%22DowJones30%22%2C%22S%26P500%22%2C%22Nasdaq100%22%2C%22BTCUSD%22%2C%22BRENT%22%5D%2C%22NodesType%22%3A%22Real%22%2C%22CollectSuffixes%22%3Atrue%2C%22RequestName%22%3A%22RequestWidgetSentiments%22%7D')
df = pd.DataFrame(r.json()['Result'])

Result:

	Symbol	Time	Bid	Ask	Digits	Spread	MinSpread	Buyers	Sellers	VolumeBuyInUsd	VolumeSellInUsd
0	AUDUSD	2021-11-10 17:57:39:216	0.73826	0.73852	5	2.6	2.1	127	126	901424	1.00069e+06
1	BRENT	2021-11-10 17:57:39:216	84.88	84.91	2	3	3	84	159	1.1943e+06	1.82476e+06
2	BTCUSD	2021-11-10 17:57:39:216	68344	68374.3	2	3024	1243	86	87	189964	341095
3	DowJones30	2021-11-10 17:57:39:216	36215	36223	0	8	7	40	75	262533	2.2185e+06
4	EURGBP	2021-11-10 17:57:39:216	0.85477	0.8549	5	1.3	1.3	65	89	454269	890184