BeautifulSoup findAll return empty list

Question

So i wanted to get all "b" tags with "price" class form this website , but soup.findAll return empty list with 0 elements. But when I try the same thing, but with local html file, everything just works fine.

I've searched all over the internet, but nothing seems to help.

The code itself:

from bs4 import BeautifulSoup
from urllib.request import Request, urlopen

url = 'https://warframe.market/items/nami_skyla_prime_blueprint'

req = Request(url, headers={'User-Agent': 'Mozilla/5.0'})
webpage = urlopen(req).read()
soup = BeautifulSoup(webpage, 'html.parser')
tags = soup.findAll('b')
print(tags)

Answer 1

The data you see on the page is loaded from external URL via JavaScript (so beautifulsoup doesn't see it). To load the data into pandas dataframe you can use this example:

import requests
import pandas as pd

api_url = (
    "https://api.warframe.market/v1/items/nami_skyla_prime_blueprint/orders"
)

df = pd.DataFrame(requests.get(api_url).json()["payload"]["orders"])
df = pd.concat([df, df.pop("user").apply(pd.Series)], axis=1)
print(df.to_markdown(index=False))

Prints:

platform	region	order_type	creation_date	platinum	quantity	last_update	visible	id	reputation	region	avatar	last_seen	ingame_name	id	status
pc	en	sell	2018-03-24T13:19:44.000+00:00	5	1	2018-03-24T13:19:44.000+00:00	True	5ab6507049ef000fcf1920a9	0	en		2022-07-30T18:00:20.943+00:00	lastcrist	62e56f91330ca40377b19895	offline
pc	en	sell	2019-01-10T18:21:06.000+00:00	14	1	2022-07-26T18:45:05.000+00:00	True	5c378d12af045700a4291477	3	en		2022-07-27T11:04:01.133+00:00	kekxqt	5a0628ddc611360011c7a66e	offline
pc	en	sell	2019-12-09T07:11:43.000+00:00	35	1	2021-12-02T04:32:53.000+00:00	True	5dedf3afc72155005cf4028c	968	en	user/avatar/55f4e189b66f831701dbff77.png?0fe7b910face5429601e7085d289f869	2022-07-30T16:18:02.366+00:00	Sahysa	55f4e189b66f831701dbff77	offline
pc	en	sell	2020-02-04T03:29:02.000+00:00	13	1	2022-07-29T23:22:45.000+00:00	True	5e38e4fe7b0275005e6d29a9	4	en		2022-07-30T06:40:38.150+00:00	YellowSnowBoi	5b3c268c0c3535012e9b6ee3	offline
pc	en	sell	2020-03-29T20:14:39.000+00:00	15	1	2022-07-24T18:14:30.000+00:00	True	5e8101aff3441505a6b292c6	19	en	user/avatar/5b2931d4eb069f04986adb6d.png?822ece14d8951796338ef3739d5c6b2c	2022-07-29T20:55:01.617+00:00	xblq	5b2931d4eb069f04986adb6d	offline
pc	ko	sell	2020-11-07T09:15:58.000+00:00	9	1	2020-11-07T09:15:58.000+00:00	True	5fa665ced5b8ca0262d04cf0	0	ko		2022-07-30T17:33:05.154+00:00	blazespirit	5a4380778c474d007384823b	ingame
pc	en	sell	2021-02-07T12:42:27.000+00:00	15	1	2021-12-26T14:35:01.000+00:00	True	601fe033ec5e7e01d47859a1	3	en	user/avatar/600fcf3dc59d3703156a15e3.png?5f9404dcf7c36b3200d2eaab5ed9dba9	2022-07-30T10:26:53.816+00:00	pavelfizek	600fcf3dc59d3703156a15e3	offline
pc	en	buy	2021-02-12T14:34:49.000+00:00	10	6	2022-07-20T21:11:53.000+00:00	True	60269209039a2702632da531	412	en	user/avatar/598d8bf2d3ffb648a838abba.png?dfe50b7091f0e8c44b63976a700eebd9	2022-07-30T10:02:13.214+00:00	Xnidior	598d8bf2d3ffb648a838abba	offline
pc	en	sell	2021-02-13T03:24:21.000+00:00	10	1	2022-07-23T06:48:55.000+00:00	True	602746651c9894026f386fb0	2	en		2022-07-28T02:31:25.058+00:00	Reaper575	5cd904444532a609fef941db	offline
pc	en	sell	2021-03-13T18:03:01.000+00:00	15	1	2022-07-22T14:35:07.000+00:00	True	604cfe55171b1903005aa375	23	en	user/avatar/5c1cc13e6ad4a90015110a59.png?3eca6f848f62560068b7c0f17f141d6d	2022-07-27T11:36:30.529+00:00	-CG-Luthix	5c1cc13e6ad4a90015110a59	offline
pc	ru	buy	2021-04-03T05:56:22.000+00:00	5	1	2022-04-07T10:38:12.000+00:00	True	606803861a02f00165d2965c	2	ru	user/avatar/624ebb41596f2f00881ce522.png?c1556e2a317fe8dfbd7e86ba051e371a	2022-07-27T14:19:19.788+00:00	1FioLet1	624ebb41596f2f00881ce522	offline
pc	en	sell	2021-06-11T13:42:09.000+00:00	20	1	2021-06-11T13:42:09.000+00:00	True	60c368311970de00c952915f	0	en		2022-07-30T02:14:54.187+00:00	NEXUREX21	6039915229f04b0128def8fa	offline

...