Unable to scrape two fields from a webpage using requests

Question

I'm trying to scrape two fields from this webpage using requests. I've used accurate selectors to locate the content but I can't fetch them as they are generated dynamically and not available in page source. However, I used the selectors as placeholders. I know how to grab the two fields using selenium but I wish to know how I can grab them using requests.

Fields that I'm after:

I've tried with:

import requests
from bs4 import BeautifulSoup

url = "https://www.namebase.io/domains/unite"

with requests.Session() as s:
    s.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 6.1; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'
    r = s.get(url)
    soup = BeautifulSoup(r.text,"lxml")
    total_bids = soup.select_one("[class='domain-highlights__container'] [class*='text_type_h4']").text
    highest_lockup = soup.select_one("[class='desktop-bid-card__right'] > [class*='text_type_h3']").text
    print(total_bids,highest_lockup)

How can I grab the two fields using requests?

Answer 1

The data is loaded via JavaScript, but you can use requests module to obtain the Json data.

For example:

import requests

url = 'https://www.namebase.io/api/domains/get/unite'
data = requests.get(url).json()

# uncomment this to print all data:
# import json
# print(json.dumps(data, indent=4))

no_bids = len(data['bids'])
highest = float(data['highestStakeAmount'] / 1_000_000)

print('No. bids', no_bids)
print('Highest lockup', highest)

Prints:

No. bids 6
Highest lockup 5.0

EDIT (Screenshot from Firefox Developer tools, where I found the API URL):