如何使用python请求和BeautifulSoup在网站上的html中查找隐藏值

Question

I'm trying to figure out how to find a hidden value that is only revealed when a button is clicked on the webpage. I don't see anything in the network log that sends another get request to reveal the hidden values. How would I get the hidden values on the webpage?

Here's my code:

product_page = self.url_session.get(
            'https://www.finishline.com/store/product/nike-mamba-fury-basketball-shoes/prod2797512',
            headers={
                'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
                'accept-encoding': 'gzip, deflate, br',
                'accept-language': 'en-US,en;q=0.9',
                'cache-control': 'max-age=0',
                'sec-fetch-dest': 'document',
                'sec-fetch-mode': 'navigate',
                'sec-fetch-user': '?1',
                'upgrade-insecure-requests': '1',
                'user-agent': user-agent
            },
            params={
                'styleId': f'{styleId}',
                'colorId': f'{colorId}'
            },
            cookies={
                '_abck': cookie # This gets a cookie
            }
        )

I am trying to reveal the "cartCatalogRefIds" value in the html.

Answer 1

You need to find the elent by id and after extract the value:

from bs4 import BeautifulSoup
soup = BeautifulSoup(product_page.body)
value= soup.find(id='cartCatalogRefIds')['value']

And you have your value