包含JSP / JS的Python Beautiful Soup抓取页面

Question

i am trying to scrape the price from this page : url = https://www.renodepot.com/en/steph-round-base-shower-kit-69375118

the price information is given in the span tag and I am not able to scrape it. the simple code which I am using for this is

from requests import get
from bs4 import BeautifulSoup
response = get(url)
html_soup = BeautifulSoup(response.text, 'html.parser')
ProductPrice = html_soup.find('div',class_ = 'product_price_wrapper')

but this returns nothing, I think

 BEGIN RenoProdDetailPriceSnippet.jsp 

which appears just above the price div tab is causing the information to be protected.

I even tried doing it with selenium but was not successful. I tried many other combination to get the price but was not able to get the same.

So, I am looking for some ideas to solve this. Thanks

Answer 1

You cannot scrape the page because it requires the completion of a reCAPTCHA to access. This is specifically designed to stop bots.

If you examine html_soup you will find that you are actually searching the reCAPTCHA page, not the desired product page.