使用 requests.get 后，我不能 g

Question

I am using python scrape to find if a LV handbag is in stock. The following is my code:

import requests
from lxml import etree


endpoint = 'https://us.louisvuitton.com/eng-us/products/graceful-pm-damier-azur-canvas-nvprod840045v'


headers = {
 'Connection': 'keep-alive',
 'Cache-Control': 'max-age=0',
 'sec-ch-ua': '"Google Chrome";v="89", "Chromium";v="89", ";Not A Brand";v="99"',
 'sec-ch-ua-mobile': '?0',
 'Upgrade-Insecure-Requests': '1',
 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36',
 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
 'Sec-Fetch-Site': 'same-origin',
 'Sec-Fetch-Mode': 'navigate',
 'Sec-Fetch-User': '?1',
 'Sec-Fetch-Dest': 'document',
 'Referer': 'https://bj.ke.com/',
 'Accept-Language': 'zh-CN,zh;q=0.9',
}      # hearders
lv_response = requests.get(endpoint,headers=headers) # request to get
content_lv = lv_response.text # to text


html_lv = etree.HTML(content_lv) # using lxml to transform to html
stock_check = html_lv.xpath("//div[@class='lv-product__price-stock']/span/text()") # to locate Item Unavailable, Check Back Soon

print(stock_check) # the result gives me an empty list

As mentioned in the code, I first using inspect elements in the browser to find the key word: Item Unavailable, Check Back Soon with Xpath.

Then in the code, I also tried to locate the text Item Unavailable, Check Back Soon with Xpath again, but it will not give me any results.

I did some research. The website may be using JavaScript and I should use Selenium, and I have not tried Selenium. I am wondering if there is any way to find the keywords with the code I have now? Or please correct me if you can find any issues. Great thank!

Answer 1

So I would suggest you to use Selenium . The code become as simple as this:

from selenium import webdriver
from time import sleep

dr = webdriver.Chrome()
dr.get("https://us.louisvuitton.com/eng-us/products/graceful-pm-damier-azur-canvas-nvprod840045v")

sleep(2)  # to load JS

stock_check  = dr.find_element_by_xpath("//div[@class='lv-product__price-stock']/span").text
print(stock_check)