为什么在 python flask 中使用 Selenium 时出现 No such element 异常？

Question

I have written a program using selenium in python flask which does webscraping of a product. My intention is to first enter the product name(this is done programmatically)->after product is displayed-> it should display the price of product which I would be displaying in the terminal. My issue however is that it doesn't scrape the website and it throws a Selenium NoSuchElementException . Here is my code.

def scrape_johnLewis(product_name):
    website_address = 'https://www.johnlewis.com/'
    options = webdriver.ChromeOptions()
    options.add_argument('start-maximized')
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)

    browser = webdriver.Chrome(ChromeDriverManager().install(), options=options)

    browser.get(website_address)
    time.sleep(10)
    browser.implicitly_wait(20)
    browser.find_element_by_css_selector('button.c-button-yMKB7 c-button--primary-39fbj c-button--inverted-UZv88 c-button--primary-3tLoH').click()
    browser.find_element_by_id('mobileSearch').send_keys(product_name)
    browser.find_element_by_css_selector('div.input-with-submit.module-inputWrapper--63f9e > button.button.module-c-button--fe2f1').click()
    time.sleep(5)
    # browser.find_elements_by_class_name('button.module-c-button--fe2f1')[0].submit()
    product_price_raw_list = browser.find_elements_by_xpath('//div[@class="info-section_c-product-card__section__2D2D- price_c-product-card__price__3NI9k"]/span')
    product_price_list = [elem.text for elem in product_price_raw_list]
    print(product_price_list)


if __name__ == "__main__":
    scrape_johnLewis('Canon EOS 90D Digital SLR Body')

The error that I am getting is over here browser.find_element_by_css_selector('button.c-button-yMKB7 c-button--primary-39fbj c-button--inverted-UZv88 c-button--primary-3tLoH').click()

and here is the stacktrace:

Traceback (most recent call last):
  File "scrapejohnLewis.py", line 32, in <module>
    scrape_johnLewis('Canon EOS 90D Digital SLR Body')
  File "scrapejohnLewis.py", line 20, in scrape_johnLewis
    browser.find_element_by_css_selector('button.c-button-yMKB7 c-button--primary-39fbj c-button--inverted-UZv88 c-button--primary-3tLoH').click()
  File "/home/mayureshk/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 598, in find_element_by_css_selector
    return self.find_element(by=By.CSS_SELECTOR, value=css_selector)
  File "/home/mayureshk/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
    'value': value})['value']
  File "/home/mayureshk/.local/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/mayureshk/.local/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"button.c-button-yMKB7 c-button--primary-39fbj c-button--inverted-UZv88 c-button--primary-3tLoH"}
  (Session info: chrome=74.0.3729.108)
  (Driver info: chromedriver=74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729@{#29}),platform=Linux 5.0.0-1034-oem-osp1 x86_64)

I have tried to replace it with find_element_by_tag_name however that didn't do the work either. By inspecting the website I have come to find out the exact element but to my surprise the error says that there is no such element. What exactly could be the case? Please help.

Answer 1

The cookie prompt was not managed properly, it didn't go away and hence it blocks the code beneath thus stopping Selenium in finding the element. I've also made some tweaks to the code.

from selenium.webdriver.common.keys import Keys

def scrape_johnLewis(product_name):
    website_address = 'https://www.johnlewis.com/'
    options = webdriver.ChromeOptions()
    options.add_argument('start-maximized')
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)

    browser = webdriver.Chrome(ChromeDriverManager(log_level='0').install(), options=options)

    browser.get(website_address)
    time.sleep(3)
    # browser.implicitly_wait(20)
    browser.find_element_by_xpath('//*[@id="pecr-cookie-banner-wrapper"]/div/div[1]/div/div[2]/button[1]').click()
    browser.find_element_by_id('desktopSearch').send_keys(product_name + Keys.ENTER)
    time.sleep(5)
    product_price_raw_list = browser.find_elements_by_xpath('//div[@class="info-section_c-product-card__section__2D2D- price_c-product-card__price__3NI9k"]/span')
    product_price_list = [elem.text for elem in product_price_raw_list]
    print(product_price_list)

# ouptut
['£1,249.00', '£1,629.99', '£1,349.99']

Also try using css_selector as the last resort if you cannot find better element locators like id , xpath , tag_name .

Answer 2

Once, I was getting the same error. You will wonder How it was fixed?

I kept my selenium web driver by mistake, then copy the xpath from there and my code worked.

Then, I found that the xpath of the particular element in webdriver windows differs from the xpath of the same element in actual browser window .

There was a single extra tag being added into the path in webdriver window but not in actual browser window .

May be there is the same issue on your side.