Python Scraping using Selenium

Question

I am trying to scrape the products from Amazon, but it keeps giving me the error NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//li[@class='a-last']"} (Session info: chrome=102.0.5005.115)

The code is the following -->

from selenium import webdriver
from time import sleep
path = '/usr/local/bin/chromedriver'
# open the browser
browser = webdriver.Chrome(executable_path=path)
# load the webpage
browser.get('https://www.amazon.in')
browser.maximize_window()
# get the input elements
input_search = browser.find_element_by_id('twotabsearchtextbox')
search_button = browser.find_element_by_xpath("(//input[@type='submit'])[1]")
# send the input to the webpage
input_search.send_keys("protein")
sleep(1)
search_button.click()
products = []
for i in range(10):
    print('Scraping page', i+1)
    product = browser.find_elements_by_xpath("//span[@class='a-size-medium a-color-base a-text-normal']")
    for p in product:
        products.append(p.text)
    next_button = browser.find_element_by_xpath("//li[@class='a-last']")
    next_button.click()
    sleep(2)

and then the error

---------------------------------------------------------------------------
NoSuchElementException                    Traceback (most recent call last)
/Users/mihailzac/Web-Scraping/Scraping Products from Amazon using Selenium-Dynamic Website.ipynb Cell 11' in <cell line: 2>()
      5 for p in product:
      6     products.append(p.text)
----> 7 next_button = browser.find_element_by_xpath("//li[@class='a-last']")
      8 next_button.click()
      9 sleep(2)

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py:394, in WebDriver.find_element_by_xpath(self, xpath)
    378 def find_element_by_xpath(self, xpath):
    379     """
    380     Finds an element by xpath.
    381 
   (...)
    392         element = driver.find_element_by_xpath('//div/td[1]')
    393     """
--> 394     return self.find_element(by=By.XPATH, value=xpath)

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py:976, in WebDriver.find_element(self, by, value)
    974         by = By.CSS_SELECTOR
    975         value = '[name="%s"]' % value
--> 976 return self.execute(Command.FIND_ELEMENT, {
    977     'using': by,
    978     'value': value})['value']

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/selenium/webdriver/remote/webdriver.py:321, in WebDriver.execute(self, driver_command, params)
    319 response = self.command_executor.execute(driver_command, params)
    320 if response:
--> 321     self.error_handler.check_response(response)
    322     response['value'] = self._unwrap_value(
    323         response.get('value', None))
    324     return response

File /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/selenium/webdriver/remote/errorhandler.py:242, in ErrorHandler.check_response(self, response)
    240         alert_text = value['alert'].get('text')
    241     raise exception_class(message, screen, stacktrace, alert_text)
--> 242 raise exception_class(message, screen, stacktrace)

NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//li[@class='a-last']"}
  (Session info: chrome=102.0.5005.115)

Answer 1

To extract the texts from the Referenzcode column you can use list comprehension and you can use either of the following locator strategies :

• Using CSS_SELECTOR :

   driver.get("https://www.amazon.in/") WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#twotabsearchtextbox"))).send_keys("protein" + Keys.RETURN) print([my_elem.text for my_elem in driver.find_elements(By.CSS_SELECTOR, "span[data-component-type='s-search-results'] span.a-size-base-plus.a-color-base.a-text-normal")])

• Using XPATH :

   driver.get("https://www.amazon.in/") WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input#twotabsearchtextbox"))).send_keys("protein" + Keys.RETURN) print([my_elem.text for my_elem in driver.find_elements(By.XPATH, "//span[@data-component-type='s-search-results']//span[@class='a-size-base-plus a-color-base a-text-normal']")])

• Note : You have to add the following imports :

   from selenium.webdriver.common.keys import Keys from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC

• Console Output:

   ['Saffola Fittify Whey Protein Peanut Butter Dark Chocolate - 925g | Super Creamy | High Protein - High Fiber | No Trans Fats | 100% Real Peanuts, Brown', 'Saffola Whey Protein Peanut Butter Unsweetened Super Creamy 925g', 'Bigmuscles Nutrition Frotein 26g Refreshing Mango Flavored Hydrolysed Whey Protein Isolate 6g Glutamine 15g EAA Per Serving 0g Sugar Light and Crisp Like Juice (15 Servings, 500 gm)', 'Better Nutrition WPC 1kg | Whey Protein Concentrate For Women & Men | No Added Sugar | 24g Protein & 5.8g BCAA per scoop | Unflavored | Muscle Building | WPC for Workouts | Bodybuilding | 100% Veg', 'Bigmuscles Nutrition Premium Gold Whey 1Kg Whey Protein Isolate Blend | USA FDA REGD. BRAND | 25g Protein Per Serving [Belgian Chocolate]', 'Bigmuscles Nutrition Essential Whey Protein 1Kg [Dutch Chocolate] | 24g Protein per serving with Digestive Enzymes, Vitamin & Minerals, No Added Sugar | Improved Strength , Faster Recovery & Muscle Building', 'Bigmuscles Nutrition Crude Whey 1kg | US FDA REGD. BRAND | 24g Protein, 5.5g BCAA, 4 g Glutamine', 'AS-IT-IS Nutrition Whey Protein Concentrate 80% Unflavoured, Labdoor Certified (1kg)', 'Bigmuscles Nutrition Frotein 26g Refreshing Mango Flavored Hydrolysed Whey Protein Isolate [30 Servings, 1 kg] | 15g EAA | 0g Sugar', 'Optimum Nutrition (ON) Gold Standard 100% Whey Protein Powder 2 lbs, 907 g (Double Rich Chocolate), for Muscle Support & Recovery, Vegetarian - Primary Source Whey Isolate', "MuscleBlaze Beginner's Whey Protein, No Added Sugar, Faster Muscle Recovery & Improved Strength (Chocolate, 1 kg / 2.2 lb, 33 Servings)", 'MuscleBlaze Biozyme Performance Whey Protein, (Informed Choice UK, Labdoor USA Certified) with US Patent Filed EAF® (Rich Chocolate, 1 kg / 2.2 lb)', 'Optimum Nutrition (ON) Gold Standard 100% Whey Protein Powder 1 lbs, 454 g (Double Rich Chocolate), for Muscle Support & Recovery, Vegetarian - Primary Source Whey Isolate', 'MuscleBlaze Raw Whey Protein Concentrate 80% with Added Digestive Enzymes, Labdoor USA Certified (Unflavoured, 1 kg / 2.2 lb, 33 Servings)', 'AS-IT-IS ATOM Whey Protein 1kg with Digestive Enzymes | Double rich chocolate | 27g protein | 5.7g BCAA | Lab Tested', 'Cadbury Chocolate Health Drink - Bournvita, 2 Kg', 'Horlicks Lite Health & Nutrition Drink For Adults 450 g Jar, High Protein & Nutrients For Immunity - No Added Sugar (Regular Malt)', 'Nestle EVERYDAY Dairy Whitener, Milk Powder for Tea, 1Kg Pouch', '', '', 'Optimum Nutrition (ON) Gold Standard 100% Whey Protein Powder - 5 lbs, 2.27 kg (Double Rich Chocolate), Primary Source Isolate,For Men and Women', 'NAKPRO PLATINUM 100% Whey Protein Isolate | 28.1g Protein, 6.4g BCAA | Easy Mixing, Low Carbs, Easy Digesting Whey Protein Supplement Powder for Men, Women & Athletes | 1 Kg Chocolate Flavour (30 Servings)', 'NAKPRO Perform Whey Protein Concentrate, Whey Protein Concentrate Supplement Powder (Chocolate, 1 Kg)', 'MuscleBlaze Fuel One Whey Protein, 24 g Protein, 5.29 g BCAA, 4.2 g Glutamic Acid (Chocolate, 1 kg / 2.2 lb)', 'XLR8 Whey Protein with 24 g protein, 5.4 g BCAA - 2 lbs / 907 g (Chocolate Flavour)', 'Bigmuscles Nutrition Crude Whey - 1 kg (Rich Chocolate) | US FDA REGD. BRAND | Whey Protein Concentrate 80%, 24g Protein, 5.5g BCAA, 4 g Glutamine', 'Optimum Nutrition (ON) Gold Standard 100% Whey Protein Powder - 5 lbs, 2.27 kg (Double Rich Chocolate), Primary Source Isolate,For Men and Women', 'Optimum Nutrition (ON) Gold Standard 100% Whey Protein Powder- 4 kg, 129 servings (Vanilla Ice Cream), for Muscle Support & Recovery, Vegetarian - Primary Source Whey Isolate', 'Phab Protein Milkshake with Immunity Boosters – 17g Milk Protein, No added sugar, Vitamin B12 & Calcium Rich: Pack of 24x 200ml (Classic Chocolate)', 'OZiva Protein & Herbs for Women, Chocolate 500g | Natural Protein Powder for Women for Weight Control, Better Metabolism & Hormonal Balance | With Multivitamin for women, 23g Whey Protein, Ayurvedic Herbs, Certified Clean', 'AS-IT-IS Nutrition Whey Protein Isolate 90% - 1 kg | Protein 27g & BCAA 5.9g per serving', 'NAKPRO GOLD 100% Whey Protein Concentrate |Easy Mixing, Low Carbs, Easy Digesting Whey Protein Supplement Powder for Men, Women & Athletes | 1 Kg Unflavored (30 Servings)', 'Protinex Vanilla Delight - 400 g', 'Optimum Nutrition (ON) Performance Whey Protein Powder, 24g Protein, 5g BCAA – 1Kg (Chocolate Milkshake), Whey Protein Blend with Whey Protein Isolate', "MuscleBlaze Beginner's Whey Protein, No Added Sugar, Faster Muscle Recovery & Improved Strength (Chocolate, 1 kg / 2.2 lb, 33 Servings) with Shaker, 650 ml (Combo Pack)", 'NAKPRO PERFORM Whey Protein Concentrate | 24g Protein, 5.7g BCAA | Added Digestive Enzymes | Easy Mixing, Easy Digesting, (Unflavoured, 1 kg (Pack of 1) - Perform)', 'MuscleBlaze 100% Whey Protein, Ultra Premium Whey Blend (Rich Milk Chocolate, 1 kg / 2.2 lb, 30 Servings)', 'Bigmuscles Nutrition Raw & Real Isolate Organic Whey Protein [1Kg] | Natural 90% Isolate Whey Protein | Additives Free | Unflavored | 27g Protein With 5.5g BCAA & 4g Glutamine', 'MuscleBlaze Raw Whey Isolate 90% with Digestive Enzymes (Unflavoured, 1 kg / 2.2 lb, 33 Servings)', "AVVATAR WHEY PROTEIN | 1Kg | Unflavoured | Made with 100% Fresh Cow's Milk", 'Scitron Advance Whey Protein (28.5 Servings, 25.5g Protein, 6g BCAAs, 0g Sugar, 20 Vitamins & Minerals) \x80\x93 1kg (Milk Chocolate)', 'Muscle Asylum- Muscle Whey 100% Whey Protein - 24g Protein, 5.29g BCAA - Strawberry (29 Servings) - 1 Kg (2.2Lbs)', 'Nutrela 100% Whey Performance Protein Powder Supplement with Biofermented Vitamins & Digestive Enzymes, Pre Post Workout - Chocolate Flavour - 1 Kg - 33 Servings', 'PENTASURE - Daily Nutrition Protein Shake- Vanilla Flavour 400gm', 'GNC AMP Pure Isolate Powder Vanilla 1kg', 'Optimum Nutrition (ON) Gold Standard 100% Whey Protein Powder- 5 X 30.4 g Single Serve Sachets (Double Rich Chocolate), for Muscle Support & Recovery, Vegetarian - Primary Source Whey Isolate', 'Scitron Raw Whey (100% Whey Protein Concentrate, 24g Protein, 0g Sugar, 33 Servings, Essential & Non-Essential Amino Acids, No Added Flavour & Sweetener) - 1 kg', 'FAST&UP Essentials Whey Protein Isolate Blend (Rich Chocolate flavour), 30 Servings, 24g Protein, 5.5g BCAA, 4g Glutamine (2.1 Lbs, 960gm)', 'Nakpro Platinum 100% Whey Protein Isolate 30.4g Protein, 6.96g BCAA & 5.38g Glutamine Helps Build Your Muscle Size For Men, Women & Athletes - 1 Kg Unflavoured - (30 Servings)', 'GNC Pro Performance 100% Whey Protein - 2 kg (Chocolate Supreme)', 'HealthOxide My First Protein with whey, casein & pea, Chocolate – 1 kg', 'MuscleBlaze Whey Gold, 100% Whey Protein Isolate, Labdoor USA Certified (Rich Milk Chocolate, 1 kg / 2.2 lb, 33 Servings)', 'Himalaya Quista Pro Advanced Whey Protein Powder - 1kg (Chocolate)', 'MuscleBlaze Raw Whey Protein Concentrate 80% with Added Digestive Enzymes, Labdoor USA Certified (Unflavoured, 2 kg / 4.4 lb, 66 Servings)', 'PRO2FIT Vegan Plant protein powder with Pea protein Brown Rice and Mungbean Protein (Non-GMO, Gluten Free, Vegan Friendly , Non dairy, soy free) for women , men and family , Mint Chocolate 500 gms', 'OZiva Protein & Herbs, Men (23g Whey Protein, 5.5g BCAA & Ayurvedic herbs like Ashwagandha, Chlorella & Musli) for Better Stamina & Lean Muscles, Chocolate, Certified Clean, 500g', 'Abbzorb Whey Isolate (Chocolate) 1kg, 27.4g Protein, 7.04g BCAA per serving with Digestive Enzymes, Guaranteed Lab Report', 'Oxin Nutrition Whey Protein Concentrate Powder | 24.56gm Protein, 5.2gm BCAAs & 4.3gm Glutamine | Whey from USA (Cold Coffee 2 lbs)', 'MuscleBlaze Whey Energy with Whey & Multivitamins Blend (Chocolate, 1 kg / 2.2 lb, 30 Servings)', 'Brave Nutrition Whey Pro Protein Isolate Blend - 24g Protein, 11g EAAs, 4g Glutamine | Lean Protein Powder for Muscle Gain | Sports Nutrition | Muscle Builder Whey Concentrate Protein Powder for Men & Women [1Kg, Chocolate] Free T-Shirt', 'Abbzorb Whey Protein (Chocolate) 1kg, 26.22g Protein, 5.46g BCAA per serving with Digestive Enzymes, Guaranteed Lab Report', 'Nutrabay Pure 100% Whey Protein Concentrate || Raw Whey - Unflavoured, 2 kg', "Nature's Island Pure Whey Protein With Digezyme?, European Whey, High Absorption, No Sugar, No Sucralose, No Thickener, No Protein Spiking - 24G Protein, 5G BCAAs -1Kg, Kesar Kulfi", 'Phab Protein Milkshake with Immunity Boosters €“ 17g Milk Protein, No added sugar, Vitamin B12 & Calcium Rich:Pack of 6x 200ml (Variety Pack)', 'Optimum Nutrition (ON) Gold Standard 100% Whey Protein Powder 2 lbs, 907 g (Double Rich Chocolate), for Muscle Support & Recovery, Vegetarian - Primary Source Whey Isolate']