简体   繁体   中英

How to go to next page until the last page in Python Selenium when scraping website?

在此处输入图片说明

Image is for CSS selector and xpath for pagination.

在此处输入图片说明

I also wanted to perform a regex in to to separate Apple, iPhone 12, Neo Galactic Silver like this I wanted to print it in new line.

After finishing the product list of this current page, I want to be able to click next and perform the same procedure with the products on the next page.

This is the problem: when it reaches the 10 items of the current page, I have no idea how to change to another page and start all over again.

import xlwt
from selenium import webdriver
import re
import time

class cometmobiles:
    def __init__(self):
        self.url='https://www.mediaworld.it/catalogo/telefonia/smartphone-e-cellulari/smartphone'
    def comet(self):
        try:
            driver=webdriver.Chrome()
            driver.get(self.url)
            time.sleep(5)
            cookies = driver.find_element_by_id("onetrust-accept-btn-handler")    
            cookies.click()
            print("accepted cookies")
            driver.maximize_window()
            print("window maximized")
            mylist = []
            hasNextPate = True
            while hasNextPate:
                containers = []
                containters =driver.find_elements_by_css_selector('article[class="product clearfix p-list-js"]')
                for container in containters:
                    #Title
                    try:
                        title = container.find_element_by_css_selector('h3[class="product-name"]').text
                        print(title)
                    except:
                        pass

                    #price
                    try:
                        price = container.find_element_by_css_selector('span[class="price mw-price enhanced"]').text
                        print(price)
                    except:               
                        pass                    
                try:
                    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
                    time.sleep(5)
                    nxt=driver.find_elements_by_css_selector('span[class="pages"] a')
                    time.sleep(5)
                    nxt.click()
                except:
                    break
        except:
            pass
comets=cometmobiles()
comets.comet()    

Instead of this part

try:
    driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
    time.sleep(5)
    nxt=driver.find_elements_by_css_selector('span[class="pages"] a')
    time.sleep(5)
    nxt.click()
except:
    break

You can use this and also if the page number doesn't exist website turn the main page so you should add

try:
  x=0
  while True:
    x+=1
    driver.get(url+"?pageNumber="+str(x)) #Get the next page
    if driver.current_url == url: #If there is no next page it will turn main page and you can break at this time
      break
except:
  pass

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM