Scraping through multiple pages when url doesn't change

Question

I need to scrape through all the pages in the (link)[https://mahabocw.in/safety-kit-benefits-distribution/]. But the url doesn't change when I move to the next page. I tried to use selenium, but I am stuck as I don't know how to click the next page. Any help or suggestions would be really appreciated.

I have implemented the following code till now.

from selenium import webdriver
import time

url = "https://mahabocw.in/safety-kit-benefits-distribution/"
driver = webdriver.Chrome()
driver.get(url)

Below is the button element I need to click

<button type="button" class="ag-paging-button">Next</button>

Thanks a lot in advance. [1]: https://mahabocw.in/safety-kit-benefits-distribution/

Answer 1

You need to tell selenium to click the next button. Add this to your code and see if it works.

next_button = '/html/body/div/div[6]/div/article/div/div/div/div/div[2]/div/div/div[2]/div/div[4]/span[2]/div[3]/button'
click_next = driver.find_element_by_xpath(next_button)
click_next.submit()

You might have to use click_next.click() instead of.submit() depending on the page. Also, to get the 'next_button' you just inspect elements on the page, find the item you want, and click copy as xpath.