Python Selenium clicking next button until the end

Question

This is a follow up question from my first question , and I am trying to scrape a website and have Selenium click on next (until it cant be clicked) and collect the results as well.

This is the html tag from the website with the button:

<div class="pager results-display">
  <span class="action-btn prev inactive" data-page="1">Prev</span>
  <span class="action-btn inactive" data-page="1">1</span>  # Currently in page 1 thus inactive
  <span class="action-btn" data-page="2">2</span>
  <span class="action-btn" data-page="3">3</span> 
  <span class="action-btn" data-page="4">4</span>
  <span class="action-btn" data-page="5">5</span>
  <span class="action-btn" data-page="6">6</span>
  <span class="action-btn" data-page="7">7</span>
  <span class="action-btn" data-page="8">8</span> 
  <span class="action-btn" data-page="9">9</span>
  <span class="action-btn" data-page="10">10</span>
  <span class="action-btn" data-page="11">11</span> 
  <span class="action-btn" data-page="12">12</span>
  <span class="action-btn" data-page="13">13</span>
  <span class="action-btn" data-page="14">14</span>
  <span class="action-btn next" data-page="2">Next</span>
</div>
<div class="no-results-display hidden" style="display: none;">
                    No companies matched your search. Try again.
                </div>

I've tried this code:

elm = driver.find_element_by_name('pager results-display')
elm.click()

And I've also checked this question out, but still wasn't able to solve it.

Any thoughts?

Answer 1

pager results-display are actually two classes that belongs to the button's parent element. They aren't name attribute, and find_element_by_name gets only one name anyway. Try

elm = driver.find_element_by_class_name('next')
elm.click()

Notice that you might have to relocate the button each time if the DOM has changed after the click.

To click in loop as long as the button is active you can check if the buuton has class inactive

while True:
    elm = driver.find_element_by_class_name('next')
    if 'inactive' in elm.get_attribute('class'):
        break;
    elm.click()