Scrolldown with python selenium to achieve dynamic appearing of new elements

Question

Url is https://donstroy.com/full-search . On that page, there is parent <div class="content"> and 21 <div class="item"> inside it. When human hovers mouse over <div class="content"> and scrolls down, multiple new <div class="item"> elements dynamically appear.

I'm trying to acheive this result (appearing of new elements) by selenium but all i can get is only first 21 <div class="item"> .

I tried different methods to scrolldown in selenium but none of them resulted in appearing new <div class="item"> elements. Below is my code. The first method with scrollIntoView is working fine when i run it in pure javascript in Element Inspector's console (it causes appearing of new elements) but still not working via selenium.

import selenium.webdriver
from selenium.webdriver.firefox.options import Options
from selenium.webdriver.common.action_chains import ActionChains

def parse_objs():
    return driver.find_elements_by_xpath('//div[@class="item"]')

options = Options()
options.headless = True
driver = selenium.webdriver.Firefox(options=options)
driver.set_page_load_timeout(300) # yes, it may take up to multiple minutes to load the page so may need to wait long time, it's normal
driver.get('https://donstroy.com/full-search')

objs = parse_objs()

# Amount of object on initial page load
print('Initial load : %s' % len(objs)) # prints 21 as expected

# Method 1
driver.execute_script('arguments[0].scrollIntoView();', objs[-1])
objs = parse_objs()
print('Method 1 : %s' % len(objs)) # expected to increase but still prints 21
#The thing is that when running the same method in pure javascript in Element Inspector's console, it's working fine:
#var objs=document.getElementsByClassName('item');
#objs[20].scrollIntoView();
#after that, amount of objs (objs.length) increases to 41 so it should work in selenium but it does not!

# Method 2
actions = ActionChains(driver)
actions.move_to_element(objs[-1]).perform()
objs = parse_objs()
print('Method 2 : %s' % len(objs)) # expected to increase but still prints 21

# Method 3
objs_container=driver.find_element_by_xpath('//div[@class="content"]')
driver.execute_script('arguments[0].scrollTop = 300;', objs_container)
print('ScrollTop=%s' % driver.execute_script('return arguments[0].scrollTop', objs_container)) # always returns 0 no matter what value i try to apply for scrollTop in above command
objs = parse_objs()
print('Method 3 : %s' % len(objs)) # expected to increase but still prints 21

driver.quit()

Answer 1

Looking at the website https://donstroy.com/full-search the next set of elements load only after reaching the end of scroll. Now with all your method it seems like webdriver is reading only the first 21 elements which has been loaded initially along with the page. You may want to see if the scroll is working so that the next set of elements appear dynamically. Add an expected conditions wait (EC), print the data and then scroll down again until the next set of data appear. Repeat this step till you get the desired length of list