Selenium：在动态加载网页时滚动到页面末尾

Question

I have a webpage that keeps loading new items when scrolling down the page until every item is loaded.

I'm working with Selenium in Java, and need to scroll down to the bottom of the page in order to load everything.

I have tried several different options, like scrolling to an element of the bottom of the page:

WebElement copyrightAtEndOfPage = webDriver.findElement(By.xpath("//a[@href='/utils/copyright.html']"));
((JavascriptExecutor) webDriver).executeScript("arguments[0].scrollIntoView();", copyrightAtEndOfPage);

This only scrolls down once though, and then the webpage keeps loading.

I also tried this approach, which also only scrolls down once, because it only takes the browser height into consideration.

Any help is highly appreciated.

Answer 1

I will provide you code in Python for this. I think it's easy to translate to Java:

def scroll_down(self):
    """A method for scrolling the page."""

    # Get scroll height.
    last_height = self.driver.execute_script("return document.body.scrollHeight")

    while True:

        # Scroll down to the bottom.
        self.driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

        # Wait to load the page.
        time.sleep(2)

        # Calculate new scroll height and compare with last scroll height.
        new_height = self.driver.execute_script("return document.body.scrollHeight")

        if new_height == last_height:

            break

        last_height = new_height

Hope it helps you!

Answer 2

Thanks to Ratmir Asanov (see the approved answer above), I translated the Python code into Java to make it easier to implement for other people.

try {
    long lastHeight = (long) ((JavascriptExecutor) webDriver).executeScript("return document.body.scrollHeight");

    while (true) {
        ((JavascriptExecutor) webDriver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
        Thread.sleep(2000);

        long newHeight = (long) ((JavascriptExecutor) webDriver).executeScript("return document.body.scrollHeight");
        if (newHeight == lastHeight) {
            break;
        }
        lastHeight = newHeight;
    }
} catch (InterruptedException e) {
    e.printStackTrace();
}

Answer 3

Updated Johannes code a bit to make it functional.

JavascriptExecutor js = (JavascriptExecutor) driver;
try {
    long lastHeight=((Number)js.executeScript("return document.body.scrollHeight")).longValue();
    while (true) {
        ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
        Thread.sleep(2000);

        long newHeight = ((Number)js.executeScript("return document.body.scrollHeight")).longValue();
        if (newHeight == lastHeight) {
            break;
        }
        lastHeight = newHeight;
    }
} catch (InterruptedException e) {
    e.printStackTrace();
}

Answer 4

C# version of the Ratmir Asanov's answer:

var lastHeight =  driver.ExecuteScript("returndocument.body.scrollHeight");
while (true)
{
    driver.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);");
    await Task.Delay(500);

    var newHeight = driver.ExecuteScript("return document.body.scrollHeight");
    Console.WriteLine(lastHeight + " - " + newHeight);
    if (newHeight.Equals(lastHeight))
        break;

    lastHeight = newHeight;
}

Answer 5

Updating the above solution by Prabhat further as it was still giving me compilation error.

    try {
        Object lastHeight = ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");

        while (true) {
            ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
            Thread.sleep(2000);

            Object newHeight = ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
            if (newHeight.equals(lastHeight)) {
                break;
            }
            lastHeight = newHeight;
        }
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

Answer 6

I found another solution to the dynamically loading page.

Count the elements that are displayed every scroll before and after the scroll and compare them to determine if you've scrolled to the bottom.

var reachedEnd = false;
oldCount = driver.FindElements(By.CssSelector(".searchDataContainer.table-row.raw")).Count;

while (!reachedEnd)
{
    driver.FindElement(By.CssSelector("body")).SendKeys(Keys.End);
    Thread.Sleep(500);
    oldCount = driver.FindElements(By.CssSelector(".searchDataContainer.table-row.raw")).Count;

    if (newCount == oldCount)
    {
        reachedEnd = true;
    }
    else
    {
        newCount = oldCount;
    }
}

Answer 7

Updated code that worked for me:

try {
                    long lastHeight = (long) ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
                    int cont=1000;
                    while (true) {
                        ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, "+cont+");");
                        Thread.sleep(2000);

                        long newHeight = (long) ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
                        if (newHeight <= cont) {
                            break;
                        }
//                      lastHeight = newHeight;
                        cont+=500;
                    }
                } catch (InterruptedException e) {
                    e.printStackTrace();
                }

Answer 8

Making a slight correction to the above stated answers. The variable 'start' of type long keeps changing after every scroll and the value becomes same after it reaches the end of the webpage. And as it an infinite loop with will keep returning the same value again and again. So, I just took the 'temp' variable and checked two consecutive values are same or not as the values remain same after the end is reached. As soon at it finds the same it exits the loop.

    try {
            long temp = 0;
            while (true) {
                ((JavascriptExecutor) driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
                Thread.sleep(2500);
                long start = (Long) ((JavascriptExecutor) driver).executeScript("return document.body.scrollHeight");
                if (start == temp) {
                    break;
                }
                temp = start;
            }
            System.out.println("completed the scroll");
        } catch (Exception e) {
            e.printStackTrace();
        }

Answer 9

您可以添加以下代码以保持按下页面向下按钮：

new Actions(driver).sendKeys(Keys.PAGE_DOWN).perform();

Answer 10

to scroll and wait to load more property till end

                lenOfPage = driver.instance.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")

                match = False
                while not match:
                    lastCount = lenOfPage
                    time.sleep(2)
                    lenOfPage = driver.instance.execute_script("window.scrollTo(0, document.body.scrollHeight);var lenOfPage=document.body.scrollHeight;return lenOfPage;")

                    if lastCount == lenOfPage:
                        match = True