How to get ajax added contents on web page using selenium webdriver?

Question

I have written a code to scrap some information from one of the website. For scrapping purpose i am using Selenium Webdriver . Now my problem is, there are some information present on one the page but initially only ten entries are displayed on page, i am able to scrap this ten entries. But there are some more entries which will be displayed only after clicking load more link which is at the bottom of the page . whenever i click load more link remaining entries will be loaded on to the page with out refreshing the whole page( probably ajax update ) i am unable to scrap this newly loaded entries. Please help

This is the code i have written

WebDriver driver = new  HtmlUnitDriver(BrowserVersion.FIREFOX_3_6);             
driver.get("some site url");
Thread.sleep(500);
driver.findElement(By.xpath("//*[@id=\"username\"]")).sendKeys("user name");     
driver.findElement(By.xpath("//*[@id=\"password\"]")).sendKeys("password");      
driver.findElement(By.xpath("//*[@id=\"login\"]//div/button")).click();
Thread.sleep(200);                  
if(driver.getPageSource().toString().contains("Hi "+un)) {
    driver.get("http://www.somesite/m/searches/Loads/new");
    driver.findElement(By.xpath("//*[@id=\"Criteria_PostingAge\"]")).clear();
    driver.findElement(By.xpath("//*[@id=\"Criteria_PostingAge\"]")).sendKeys("12");   
    driver.findElement(By.xpath("//*[@id=\"Criteria_Origin_RawValue\"]")).clear();
    driver.findElement(By.xpath("//*[@id=\"Criteria_Origin_RawValue\"]")).sendKeys(orgn[paraCount]);                            
    driver.findElement(By.xpath("//*[@id=\"Criteria_Destination_RawValue\"]")).clear();
    driver.findElement(By.xpath("//*[@id=\"Criteria_Destination_RawValue\"]")).sendKeys(destn[paraCount]);              
    date=new Date();
    calendar=Calendar.getInstance();
    calendar.setTime(date);                             
    driver.findElement(By.xpath("//*[@id=\"Criteria_PickupFrom\"]")).clear();
    driver.findElement(By.xpath("//*[@id=\"Criteria_PickupFrom\"]")).sendKeys(pickupDtFmt.format(date).toString());
    calendar.add(Calendar.DAY_OF_MONTH, 1);
    date=calendar.getTime();                            
    driver.findElement(By.xpath("//*[@id=\"Criteria_PickupTo\"]")).clear();
    driver.findElement(By.xpath("//*[@id=\"Criteria_PickupTo\"]")).sendKeys(pickupDtFmt.format(date).toString());    
    driver.findElements(By.xpath("//*[@id=\"search-entry\"]//div/input")).get(11).click();
    Thread.sleep(2000);                                 
    // The code upto here will get me starting ten entries

   // In order to get more entries i need to click on load more o link                   
    driver.findElement(By.xpath("//*[@id=\"loadMore\"]")).click();
    Thread.sleep(3000);                              

    WebElement myDynamicElement = (new WebDriverWait(driver, 30))
                              .until(ExpectedConditions.presenceOfElementLocated(By.xpath("//*[@id=\"search-results\"]/div[2]/ul/li[12]"))); 

Actually after clicking on load more anchor link i was supposed to get ten more entries. So totally 20 entries. But i am getting only same 10 entries which i got when page loaded for the first time

Answer 1

I bet you would get a different result if you switched to using GhostDriver instead. No reason to use HtmlUnitDriver when GhostDriver can do the same and is probably more cutting edge technology .