Selenium C# - Unable to find element

Question

At first I created a code in Selenium IDE firefox addon which is scraping a data from website. Of course it is working properly in IDE.

I want to scrape url from this:

<div class="gs-per-result-labels" url="http://example.com/foo/bar"></div>

As HTML is is shown as:

<tr>
    <td>open</td>
    <td>http://example.com</td>
    <td></td>
</tr>

<tr>
    <td>storeAttribute</td>
    <td>//div[@class='gs-per-result-labels']@url</td>
    <td>myValue</td>
</tr>

<tr>
    <td>echo</td>
    <td>${myValue}</td>
    <td></td>
 </tr>

Both command are executing properly and echo is giving right value. Next I changed format to C# / NUnit / WebDriver and I copied code to Visual Studio 2015. I added FirefoxDriver and IWebDriver references. That is the code:

private static IWebDriver driver;
static void Main(string[] args)
{
    driver = new FirefoxDriver();

    driver.Navigate().GoToUrl("http://example.com");
    Thread.Sleep(10000);
    string myValue = driver.FindElement(By.XPath("//div[@class='gs-per-result-labels']")).GetAttribute("url");
}

I also added Sleep to be sure that the page is fully loaded when it comes to scraping value. The thing is I am getting error on FindElement function, because driver was unable to find element. I am wondering why does that happen. Everything seems to be the same. Do you have any tips?

I think it is worth to said: The content which I want to scrape is generated by php or javascript (thats are google search result on "example" page which is not google)

Answer 1

Best approach would be to check if the Xpath is absolutely correct. You can always use Firebug or if you want to do it programmatically, below lines should help

 List<WebElement> existList=driver.FindElement(By.XPath("//div[@class='gs-per-result-labels']"));
 if(existList.size()>0){
    //SOPL("element present");
 }

I would suggest you to use explicit wait

 WebDriverWait wait=new WebDriverWait (driver,60);
 wait.until(ExpectedConditions.visibilityOfElementLocated(By.XPath("//div[@class='gs-per-result-labels']")));
  string myValue = driver.FindElement(By.XPath("//div[@class='gs-per-result-labels']")).GetAttribute("url");