使用webdriver从Web表中检索列数据的更好方法

Question

I'm trying to fetch data from a table into List<List<String>> in java. Below code works. But it is taking 20+ seconds to fetch data. Would like to know is there any other faster way to fetch data from table?

List<WebElement> rows = table.findElements(By.xpath(".//tbody//tr//td//.."));
List<ArrayList<String>> rowsData = new ArrayList<ArrayList<String>>();

for(WebElement row:rows){
    List<WebElement> rowElements = row.findElements(By.xpath(".//td"));

    ArrayList<String> rowData = new ArrayList<String>();

    for(WebElement column:rowElements){
        rowData.add(column.getText().toString());
    }

    rowsData.add(rowData);
}

return rowsData;

Answer 1

I think JSoup is better option for larger html parsing. It provides pretty similar API to Selenium .

String html =  driver.findElement(By.tagName("table")).getAttribute("innerHTML");
ArrayList<String> colsArray = new ArrayList<>();
HashMap<Element, ArrayList<String>> dict = new HashMap<>();

Document document = Jsoup.connect(html).get();
Elements table = document.select("table");

Elements rows = table.select("tr");

for (Element row: rows){

    Elements list = row.select("td");
    ArrayList<String> newList = new ArrayList<>();

    for (Element str: list){
        newList.add(str.text());
    }

    dict.put(row ,newList);
}

return dict;

Answer 2

First of all your question is bit surprising for me, how does it work? You have . in xpaths and as per my knowledge selenium does need . in xpath. Anyways answer to your question:

1. If there is any possibility to use any other element locator than xpath then use that, it will definitely reduce the execution time. Since you have used for loop there and each loop will try to locate element using xpath and selenium parses entire html document to locate element so obviously it will increase the execution time.

2. If there is no possibility to use any other locator than xpath then you can disable implicit wait before performing above operation. Since your code does not perform any action like click which refreshes the loaded page so there wont be any issue related to time. Just make sure before performing above operation required table dom is completely loaded.

Don't forget to enable implicit wait after finishing above.

It will be like this:

driver.manage().timeouts().implicitlyWait(0, TimeUnit.SECONDS);
List<WebElement> rows = table.findElements(By.xpath("//tbody//tr//td//.."));
List<ArrayList<String>> rowsData = new ArrayList<ArrayList<String>>();

for(WebElement row:rows){
List<WebElement> rowElements = row.findElements(By.xpath("//td"));

ArrayList<String> rowData = new ArrayList<String>();

for(WebElement column:rowElements){
    rowData.add(column.getText().toString());
}

rowsData.add(rowData);}
return rowsData;

driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);

Answer 3

Look, the problem is caused by slowness of selenium. If you will use some lib for grabbing html -- the same algorithm will work in 1000 times faster.

Main idea:

1. do all work in selenium except of parsing table.

2. When you need to parse table, take InnerHtml of the this table via Selenium

3. Parse this html via external lib

In case of c# you can use HTMLAgilityPack. In case of java -- you need to google it. I had more than 1000 times faster result with the same algorithm of parsing by this way.

Answer 4

I have created a blog post and an example github project describing this type of situation -- it might help

http://simpleseleniumnotes.blogspot.com/2015/02/interaction-with-html-tables.html https://github.com/5hawnknight/solid-prototype-table