仅加载一次网址

Question

I have a table of url I want to load, the table can have one or more time an url.

For example, a table with three values : url1, url2 url1.

So, after, I load an url, an extract one of his html piece(for example a ).

I have this :

    HtmlPage page=null;


for (int i = 0; i < tableUrlSource.length; i++) {
try {
                page = webClient.getPage(tabUrlSource[i]);
                List<HtmlElement> nbElements = (List<HtmlElement>) page.getByXPath(tabXpathSource[i]);
                if (null != nbElements && !nbElements.isEmpty()) {
                    htmlResult = nbElements.get(0).asText();

    }

...

But this is not the more efficient, because it will load url1 two times and url one time. So it will like there is three url to load, and then, make the treatment longer.

How can I load an url only one time and keep the same final result?

I hope my english is clear, so as my question.

Regards.

Thank you.

Answer 1

You could use a Set<HtmlElement> instead of a List . This will remove duplicates automatically.

This of course is dependant on the fact that HtmlElement s are comparable. If they aren't, you could instead add all the URLs to a Set<String> and then iterate over that.

Update

To clarify the second part:

A Set is declared like this in the Javadocs:

A collection that contains no duplicate elements. More formally, sets contain no pair of elements e1 and e2 such that e1.equals(e2), and at most one null element. As implied by its name, this interface models the mathematical set abstraction.

In other words, to ensure that there are no duplicates, it relies on the elements being comparable via the equals() method. If HtmlElement hasn't overridden this method, the Set will just use the Object.equals() method, which just compares object references instead of the actual data in the HtmlElements .

However, String has overridden the equals() method, and you can therefor be certain that duplicate String s will be removed from a Set<String> .

Answer 2

What Keppil answered is correct but you would have to use the Set in place of tabUrlSource[i] rather than for Set<HtmlElement>

EDIT: Okay what is the content of tabUrlSource[i]?Is it of type URL or custom? This is how it would look like if it is URL

  Set <URL>uniqueURLs = new HashSet <URL>();

  for (int i = 0; i < tableUrlSource.length; i++) { 
  uniqueURLs.add(tableUrlSource[i])
  }

And then iterate over this Set instead of tableUrlSource array like this

  for(Iterator itr = uniqueURLs.iterator(); itr.hasNext();  ){
   page = webClient.getPage((URL)itr.next());
    .............
    .............

Continue the rest of the code

Also you said you are using index 'i' to associate url and xpath. Will that xpath be same for same url? If so you can use HashMap instead with key as URL and value as xpath so that duplicate keys will be overridden. Then you can iterate over this hashmap keys to get the 'page' and use the 'value' for to fetch HTMLELEMENT

If they are not same you can still use a HashSet like this

  Set <URL>uniqueURLs = new HashSet <URL>();
      HtmlPage page=null;


   for (int i = 0; i < tableUrlSource.length; i++) {
   try {    
            if(uniqueURLs.contains(tabUrlSource[i]) continue;
            else
             uniqueURLs.add( tabUrlSource[i] );
            page = webClient.getPage(tabUrlSource[i]);
            List<HtmlElement> nbElements = (List<HtmlElement>) 
            page.getByXPath(tabXpathSource[i]);
            if (null != nbElements && !nbElements.isEmpty()) {
                htmlResult = nbElements.get(0).asText();

}

Hope this helps :)