Concurrent HTTP URL connections in Java

Question

I am developing a tool that grabs all the text files from a given folder. These text files contain bunch of http URL. The requirement is that I do a GET request to these URL's, log the response code and optionally print the input stream. I have to perform this in multi-threaded fashion and the thread count should be configurable. So I am using TestNG and using data provider with parallel=true to set this up. Here is what I have so far. Am I doing this right?

Edit#2 The other issue I found is that when I run this tool for 90K URLs and with 100 threads..The requests are sent fast in the beginning but they slow down remarkably after a while. What could be the issue here?

Edit#1 I am able to run this tool for small number of records like a few thousands but I am not able it for a significantly high number of URLs(250K) and threads (2500). I am using ReportNG to generate reports. Here are the exceptions I get intermittently when I run my tool for 250K URLs and 2500 threads.

java.net.SocketException: Unexpected end of file from server
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTPHeader(Unknown Source)
at sun.net.www.http.HttpClient.parseHTTP(Unknown Source)
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.net.HttpURLConnection.getResponseCode(Unknown Source)
at com.HTTPConn.doHttpUrlConnectionAction(HTTPConn.java:83)
at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at 

org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:80)
    at org.testng.internal.Invoker.invokeMethod(Invoker.java:673)
    at org.testng.internal.Invoker.invokeTestMethod(Invoker.java:842)
    at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:73)
    at org.testng.internal.TestMethodWithDataProviderMethodWorker.call(TestMethodWithDataProviderMethodWorker.java:14) 

    [TestNG] Reporter org.uncommons.reportng.HTMLReporter@eee36c failed
org.uncommons.reportng.ReportNGException: Failed generating HTML report.
    at org.uncommons.reportng.HTMLReporter.generateReport(HTMLReporter.java:118)
    at org.testng.TestNG.generateReports(TestNG.java:1053)
    at org.testng.TestNG.run(TestNG.java:1016)
    at com.TestNGRunner.main(TestNGRunner.java:83)
Caused by: java.lang.NullPointerException
    at java.lang.String.compareTo(Unknown Source)
    at org.uncommons.reportng.TestResultComparator.compare(TestResultComparator.java:29)
    at org.uncommons.reportng.TestResultComparator.compare(TestResultComparator.java:25)
    at java.util.Collections.indexedBinarySearch(Unknown Source)
    at java.util.Collections.binarySearch(Unknown Source)
    at org.uncommons.reportng.HTMLReporter.sortByTestClass(HTMLReporter.java:229)
    at org.uncommons.reportng.HTMLReporter.createResults(HTMLReporter.java:180)
    at org.uncommons.reportng.HTMLReporter.generateReport(HTMLReporter.java:110)
    ... 3 more


public class HTTPConn {
    Logger log;
    File folder;
    boolean isStreamRequired;
    @BeforeClass(alwaysRun=true)
    public void setup(){
        log = Logger.getLogger(HTTPConn.class);
        folder = new File("c:\path\to\folder\from\where\I\want\to\read\textfiles");
        isStreamRequired = false;
    }

    @DataProvider(name="getRecords", parallel=true)
    public Object[][] getURLsFromTheFile() throws IOException {
        try {
            File [] listFiles = folder.listFiles();
            int i=0;
            Object[][] result = null;
            ArrayList<String> totalList = new ArrayList<String>();          
            for(int ind=0;ind<listFiles.length;ind++){
                if(listFiles[ind].isFile()){
                    List<String> arrList =FileUtils.readLines(listFiles[ind]); 
                    totalList.addAll(arrList);//add files one by one
                }
            }           
            result=new Object[totalList.size()][];                  
            for(String s:totalList){                        
             result[i]=new Object[]{s};
             i++;
         }              
         return result;
    } catch (Exception e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
        return null;
    }
 }
    @Test(dataProvider = "getRecords")
    public void doHttpUrlConnectionAction(String desiredUrl){
         try {
              URL url = new URL(desiredUrl);
              HttpURLConnection connection = (HttpURLConnection) url.openConnection();
              connection.setRequestMethod("GET");             
              connection.connect();
              int responseCode = connection.getResponseCode();  
              log.info("\nResponse Code:"+ desiredUrl+"\n"+responseCode+"\n");        
              if(responseCode!=200)
                  Assert.fail("Response Code:"+responseCode);
              else
                  Reporter.log("Response Code:"+responseCode);
              // read the output from the server
              if(isStreamRequired) {
                 //read the input stream and print it to log
                  reader.close();               
            }//isStreamOutputRequired           
              connection.disconnect();           
        } catch (Exception e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        } 
    }
}

Answer 1

Have you tried it? Just print the thread ids to double check that these methods are indeed running in parallel. If they're not, email me a small self contained test class that shows the problem.

Answer 2

Here is the update and answer.

I was running out of heap space and I was not catching Throwable in my code. Once I added that found out that I am running out of memory. So I used -Xmx512 flag like below

java -Xmx512m -jar Tool.jar

This in itself did not solve the problem. There was an issue with ReportNG and I was continuously getting exceptions. So I disabled using ReportNG

Again this did not solve the issue, For URLs more that 100K, now TestNG report started consuming lot of memory, so I had to disable the TestNG report as well by changing the flag setUseDefaultListeners to false

After all these changes everything started working for any number of records. It is just that the only reporting I have at this point is log4j and I can live with that.

Thanks to all for taking a look at this.