控制器服务上的多个连接（Spring）

Question

I have written a controller which takes as a input the domain name , crawls the whole site and gives back the result in JSON format

http://crawlmysite-tgugnani.rhcloud.com/getUrlCrawlData/www.google.com

This gives the data google

http://crawlmysite-tgugnani.rhcloud.com/getUrlCrawlData/www.yahoo.com

This gives data for yahoo

If I try to run these two URL's simultaneously, I see that I am getting the mixed data, and the results of one is affecting the another, even though I try to hit them from different machines.

Here is my controller

@RequestMapping("/getUrlCrawlData/{domain:.+}")
@ResponseBody
 public String registerContact(@PathVariable("domain") String domain) throws       HttpStatusException, SQLException, IOException {
      List<URLdata> urldata = null;
    Gson gson = new Gson();
     String json;
     urldata = crawlService.crawlURL("http://"+domain);
     json = gson.toJson(urldata);
     return json;
 }

What do I need to do modify to allow many multiple independent connections.

Update

Following is my crawl Service

public List<URLdata> crawlURL(String domain) throws HttpStatusException, SQLException, IOException{
    testDomain = domain;
    urlList.clear();
    urlMap.clear();
    urldata.clear();
    urlList.add(testDomain);
    processPage(testDomain);
    //Get all pages
    for(int i = 1; i < urlList.size(); i++){
        if(urlList.size()>=500){
            break;
        }
        processPage(urlList.get(i));
        //System.out.println(urlList.get(i));
    }
    //Calculate Time
    for(int i = 0; i < urlList.size(); i++){
        getTitleAndMeta(urlList.get(i));
    }
    return urldata;
}

public static void processPage(String URL) throws SQLException, IOException, HttpStatusException{

    //get useful information
try{

    Connection.Response response = Jsoup.connect(URL)
            .userAgent("Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.21 (KHTML, like Gecko) Chrome/19.0.1042.0 Safari/535.21")
            .timeout(10000)
            .execute();
    Document doc = response.parse();

    //get all links and recursively call the processPage method
    Elements questions = doc.select("a[href]");
    for(Element link: questions){
        String linkName = link.attr("abs:href");
        if(linkName.contains(testDomain.replaceAll("http://www.", ""))){
            if(linkName.contains("#")){
                linkName = linkName.substring(0, linkName.indexOf("#"));
            }
            if(linkName.contains("?")){
                linkName = linkName.substring(0, linkName.indexOf("?"));
            }
            if(!urlList.contains(linkName) && urlList.size() <= 500){

                urlList.add(linkName);
            }
        }
    }
}
catch(HttpStatusException e){
    System.out.println(e);
}
catch(SocketTimeoutException e){
    System.out.println(e);
}
catch(UnsupportedMimeTypeException e){
    System.out.println(e);
}
catch(UnknownHostException e){
    System.out.println(e);
}
catch(MalformedURLException e){
    System.out.println(e);
}
}

Answer 1

Each of your requests ( http://crawlmysite-tgugnani.rhcloud.com/getUrlCrawlData/www.google.com and http://crawlmysite-tgugnani.rhcloud.com/getUrlCrawlData/www.yahoo.com ) is processed in a separate thread. You have two instances of the crawlURL() method working simultaneously, but both methods use the same variables ( testDomain , urlList , urlMap and urldata ). So they mess up each other's data in these variables.

One way to fix the problem is to declare these variables locally (inside the method). This way, new instances of these variables will be created for each invocation of crawlURL() . Alternatively, you can create a new instance of your CrawlService class for each invocation of the crawlURL() method.

Synchronizing threads would be a bad idea here because one requests will wait for another to complete before it can be processed by crawlURL() .

Answer 2

As far as SpringMVC is concerned every request running in separate thread. So I think problem is in crawlService which, I suppose, is not stateless (singleton-like). Try to create new crawl service for every request and check if your data is not mixed. If creating crawl service is expensive operation you should rewrite it to work in stateless way.

@RequestMapping("/getUrlCrawlData/{domain:.+}")
@ResponseBody
public String registerContact(@PathVariable("domain") String domain) throws HttpStatusException, SQLException, IOException {

    Gson gson = new Gson();
    List<URLdata> = new CrawlService().crawlURL("http://"+domain);
    return gson.toJson(urldata);
}

Answer 3

I think

urldata = crawlService.crawlURL("http://"+domain);

This call to crawl Service is the one which is affected by Multiple requests coming simultaneously.

check whether crawlService is safe from multithreading .

ie check whether crawlURL() method is synchronized , if not make it synchronized .

or else synchronize the block of calling crawlservice inside controller .