在 JAVA 中發出多個 http 請求的快速和異步方式

Question

我有一個程序可以發出非常快的 http 請求。 請求應異步進行，以免阻塞主線程。

所以我創建了一個隊列，由 10 個發出 http 請求的獨立線程觀察。 如果隊列中插入了某些內容，則獲取數據的第一個線程將發出請求並處理結果。

隊列中塞滿了數以千計的項目，因此多線程對於盡快獲得響應是非常必要的。

由於我有很多代碼，我將舉一個簡短的例子。

主類

package fasthttp;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.LinkedBlockingQueue;

public class FastHTTP {

    public static void main(String[] args) {
        ExecutorService executor = Executors.newFixedThreadPool(10);

        for (int i = 0; i < 10; i++) {
            LinkedBlockingQueue queue = new LinkedBlockingQueue();
            queue.add("http://www.lennar.eu/ip.php");//for example
            executor.execute(new HTTPworker(queue));
        }
    }

}

FastHTTP 類

package fasthttp;

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.URL;
import java.util.concurrent.LinkedBlockingQueue;

public class HTTPworker implements Runnable {

    private final LinkedBlockingQueue queue;

    public HTTPworker(LinkedBlockingQueue queue) {        
        this.queue = queue;
    }

    private String getResponse(String url) throws IOException {

        URL obj = new URL(url);
        HttpURLConnection con = (HttpURLConnection) obj.openConnection();

        StringBuilder response;
        try (BufferedReader in = new BufferedReader(
                new InputStreamReader(con.getInputStream()))) {
            String inputLine;
            response = new StringBuilder();
            while ((inputLine = in.readLine()) != null) {
                response.append(inputLine);
            }
        }
        return response.toString();
    }

    @Override
    public void run() {
        while (true) {
            try {
                String data = (String) queue.take();
                String response = getResponse(data);
                //Do something with response
                System.out.println(response);
            } catch (InterruptedException | IOException ex) {
                //Handle exception
            }
        }
    }
}

有沒有更好或更快的方法來異步處理數千個 http 請求響應？ 速度和性能是我所追求的。

Answer 1

回答我自己的問題。 嘗試了 Apaches 異步 http 客戶端，但過了一段時間我開始使用 Ning 的異步客戶端，我對它很滿意。

Answer 2

import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.*;
import java.util.stream.Collectors;

import org.apache.http.client.methods.HttpGet;

import java.util.Iterator;

import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClientBuilder;

public class RestService {
    private final static Executor executor = Executors.newCachedThreadPool();
    private final static CloseableHttpClient closeableHttpClient = HttpClientBuilder.create().build();

    public static String sendSyncGet(final String url) {
        return sendAsyncGet(List.of(url)).get(0);
    }

    public static List<String> sendAsyncGet(final List<String> urls){
        List<GetRequestTask> tasks = urls.stream().map(url ->  new GetRequestTask(url, executor)).collect(Collectors.toList());
        List<String> responses = new ArrayList<>();
        while(!tasks.isEmpty()) {
            for(Iterator<GetRequestTask> it = tasks.iterator(); it.hasNext();) {
                final GetRequestTask task = it.next();
                if(task.isDone()) {
                    responses.add(task.getResponse());
                    it.remove();
                }
            }
            //if(!tasks.isEmpty()) Thread.sleep(100); //avoid tight loop in "main" thread
        }
        return responses;
    }

    private static class GetRequestTask {
        private final FutureTask<String> task;

        public GetRequestTask(String url, Executor executor) {
            GetRequestWork work = new GetRequestWork(url);
            this.task = new FutureTask<>(work);
            executor.execute(this.task);
        }

        public boolean isDone() {
            return this.task.isDone();
        }

        public String getResponse() {
            try {
                return this.task.get();
            } catch(Exception e) {
                throw new RuntimeException(e);
            }
        }
    }

    private static class GetRequestWork implements Callable<String> {
        private final String url;

        public GetRequestWork(String url) {
            this.url = url;
        }
        public String getUrl() {
            return this.url;
        }
        public String call() throws Exception {
            return closeableHttpClient.execute(new HttpGet(getUrl()), new BasicResponseHandler());
        }
    }
}