[英]Java 8: How can I convert a for loop to run in parallel?
for (int i=0; i<100000; i++) {
// REST API request.
restTemplate.exchange(url, HttpMethod.GET, request, String.class);
}
I have a situation where I have to request a resource for 100k users and it takes 70 minutes to finish. 我有一种情况,我必须为10万用户请求一个资源,它需要70分钟才能完成。 I tried to clean up my code as much as possible and I was able to reduce it only by 4 minutes).
我试图尽可能地清理我的代码,我只能将它减少4分钟)。
Since each request is independent of each other, I would love to send requests in parallel (may be in 10s, 100s, or even 1000s of chunks which every finishes quickly). 由于每个请求彼此独立,我很乐意并行发送请求(可能是10s,100s甚至1000s的块,每次快速完成)。 I'm hoping that I can reduce the time to 10 minutes or something close.
我希望我能把时间减少到10分钟或者接近的时间。 How do I calculate which chunk size would get the job done quickly?
如何计算哪个块大小可以快速完成工作?
I have found the following way but I can't tell if the program processes all the 20 at a time; 我找到了以下方法,但我不知道该程序是否一次处理所有20个; or 5 at a time;
或一次5个; or 10 at a time.
或者一次10个。
IntStream.range(0,20).parallel().forEach(i->{
... do something here
});
I appericiate your help. 我知道你的帮助。 I am open to any suggestions or critics!!
我对任何建议或评论都持开放态度!
UPDATE: I was able to use IntStream and the task finished in 28 minutes. 更新:我能够使用IntStream,并在28分钟内完成任务。 But I am not sure this is the best I could go for.
但我不确定这是我能做的最好的。
I used the following code in Java 8 and it did the work. 我在Java 8中使用了以下代码,它完成了工作。 I was able to reduce the batch job to run from 28 minutes to 3:39 minutes.
我能够将批处理作业从28分钟减少到3:39分钟。
IntStream.range(0, 100000).parallel().forEach(i->{
restTemplate.exchange(url, HttpMethod.GET, request, String.class);
}
});
The standard call to parallel()
will create a thread for each core your machine has available minus one core, using a Common Fork Join Pool . 对
parallel()
的标准调用将使用Common Fork Join Pool为您的机器可用的每个核心减去一个核心创建一个线程。
If you want to specify the parallelism on your own, you will have different possibilities: 如果您想自己指定并行性,您将有不同的可能性:
System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "20")
System.setProperty("java.util.concurrent.ForkJoinPool.common.parallelism", "20")
Example: 例:
int allRequestsCount = 20;
int parallelism = 4; // Vary on your own
ForkJoinPool forkJoinPool = new ForkJoinPool(parallelism);
IntStream.range(0, parallelism).forEach(i -> forkJoinPool.submit(() -> {
int chunkSize = allRequestsCount / parallelism;
IntStream.range(i * chunkSize, i * chunkSize + chunkSize)
.forEach(num -> {
// Simulate long running operation
try {
Thread.sleep(1000);
} catch (InterruptedException e) {
e.printStackTrace();
}
System.out.println(Thread.currentThread().getName() + ": " + num);
});
}));
This implementation is just examplary to give you an idea. 这个实现只是为了给你一个想法的示例。
For your situation you can work with fork/join framework or make executor service pool of threads. 根据您的情况,您可以使用fork / join框架或创建执行程序服务池线程。
ExecutorService service = null;
try {
service = Executors.newFixedThreadPool(8);
service.submit(() -> {
//do your task
});
} catch (Exception e) {
} finally {
if (service != null) {
service.shutdown();
}
}
service.awaitTermination(1, TimeUnit.MINUTES);
if(service.isTerminated())
System.out.println("All threads have been finished");
else
System.out.println("At least one thread running");
And using fork/join framework 并使用fork / join框架
class RequestHandler extends RecursiveAction {
int start;
int end;
public RequestHandler(int start, int end) {
this.start = start;
this.end = end;
}
@Override
protected void compute() {
if (end - start <= 10) {
//REST Request
} else {
int middle = start + (end - start) / 2;
invokeAll(new RequestHandler(start, middle), new RequestHandler(middle, end));
}
}
}
Public class MainClass{
public void main(String[] args){
ForkJoinTask<?> task = new RequestHandler(0, 100000);
ForkJoinPool pool = new ForkJoinPool();
pool.invoke(task);
}
}
I've written a short article about that. 我写过一篇关于此的短文。 It contains simple tool that allows you to control pool size:
它包含允许您控制池大小的简单工具:
https://gt-dev.blogspot.com/2016/07/java-8-threads-parallel-stream-how-to.html https://gt-dev.blogspot.com/2016/07/java-8-threads-parallel-stream-how-to.html
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.