Java - 并行处理文档

Question

I have 5 documents(say) and I have some processing on each of them. Processing here includes open the document/file, read the data, do some document manipulation(edit text etc). For document manipulation I will probably be using docx4j or apache-poi. But my use case is this - I want to somehow process these 4-5 documents in parallel utilizing multiple cores available to me on my CPU. The processing on each document is independent of each other.

What would be the best way to achieve this parallel processing in Java. I have used ExecutorService in java before and Thread class too. But I dont have much idea about the newer concepts like Streams or RxJava . Can this task be achieved by using Parallel Stream in Java as introduced in Java 8? What would be better to use Executors/Streams/Thread Class etc. If Streams can be used please provide a link where I can find some tutorial on how to do that. Thanks for your help!

Answer 1

You can process in parallel using Java Streams using the following pattern.

List<File> files = ...
files.parallelStream().forEach(f -> process(f));

or

File[] files = dir.listFiles();
Stream.of(files).parallel().forEach(f -> process(f));

Note: process cannot throw a CheckedException in this example. I suggest you either log it or return a result object.

Answer 2

If you want to learn about ReactiveX, I would recomend use rxJava Observable.zip http://reactivex.io/documentation/operators/zip.html

Where you can run multiple process on parallel here an example:

 public class ObservableZip {

  private Scheduler scheduler;
  private Scheduler scheduler1;
  private Scheduler scheduler2;

  @Test
  public void testAsyncZip() {
           scheduler = Schedulers.newThread();//Thread to open and read 1 file
           scheduler1 = Schedulers.newThread();//Thread to open and read 1 file
           scheduler2 = Schedulers.newThread();//Thread to open and read 1 file
           Observable.zip(obAsyncString(file1), obAsyncString1(file2), obAsyncString2(file3), (s, s2, s3) -> s.concat(s2)
                                                                                        .concat(s3))
              .subscribe(result -> showResult("All files in one:", result));
       }

       public void showResult(String transactionType, String result) {
           System.out.println(result + " " +
                               transactionType);
       }

       public Observable<String> obAsyncString(File file) {
           return Observable.just(file)
                     .observeOn(scheduler)
                     .doOnNext(val -> {
                        //Here you  read your file
                     });
       }

       public Observable<String> obAsyncString1(File file) {
           return Observable.just(file)
                     .observeOn(scheduler1)
                     .doOnNext(val -> {
                         //Here you  read your file 2

                     });
       }

       public Observable<String> obAsyncString2(File file) {
           return Observable.just(file)
                     .observeOn(scheduler2)
                     .doOnNext(val -> {
                         //Here you  read your file 3

                     });
       }
      }

Like I said, just in case that you want to learn about ReactiveX, because if it not, add this framework in your stack to solve the issue would be a little overkill, and I would much rather the previous stream parallel solution