简体   繁体   中英

Converting serially reading multiple files to reading them in parallel?

Following code in Java reads multiple files one after another serially and it works well till here. (The files are JSON and at this step they are stored in a String/Buffer without parsing.)

for (int fileIndex = 0; fileIndex < numberOfFiles; fileIndex++) {

        BufferedReader br = new BufferedReader(new FileReader("Files/file" + fileIndex + ".json"));
        try {
            StringBuilder sb = new StringBuilder();
            String line = br.readLine();

            while (line != null) {
                sb.append(line);
                sb.append(System.lineSeparator());
                line = br.readLine();
            }
            String contentJSON = sb.toString();
        } finally {
            br.close();
        }
    }

How to read those files in parallel by using Threads ?

I could not match Multithreading to above code and every time got errors.

Edit: I tested the answer of user @MadProgrammer and I got following error message:

SEVERE: null
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: Files\file0.json (The system cannot find the path specified)
    at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122)
    at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at com.Test.<init>(Test.java:24)
    at com.MainDefaultComponent.main(MainDefaultComponent.java:507)
Caused by: java.io.FileNotFoundException: Files\file0.json (The system cannot find the path specified)
    at java.base/java.io.FileInputStream.open0(Native Method)
    at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
    at java.base/java.io.FileReader.<init>(FileReader.java:60)
    at com.Test$ReadWorker.call(Test.java:44)
    at com.Test$ReadWorker.call(Test.java:34)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833) 

I've not tested this code directly (as I don't have a bunch of files to read), but the basic idea would be to do something like...

import java.io.BufferedReader;
import java.io.FileReader;
import java.util.ArrayList;
import java.util.List;
import java.util.concurrent.Callable;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;
import java.util.logging.Level;
import java.util.logging.Logger;

public class Test {

    public static void main(String[] args) {
        new Test();
    }

    public Test() {
        try {
            int numberOfFiles = 10;
            ExecutorService service = Executors.newFixedThreadPool(10);
            List<ReadWorker> workers = new ArrayList<>(numberOfFiles);
            for (int fileIndex = 0; fileIndex < numberOfFiles; fileIndex++) {
                workers.add(new ReadWorker(fileIndex));
            }
            List<Future<String>> results = service.invokeAll(workers);
            for (Future<String> result : results) {
                try {
                    String value = result.get();
                } catch (ExecutionException ex) {
                    Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
                }
            }
        } catch (InterruptedException ex) {
            Logger.getLogger(Test.class.getName()).log(Level.SEVERE, null, ex);
        }
    }

    public class ReadWorker implements Callable<String> {

        private int fileIndex;

        public ReadWorker(int fileIndex) {
            this.fileIndex = fileIndex;
        }

        @Override
        public String call() throws Exception {
            try (BufferedReader br = new BufferedReader(new FileReader("Files/file" + fileIndex + ".json"))) {
                StringBuilder sb = new StringBuilder();
                String line = br.readLine();

                while (line != null) {
                    sb.append(line);
                    sb.append(System.lineSeparator());
                    line = br.readLine();
                }
                return sb.toString();
            }
        }

    }
}

This will basically execute a series of Callable s and wait for them all to complete, at which time, you can then read the results (or errors)

See the Executors trail for more details

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM