如何一次从文件中扫描n个单词？

Question

I have a really large file of text (1 million words+) and am trying to read it in in sections to avoid excessive memory usage and to try to speed it up. I am trying to read in 10k words at a time, place the unique words in that section in an array, and read the next 10k to do the same. I have worked out this so far:

while(scn.hasNext) {                       // Check if there is anything in the file
    for(int i = 10000; i > 0; i--) {       // For the next 10000 strings,
        if(scn.hasNext) {                  // as long as the file doesnt end,
            fullBook.add(scn.next());      // add the word to the string I am working on.
        }
        else {
            break;
        }
    }
}

All of this would be encased in yet another while so that I can work with each string before reading in the next 10k. I figure there is a faster way, but I haven't found it yet. I have looked through Scanner and Buffered reader to see if I could find a method that would only do so many words but I keep coming up empty. I don't mind learning a new method in order to do this or just some trick to speed it up. Thanks for the help in advance!

Answer 1

Your code is without difference from the below code in single loop.

while (scn.hasNext()) {
    fullBook.add(scn.next());

In fact, it is not meaningful to do in using 2 loops. The buffer of the Scanner is not changed and is always 1024. You may see it in the source of Scanner .

As the speed of I/O is slow, you may want to increase the buffer size and read from file with less frequency. You may change the creation of your Scanner with below code instead.

// Create a buffered reader with 1M buffer
Scanner scn = new Scanner (new BufferedReader(new FileReader(fileLocation), 1048576)); 

Answer 2

Let me know if there is a better way to do this.

Note: Scanner is NOT thread safe, @Alex recommends using a RandomAccessFile to bypasst his issue.

Use a Thread like

public class Parser implements UnitParserListener {


public Parser() {
    for(int i = 0; i < 1_000_000; i += 10_000) {
        new UnitParser(scanner, this, i);
    }
}

public void unitCompleted(int startCount, String[] words) {
    // This method will be called once for each thread completion
}

private class UnitParser implements Runnable {

    private UnitParserListener listener;
    private Thread thread;
    private int startCount;
    private Scanner scanner;

    public UnitParser(Scanner scanner, UnitParserListener listener, int startCount) {
        this.scanner = scanner.
        // Start the thread
        this.startCount = startCount;
        this.listener = listener;

        thread = new Thread(this);
        thread.start();
    }

    public void run() {
        // You'll have to edit this to your liking
        while(scn.hasNext()) {                       // Check if there is anything in the file
            for(int i = startCount; i < startCount + 10_000; i++) {

            }
        }

        // Thread complete
        listener.unitCompleted(startCount, results);

        // Attempt to stop the thread
        try {
            thread.join();
        } catch(Exception e) {}
    }

}

}
interface UnitParserListener {
    // startCount will give us a way to identify the thread
    void unitCompleted(int startCount, String[] words);
}