Reading (Potentially Large) Text Files without Reading to Memory in Java?

Question

I'm working on a program to read in various text files and display them without having to read the entire file to memory.

A brief description of what I want the program to do:

1. Assume a file of say, 3000 lines.
2. Display 50 lines at a time
3. Allow user to scroll down to read further lines, however these lines are loaded in real-time from reader. lines further up that are now not displayed are not stored in memory.
4. Allow user to scroll up to read previous lines, however these lines are also loaded in real-time or at least in a similar manner as reading forward.

What I want is to tailor the program to be memory-efficient and handle large files without fizzling out. I was looking at BufferedReaders but there doesn't seem to be a dependable way of backwards traversal. I was originally looking at how mark() and reset() functioned but I couldn't find a class that could set multiple marks.

I'm wondering if anybody could help me out and give me a few pointers towards some useful classes I could use. I was starting to poke around NIO classes like ByteBuffers and CharBuffers but I'm rather lost as to how I could implement them towards what I want to accomplish.

Thanks!

Answer 1

Back in the olden days of computing (1980's), this is exactly how we had to process large files for display.

Basically, you need an input method that can read specified lines from a file. Something like

List<String> block = readFile(file, 51, 100);

which would read the 51st through 100th lines of the file.

I see two ways you could accomplish this.

1. Read from the beginning of the file each time, skipping the first nth records and retrieving 50 (or some other number) strings.

2. Read the file once, and break it up into x files of length 50 (or some other number). Read your temporary files to get blocks of strings.

Either way, you would keep 3 blocks in memory; the current block, the previous block and the next block.

As you move forward through the strings, the current block becomes the previous block, the next block becomes the current block, and you read a new next block.

As you move backward through the strings, the current block becomes the next block, the previous block becomes the current block, and you read a new previous block.

Answer 2

Random access to a file is available in Java . So you can surf through the bytes of a file pretty easily, and have a region of file mapped to memory at a time.

You can have a Deque<E> implementation for readable region. Then you can add/remove data chunks or "lines" from both ends, to represent a visual data scroll.

If the "lines" are defined the characters that fit the width of the visual display (such as a console window), then you might just keep loading next x bytes/characters, and removing previous x bytes/characters.

Otherwise, you may need to scan ahead, and build some metadata about the file, noting down the positions of lines, or other interesting structures within the file. Then you can use this metadata to quickly navigate the file.