Java：扫描文件，但从特定的行索引开始？

Question

I need to scan through newline-delimited text files with potentially over a million lines apiece. Due to webserver limitations, the only way to do this reasonably is to break the process up into smaller scanning chunks.

One way to do this that I've been able to find is using a Scanner and skipping lines until you reach your desired line index... but this has a less than desirable amount of overhead for numerous scanning visits to files with hundreds of thousands of lines.

RandomAccessFile.skip() and InputReader.skip() both allow seeking, but it is measured by bytes, and I cannot guarantee that every line will have the same number of bytes. Is there any way to skip several lines based on a delimiter rather than by bytes?

Or is there any other way to pull this off?

Answer 1

If you want to start at a particular line, you have to count new line characters. There's no way to do this other than some form of linear scan. A new line character is not "special" from a file system point of view.

I've had poor experience with the performance of Scanner though. I think your best bet is to use a BufferedReader with a large buffer.

If you're using the same file over and over again, you should create an index for line offsets so you can quickly seek to a given line.

Answer 2

No. If your lines are variable-length, such that you need to analyze whitespace to determine where they end, then there is no alternative to scanning sequentially through the file. You can write your code in a way that disguises the fact that you're doing so, but that doesn't change the performance characteristics.

Answer 3

Why do you need to seek by lines? Grab a chunk of N bytes, do whatever processing you need up to the last newline. There will be some bytes left unprocessed, possibly that number will be zero. Use that to step back, grab another chunk of N bytes, and so forth. (this is probably easier than gluing sections together)

(I'm assuming that you're looking to do some sort of processing on the whole file. If you're trying to seek for some line k, let your processing step be just "count newlines")