Java: stream processing of data which comes in chunks

Question

Given we have some network or other process, which fetches data we need in chunks. Each chunk is an array of bytes. The nature of data is just a simple text file which consist of many lines. We want to process this file line by line. Is this possible?

A straightforward way to do this is to wait until all data comes, meanwhile adding all chunks to ByteBuffer, or simply merging them to one big byte array by System.arrayCopy. After this we can create big String from this and read it line by line, or create ByteArrayInputStream and read it with some Reader after transforming to InputStreamReader.

OK, but can we do it in a real stream fashion, reading next chunk while it arrives? No guarantees are made that chunk consists of some complete number of lines. It can end in the middle of the line and this should be processed ie in this case we should wait for next chunk.

Is there a way to do this without waiting for the end of file?

Answer 1

This isn't all that different from just reading from a BufferedReader; the difference is the BufferedReader doesn't buffer more data in the background as the current chunk is being processed; it waits until it's empty and you call some read() method. But if that's ok, wire a BufferedReader to you input and keep things simple.

If you need to read in parallel, Look into PipedInputStream/PipedOutputStream. They're paired, and the idea is to have one thread writing the data it reads from the stream to the PipedOutputStream, then another thread reads from the PipedInputStream.

Or you can use non-blocking IO, but that involves saving the processing context so you can resume it later.