Read the newly appended file content to an InputStream in Java

Question

I have a writer program that writes a huge serialized java object (at the scale of 1GB) into a binary file on local disk at a specific speed. Actually, the writer program (implemented in C language) is a network receiver that receives the bytes of the serialized object from a remote server. The implementation of the writer is fixed.

Now, I want to implement a Java reader program that reads the file and deserializes it to a Java object. Since the file could be very large, it is beneficial to reduce the latency of deserializing the object. Particularly, I want the Java reader starts to read/deserialize the object once the first byte of the object has been written to the disk file so that the reader can start to deserialize the object even before the entire serialized object has been written to the file. The reader knows the size of the file ahead of time (before the first byte is written to the file).

I think what I need is something like a blocking file InputStream that will be blocked when it reaches the EndOfFile but it has not read the expected number of bytes (the size of the file will be). Thus, whenever new bytes have been written to the file, the reader's InputStream could keep reading the new content. However, FileInputStream in Java does not support this feature.

Probably, I also need a file listener that monitoring the changes made to the file to achieve this feature.

I am wondering if there is any existing solution/library/package can achieve this function. Probably the question may be similar to some of the questions in monitoring the log files.

The flow of the bytes is like this: FileInputStream -> SequenceInputStream -> BufferedInputStream -> JavaSerializer

Answer 1

You need two threads: Thread1 to download from the server and write to a File, and Thread2 to read the File as it becomes available.

Both threads should share a single RandomAccessFile, so access to the OS file can be synchronized correctly. You could use a wrapper class like this:

public class ReadWriteFile {
    ReadWriteFile(File f, long size) throws IOException {
        _raf = new RandomAccessFile(f, "rw");
        _size = size;

        _writer = new OutputStream() {

            @Override
            public void write(int b) throws IOException {
                write(new byte[] {
                        (byte)b
                });
            }

            @Override
            public void write(byte[] b, int off, int len) throws IOException {
                if (len < 0)
                    throw new IllegalArgumentException();
                synchronized (_raf) {
                    _raf.seek(_nw);
                    _raf.write(b, off, len);
                    _nw += len;
                    _raf.notify();
                }
            }
        };
    }

    void close() throws IOException {
        _raf.close();
    }

    InputStream reader() {
        return new InputStream() {
            @Override
            public int read() throws IOException {
                if (_pos >= _size)
                    return -1;
                byte[] b = new byte[1];
                if (read(b, 0, 1) != 1)
                    throw new IOException();
                return b[0] & 255;
            }

            @Override
            public int read(byte[] buff, int off, int len) throws IOException {
                synchronized (_raf) {
                    while (true) {
                        if (_pos >= _size)
                            return -1;
                        if (_pos >= _nw) {
                            try {
                                _raf.wait();
                                continue;
                            } catch (InterruptedException ex) {
                                throw new IOException(ex);
                            }
                        }
                        _raf.seek(_pos);
                        len = (int)Math.min(len, _nw - _pos);
                        int nr = _raf.read(buff, off, len);
                        _pos += Math.max(0, nr);
                        return nr;
                    }
                }
            }

            private long _pos;
        };
    }

    OutputStream writer() {
        return _writer;
    }

    private final RandomAccessFile _raf;
    private final long _size;
    private final OutputStream _writer;
    private long _nw;
}

The following code shows how to use ReadWriteFile from two threads:

public static void main(String[] args) throws Exception {
    File f = new File("test.bin");
    final long size = 1024;
    final ReadWriteFile rwf = new ReadWriteFile(f, size);

    Thread t1 = new Thread("Writer") {
        public void run() {
            try {
                OutputStream w = new BufferedOutputStream(rwf.writer(), 16);
                for (int i = 0; i < size; i++) {
                    w.write(i);
                    sleep(1);
                }
                System.out.println("Write done");
                w.close();
            } catch (Exception ex) {
                ex.printStackTrace();
            }
        }
    };

    Thread t2 = new Thread("Reader") {
        public void run() {
            try {
                InputStream r = new BufferedInputStream(rwf.reader(), 13);
                for (int i = 0; i < size; i++) {
                    int b = r.read();
                    assert (b == (i & 255));
                }
                int eof = r.read();
                assert (eof == -1);
                r.close();
                System.out.println("Read done");
            } catch (IOException ex) {
                ex.printStackTrace();
            }
        }
    };

    t1.start();
    t2.start();
    t1.join();
    t2.join();
    rwf.close();
}