Write and read multiple byte[] in file with Java

Question

I have to write byte arrays in a file. I can't do it in one time so I can't put my arrays in a container. Also the size of my arrays is variable. Secondly, the file is very huge, so I have to split it, in order to read it array by array.

How can I do that ? I tried to write line by line my byte arrays but I haven't been able. How can I put a separator between my arrays and after split it over this separator ?

EDIT :

I tried this :

ByteArrayOutputStream bos = new ByteArrayOutputStream();
ObjectOutput out = new ObjectOutputStream(bos);
out.writeObject(byteArray);

But, I execute this code several times, so the ObjectOutputStream adds each time a new header which corrupt the file.

I also try :

out.write(byteArray);

but I couldn't separate my arrays. So I tried to append a '\\n', which didn't work. After I was looking for library like FileUtils in order to write byte[] line by line but I didn't find.

Answer 1

You can use existing collections Like eg List to maintain List of byte[] and transfer it

    List<byte[]> list = new ArrayList<byte[]>();
    list.add("HI".getBytes());
    list.add("BYE".getBytes());

    ObjectOutputStream out = new ObjectOutputStream(new FileOutputStream(
            "test.txt"));
    out.writeObject(list);

    ObjectInputStream in = new ObjectInputStream(new FileInputStream(
            "test.txt"));
    List<byte[]> byteList = (List<byte[]>) in.readObject();

    //if you want to add to list you will need to add to byteList and write it again
    for (byte[] bytes : byteList) {
        System.out.println(new String(bytes));
    }

Output:

   HI
   BYE

Another option is use RandomAccessFile . Which will not force you to read complete file and you can skip the data that you don't want to read.

     DataOutputStream dataOutStream = new DataOutputStream(
            new FileOutputStream("test1"));
    int numberOfChunks = 2;
    dataOutStream.writeInt(numberOfChunks);// Write number of chunks first
    byte[] firstChunk = "HI".getBytes();
    dataOutStream.writeInt(firstChunk.length);//Write length of array a small custom protocol
    dataOutStream.write(firstChunk);//Write byte array

    byte[] secondChunk = "BYE".getBytes();
    dataOutStream.writeInt(secondChunk.length);//Write length of array
    dataOutStream.write(secondChunk);//Write byte array

    RandomAccessFile randomAccessFile = new RandomAccessFile("test1", "r");
    int chunksRead = randomAccessFile.readInt();
    for (int i = 0; i < chunksRead; i++) {
        int size = randomAccessFile.readInt();
        if (i == 1)// means we only want to read last chunk
        {
            byte[] bytes = new byte[size];
            randomAccessFile.read(bytes, 0, bytes.length);
            System.out.println(new String(bytes));
        }
        randomAccessFile.seek(4+(i+1)*size+4*(i+1));//From start so 4 int + i* size+ 4* i ie. size of i
    }

Output:

BYE

Answer 2

You have to described your data in your encoding. ie add some metadata.

For example, the length of the array, then the data of the array.

This is called serialization.

Array of int: length(4 bytes), data[0] (4 bytes), data[1] (4 bytes), data[2] (4 bytes)

Answer 3

There are several ways of doing this. Basically first thing to point out is that a file is just an unstructured sequence of bytes. Any bytes. Basically you want o store several byte arrays of variable length. That means you have to add structure in some way, which you then parse when you read.

The easiest is probably to use some sequence of bytes that you treat as a separator (specific enough that it has a low probability of showing up in your data otherwise).

Then on writing, first simply write your bytes, then the separator

out.write(myarray);
out.write(separator);
out.write(anotherarray);

On reading, you need to use some kind of sliding window that you can inspect so that you can figure out if you have read the separator. Basically just iterate over the bytes one by one and keep the last few you've read in some buffer. When you see the separator in your buffer, you have just found the end of your array.

An alternative to this could be writing blocks of a fixed length with some header that describes the length and maybe a bit for denoting whether there is another packet for the current array. Then you simply read and write whole blocks.