Java - Reading Writing Large Long Array Code

Question

I am trying to write a large Long Array of size 400000000 into a file and then read it back. What code I am using is following:

import java.io.*;
import java.nio.*;
import java.nio.channels.FileChannel;
import java.io.RandomAccessFile ;
import java.util.* ;

class Checks {
public static FileChannel channel;
public static MappedByteBuffer mbb;

public static void main(String[] args){
    try{


      long k[] = new long[400000000] ;
          for(int i = 0 ; i < 400000000 ; i++){
            k[i] = i ;
          }

    channel = new RandomAccessFile("abc.dat", "rw").getChannel();
        mbb = channel.map(FileChannel.MapMode.READ_WRITE, 0, 1 << 24);
        mbb.order(ByteOrder.nativeOrder());

          for(int i = 0 ; i < 400000000 ;i++ ){
             getMbb().putLong(k[i]);
        }

        channel.close();

       long ks[] = new long[400000000] ;

        channel = new RandomAccessFile("abc.dat", "rw").getChannel();
        mbb = channel.map(FileChannel.MapMode.READ_WRITE, 0, 1 << 24);
        mbb.order(ByteOrder.nativeOrder());

        for(int r = 0 ; r < 400000000; r++){
            ks[r] = getMbb().getLong();
         }

             for(int r = 0 ; r < 400000000; r++){
                 if(k[r] != ks[r]){
                  System.out.println("Error at " + r);
                  break ;
                  }
              }


}
    catch(Exception e)
    {
        e.printStackTrace();
    }

   }

    public static ByteBuffer getMbb() throws IOException {
        if (mbb.remaining() <= 0) {
            mbb = channel.map(FileChannel.MapMode.READ_WRITE, channel.size(), 1 << 24);
            mbb.order(ByteOrder.nativeOrder());
        }
        return mbb;
    }
}

However, this code is giving error that write and read array are not same. Can anybody help me why is this happening ?

Answer 1

Try forcing the changes made to the mapped buffer to disk, before closing the channel:

 mbb.force();

This way, in case you're using a file stored on a local device, you have the guarantee that all the changes made to the mapped buffer will be reflected on disk.

Considering that you only append to the file, an alternative way of accomplishing what you want is:

  channel = new RandomAccessFile("abc.dat", "rw").getChannel();

  ByteBuffer buffer = ByteBuffer.allocate(Long.SIZE/8 * 1001);
  for(int i = 0 ; i < 400000000 ;i++ ){
     //write a number of byte to a ByteBuffer
      buffer.putLong(k[i]);
      if(i % 1000 == 0){
          channel.write(buffer);
          buffer.clear();
      }
   }
   channel.write(buffer);
   buffer.clear();

For reading:

    buffer.clear();
    int bytesCnt;
    while((bytesCnt = channel.read(buffer))!=-1){
    for(int r = 0 ; r < bytesCnt; r++){
        ks[r] = buffer.getLong();
     }
    }

Regarding performance: In applications doing I/O, the performance penalties are due to the number of seeks. Consequently the lower number of seeks, the higher the performance. This is equivalent to writing as much data as possible (<=> smaller number of flushes to disk; a flush mean, in fact, a seek) in a sequential manner.

In your scenario the data is written only in a sequential way (appending to the file), so the only aspect to worry about is the number of flushes to the disk; this number is inversely proportional to the size of the buffer. So you should try to increase the size of the buffer as much as possible.

Answer 2

Here's what I'd suggest:

public static void main(String[] args) {

  File f = new File("abc.txt");

  DataOutputStream s = new DataOutputStream( new FileOutputStream(f));

  for ( int i = 0; i < 400000000; i++ ) {
    s.writeLong(i);
  }
  s.close();

  DataInputStream is = new DataInputStream( new FileInputStream(f));
  for (int i = 0; i < 400000000; i++ ) {
    if ( i != is.readLong() ) System.out.println("Error at " + i);
  }
}

This does all your program does, but without allocating memory explicitly and certainly without duplicating that allocation through a memory mapped buffer. If this solution actually applies to what you really want to do is hard to tell, as you didn't say what this is for.

Answer 3

I modified a bit your code to use DataInputStream and DataOutputStream instead of RandomAccessFile and it works like a charm.

    try {
        long k[] = new long[400000000];
        for (int i = 0; i < k.length; i++) {
            k[i] = i;
        }

        DataOutputStream dos = new DataOutputStream(new FileOutputStream("abc.dat"));
        for (int i = 0; i < k.length; i++) {
            dos.writeLong(k[i]);
        }

        long ks[] = new long[k.length];

        DataInputStream dis = new DataInputStream(new FileInputStream("abc.dat"));

        for (int r = 0; r < ks.length; r++) {
            ks[r] = dis.readLong();
        }

        for (int r = 0; r < k.length; r++) {
            if (k[r] != ks[r]) {
                System.out.println("Error at " + r);
                break;
            }
        }
    } catch(Exception e) {
    } finally {
        // Make sure to close the streams
        dos.close();
        dis.close();
    }

Answer 4

I think your getMbb() method is broken. Every time you remap the chunk of the file in memory, you map to channel.size() . That only works while you are creating the file, but not when you are reading it. When you are reading the file, you map the "region" of the file that come after the end of the file, and that has random contents.

You will have to fix the remapping code to keep track of where in the file you already are.