How to read the large text files efficiently in java

Question

Here, I am reading the 18 MB file and store it in a two dimensional array. But this program takes almost 15 minutes to run. Is there anyway to optimize the running time of the program. The file contains only binary values. Thanks in advance…

public class test 
{
    public static void main(String[] args) throws FileNotFoundException, IOException 
    {
        BufferedReader br;

        FileReader fr=null;
        int m = 2160;
        int n = 4320;
        int[][] lof = new int[n][m];
        String filename = "D:/New Folder/ETOPOCHAR";
       try {
         Scanner input = new Scanner(new File("D:/New Folder/ETOPOCHAR"));
        double range_km=1.0;
        double alonn=-57.07; //180 to 180
        double alat=38.53;

        while (input.hasNextLine()) {
            for (int i = 0; i < m; i++) {
                for (int j = 0; j < n; j++) {
                   try
                   {
                      lof[j][i] = input.nextInt();
                      System.out.println("value[" + j + "][" + i + "] = "+ lof[j][i]);
                    }
                   catch (java.util.NoSuchElementException e) {
                      //  e.printStackTrace();
                    }
                }
            }         //print the input matrix
        }

I have also tried with byte array but i can not save it in twoD array...

public class FileToArrayOfBytes
{
    public static void main( String[] args )
    {
        FileInputStream fileInputStream=null;

        File file = new File("name of file");

        byte[] bFile = new byte[(int) file.length()];

        try {
            //convert file into array of bytes
        fileInputStream = new FileInputStream(file);
        fileInputStream.read(bFile);
        fileInputStream.close();

        for (int i = 0; i < bFile.length; i++) {
            System.out.print((char)bFile[i]);
            }

        System.out.println("Done");
        }catch(Exception e){
            e.printStackTrace();
        }
    }
}

Answer 1

You can read the file into a byte array first, then deserialize these bytes. Start with 2048 bytes buffer (as input buffer), then experiment by increasing/decreasing its size, but the experimental buffer size values should be a power of two (512, 1024, 2048, etc).

As far as I rememenber, there are good chances that the best performance can be achived with a buffer of size 2048 bytes, but it is OS dependent and should be verified.

Code sample (here you can try different values of BUFFER_SIZE variable, in my case I've read a test file of size 7.5M in less then one second):

public static void main(String... args) throws IOException {
    File f = new File(args[0]);
    byte[] buffer = new byte[BUFFER_SIZE];
    ByteBuffer result = ByteBuffer.allocateDirect((int) f.length());
    try (FileInputStream fos = new FileInputStream(f)) {
      int bytesRead;
      int totalBytesRead = 0;
      while ((bytesRead = fos.read(buffer, 0, BUFFER_SIZE)) != -1) {
        result.put(buffer, 0, bytesRead);
        totalBytesRead += bytesRead;
      }
      // debug info
      System.out.printf("Read %d bytes\n", totalBytesRead);

      // Here you can do whatever you want with the result, including creation of a 2D array...
      int pos = result.position();
      result.rewind();
      for (int i = 0; i < pos / 4; i++) {
        System.out.println(result.getInt());
      }
    }
  }

Take your time and read docs for java.io, java.nio packages as well as Scanner class, just to improve understanding.