在java中将大文件加载到arraylist的最佳方法

Question

I have a file whose size is about 300mb. I want to read the contents line by line and then add it into ArrayList. So I have made an object of array list a1 , then reading the file using BufferedReader , after that when I add the lines from file into ArrayList it gives an error Exception in thread "main" java.lang.OutOfMemoryError: Java heap space.

Please tell me what should be the solution for this.

  public static void main(String[] args) {
    // TODO Auto-generated method stub
    try {
      FileReader file = new FileReader(
          "/home/dmdd/Desktop/AsiaData/RawData/AllupperairVcomponent.txt");
      ArrayList a1 = new ArrayList();
      BufferedReader br = new BufferedReader(file);
      String line = "";
      while ((line = br.readLine()) != null) {
        a1.add(line);
      }
    } catch (Exception e) {
      // TODO: handle exception
      e.printStackTrace();
    }
  }

Answer 1

Naively, increase the size of the heap via the Xmx command line argument (see this excellent answer for some guidance)

This'll only work up to a point though, instead consider structuring your data so that the memory requirements are minimized. Do you need the whole thing in memory at once? Perhaps you only need to test whether an item is in that set, consider using a hash or a bloom filter (etc).

Answer 2

Just increase the heap size of Java

java -Xmx250m

If you running your project from IDE set -Xmx250m in arguments.

250m is 250mb

Answer 3

If you have to have it in memory, you could try increasing the heap size by passing the -mx option to the java executable.

It may also be worth considering the question if you really need all that data in memory at the same time. It could be that you can either process it sequentially, or keep most or all of it on disk.

Answer 4

Pass -Xmx1024m to increase your heap sapce to 1024 mb.

java -Xms1024m -Xmx512m HelloWorld

You can increase up-to 4GB on a 32 bit system and on a 64 bit system you can go much higher.

Answer 5

Use java.nio.file.Files.readAllLines, it returns List<String>. And if you're getting OOME increase heap size as java -Xmx1024m

Answer 6

I agree with @Murali partly this will fix the problem you are facing. But it is advisable to use Caching when handling large files. What if the file size becomes 500Mb in a rare case. Make use of a Caching API like Memcached this will eliminate Memory Outages in JVM.

Answer 7

If you can: process the file in batches of 10000 lines or so.

read 10k lines process repeat until done