获得Out of Memory Exception的根本原因是什么？ 我们怎样才能克服这一点？

Question

Here is my sample snippet for reading and writing by output stream and I am getting out of memory exception.

 public static void readFileContent(InputStream in, OutputStream out) throws IOException {
    byte[] buf = new byte[500000];
    int nread;
    int navailable;
    int total = 0;
    synchronized (in) {
        try {
            while((nread = in.read(buf, 0, buf.length)) >= 0) {
                out.write(buf, 0, nread);
                total += nread;
            }
        }
        finally {
            if (in != null) {
                try {
                    in.close();
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }
    }

    out.flush();
    buf = null;
}

1. What are the possible scenarios with the above snippet to get "out of memory exception" ?
2. Is it necessary to close the output stream here? And does stream, flush is enough or do we need to close the stream always? If so why?
3. How could I avoid Out of memory exception in general?

Please clarify me.

Answer 1

1. What are the possible scenarios with the above snippet to get "out of memory exception" ?

There are various root causes for out of memory exceptions. Refer to oracle documentation page for more details.

java.lang.OutOfMemoryError: Java heap space :

Cause : The detail message Java heap space indicates object could not be allocated in the Java heap.

java.lang.OutOfMemoryError: GC Overhead limit exceeded :

Cause: The detail message "GC overhead limit exceeded" indicates that the garbage collector is running all the time and Java program is making very slow progress

java.lang.OutOfMemoryError: Requested array size exceeds VM limit :

Cause : The detail message "Requested array size exceeds VM limit" indicates that the application (or APIs used by that application) attempted to allocate an array that is larger than the heap size.

java.lang.OutOfMemoryError: Metaspace :

Cause: Java class metadata (the virtual machines internal presentation of Java class) is allocated in native memory (referred to here as metaspace)

java.lang.OutOfMemoryError: request size bytes for reason. Out of swap space? :

Cause: The detail message "request size bytes for reason. Out of swap space?" appears to be an OutOfMemoryError exception. However, the Java HotSpot VM code reports this apparent exception when an allocation from the native heap failed and the native heap might be close to exhaustion

2. Is it necessary to close the output stream here? And does stream, flush is enough or do we need to close the stream always? If so why?

since you are using raw InputStream and OutputStream in your method, we don' t know which type of actual Stream is getting passed to this method and hence explicitly close these Streams is good idea.

3. How could I avoid Out of memory exception in general?

This question is already answered in response to your first question.

Refer to this SE question on handling large files for IO operations :

Java OutOfMemoryError in reading a large text file

Answer 2

I think it's obvious that the problem is that you allocate 500000 bytes at once, and they may not be available in the heap at runtime.

Explanation: I would not suggest it, but you could increment the heap size of your program. The default heap size for a java program is determined at runtime , but it can also be parameterized.

Recommendation: As far as I can see by the provided snippet, it's not absolutely necessary to read 500000 bytes at once. So, you can initialize your byte array with a smaller number that would result in having more reading loops. But if it's not a problem for your program... I guess.

Conclusion: Try by setting the initial byte array size to 5000 , or even 1000 .

EDIT:

An extra point to take into consideration is that in the above code snippet you only flush once at the end. The bytes you are writting to the OutputStream are kept in memory, and their size may cause an OutOfMemoryException also.

In order to overcome this, you should flush more often. It will affect your performance if you flush too often, but you can always experiment with a condition in your loop eg

...
if (total % 5000 == 0) {
    out.flush();
}
...

EDIT 2:

As the InputStream and OutputStream objects are passed to the given method as parameters, so, in my opinion this method is not responsible for closing them. The method that initializes the Streams is also responsible for close them gracefully. Flush is enough for this method. But consider doing it in smaller chunks.

EDIT 3:

To summarize the suggested tweaks:

public static void readFileContent(InputStream in, OutputStream out) throws IOException {
    byte[] buf = new byte[1000];
    // wrap your OutputStream in a BufferedOutputStream
    BufferedOutputStream bos = new BufferedOutputStream(out, 5000);
    int nread;
    int navailable;
    int total = 0;
    synchronized (in) {
        try {
            while((nread = in.read(buf, 0, buf.length)) >= 0) {
                // use the BufferedOutputStream to write data
                // you don't need to flush regularly as it is handled automatically every time the buffer is full
                bos.write(buf, 0, nread);
                total += nread;
            }
        }
        finally {
            if (in != null) {
                try {
                    in.close();
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }
        }
    }

    // flush the last contents of the BufferedOutputStream
    bos.flush();
    buf = null;
}

Please note also that BufferedOutputStream will automatically call flush() when you will close it gracefully.

EDIT 4:

Example calling the above method:

public static void main(String[] args) {
    String filename = "test.txt";
    String newFilename = "newtest.txt";

    File file = new File(filename);
    File newFile = new File(newFilename);

    try (InputStream fis = new FileInputStream(file);
            OutputStream fout = new FileOutputStream(newFile)) {
        readFileContent(fis, fout);
    }
    catch(IOException ioe) {
        System.out.println(ioe.getMessage());
    }
}

Answer 3

1. Change the buf to new byte[1*1024]
2. Read using just buf no need to specify length eg pos = in.read(buf)

The rest of the code looks good. No need to increase the memory. Also, any points of synchronised inputStream?

Answer 4

In Java there isn't any brute way of freeing memory. Even calling the built-in Garbage Collector ( System.gC() ) might not solve the problem, as the GC only frees objects that are not referenced anymore. You need to take care of the code that you are writing so that it can employ the resources in its best way it can. Certainly, there are cases that you are left out of options, especially when you are using big or giant data structures regardless of any code optimization you can think of (in your case, you are creating an array with half a million records of bytes).

As a partial solution, you can increase your Heap Size Memory so that Java can allocate more memory.