读取字节数组中的文件时出现Java堆空间错误

Question

I am getting java out of heap error while using following code. Can someone tell me what I am doing wrong here ?

On debugging I see taht value of length is 709582875

In main function 

File file = new File(fileLocation+fileName);
if(file.exists()){
s3Client.upload(bucketName,fileName,getBytesFromFile(file));
}


// Returns the contents of the file in a byte array.
public static byte[] getBytesFromFile(File file) throws IOException {
InputStream is = new FileInputStream(file);

// Get the size of the file
long length = file.length();

// You cannot create an array using a long type.
// It needs to be an int type.
// Before converting to an int type, check
// to ensure that file is not larger than Integer.MAX_VALUE.
if (length > Integer.MAX_VALUE) {
// File is too large
log.debug("file is too large"+length);
System.out.println("file is too large"+length);
}

if (length < Integer.MIN_VALUE || length > Integer.MAX_VALUE) {
throw new IOException
(length + " cannot be cast to int without changing its value.");
}

// return "test".getBytes();
// Create the byte array to hold the data

try{
byte[] bytes = new byte[(int)length];
}
catch(OutOfMemoryError e){ System.out.println(e.getStackTrace().toString());}

// Read in the bytes
int offset = 0;
int numRead = 0;
while (offset < bytes.length
&& (numRead=is.read(bytes, offset, bytes.length-offset)) >= 0) {
offset += numRead;
}

// Ensure all the bytes have been read in
if (offset < bytes.length) {
throw new IOException("Could not completely read file "+file.getName());
}

// Close the input stream and return bytes
is.close();
return bytes;
}

Answer 1

The problem is that the byte array you are allocating is too large and it use up the heap space.

You may try running your program with -Xms and -Xmx option to specify the min and max heap space the java virtual machine uses to run your program.

But I suggest you not to read the whole file into a byte array to process it. you can read part of it into a small byte array, process the portion, and continue to the next part. This way uses less heap space.

Answer 2

You are consuming 709582875 bytes (about 677MB) at the moment the byte array in the try block is allocated. This is quite large by conventional personal computing standards, and would consume most (if not all) of the memory of a JVM started with default settings.

Some information on default JVM memory settings can be found here

Answer 3

Try to increase heap size allocated by the Java Virtual Machine (JVM), something like:

java -Xms<initial heap size> -Xmx<maximum heap size>

For example:
java -Xms64m -Xmx256m HelloWorld

Answer 4

Donot create such a huge byte[] array . Your heap may go out of memory. It is bad idea to create byte[] array of file length for such a large file. create small byte array and read the file in chunk by chunk basis

Answer 5

需要一些jvm调整java -Xms256m -Xmx1024m

Answer 6

Is there a particular reason you nee to read the whole file at once as a byte[]? Can you use a memory mapped ByteBuffer instead as this uses very little heap regardless of the size of the file.