Out of memory java heap space

Question

I am trying to send chunks of files from server to more than one clients. When I am trying to send file of size 700mb, it showed "OutOfMemory java heap space" error. I am using Netbeans 7.1.2 version. I also tried VMoption in the properties. But still the same error happens. I think there is some problem with reading the entire file. Below code is working for up to 300mb. Please give me some suggestions.

Thanks in advance

public class SplitFile {   
    static int fileid = 0  ;

    public static DataUnit[] getUpdatableDataCode(File fileName) throws FileNotFoundException, IOException{

    int i = 0;
    DataUnit[] chunks = new DataUnit[UAProtocolServer.singletonServer.cloudhosts.length];

    FileInputStream fis;

    long Chunk_Size = (fileName.length())/chunks.length;
    int cursor = 0;

    long fileSize = (long) fileName.length();
    int nChunks = 0, read = 0;long readLength = Chunk_Size;
    byte[] byteChunk;

    try {
        fis = new FileInputStream(fileName);
        //StupidTest.size = (int)fileName.length();
        while (fileSize > 0) {
            System.out.println("loop"+ i);
            if (fileSize <= Chunk_Size) {
                readLength = (int) fileSize;
            }
            byteChunk = new byte[(int)readLength];
            read = fis.read(byteChunk, 0, (int)readLength);
            fileSize -= read;
           // cursor += read;
            assert(read==byteChunk.length);                                          
            long aid = fileid;
            aid = aid<<32 | nChunks;                
            chunks[i] = new DataUnit(byteChunk,aid);                

         //   Lister.add(chunks[i]);
            nChunks++;
            ++i;                       
        }
        fis.close();
        fis = null;

      }catch(Exception e){
            System.out.println("File splitting exception");
        e.printStackTrace();
    }

       return chunks;               
    }

Answer 1

Reading in the whole file would definitely trigger OutOfMemoryError as file size grow. Tuning the -Xmx1024M may be good for temporary fix, but it's definitely not the right/scalable solution. Also, doesn't matter how you move your variables around (like creating buffer outside of the loop instead of inside the loop) you will get OutOfMemoryError sooner or later. The only way to not get OutOfMemoryError for you is to not to read the complete file in memory.

If you have to use just memory, then an approach is to send off chunks to the client so you don't have to keep all the chunks in memory:

instead of:

chunks[i] = new DataUnit(byteChunk,aid);

do:

sendChunkToClient(new DataUnit(byteChunk, aid));

But the above solution has the drawback that if some error happened in-between chunk sending, you may have hard time trying to resume/recover from the error point.

Saving the chunks to temporary files like Ross Drew suggested is probably better and more reliable.

Answer 2

How about creating the

byteChunk = new byte[(int)readLength];

outside of the loop and just reuse it instead of creating an array of bytes over and over if it's always the same.

Alternatively

You could write incoming data to a temporary file as it comes in instead of maintaining that huge array then process it once it's all arrived.

Also

If you are using it multiple times as an int, you should probably just case readLength to an int outside the loop as well

int len = (int)readLength;

And Chunk_Size is a variable right? It should begin with a lower case letter.