Can't send large files over socket in Java

Question

I got working server and client applications, they work perfect while sending small files, but when I try to send for example movie file that is 700mb over socket it gives me

Exception in thread "AWT-EventQueue-0" java.lang.OutOfMemoryError: Java heap space

I searched the internet and found some tutorials on sending large files, but couldn't quite understand them, but I think my porblem is in writing file.

This is the code that server uses to write my file:

output = new FileOutputStream(directory + "/" + fileName);
            long size = clientData.readLong();
            byte[] buffer = new byte[1024];

            while (size > 0 && (bytesRead = clientData.read(buffer, 0, (int) Math.min(buffer.length, size))) != -1) {
                output.write(buffer, 0, bytesRead);
                size -= bytesRead;
            }
            output.close();

And here is the code that my client uses to send a file:

byte[] fileLength = new byte[(int) file.length()];  

        FileInputStream fis = new FileInputStream(file);  
        BufferedInputStream bis = new BufferedInputStream(fis);

        DataInputStream dis = new DataInputStream(bis);     
        dis.readFully(fileLength, 0, fileLength.length);  

        OutputStream os = socket.getOutputStream();  

        //Sending size of file.
        DataOutputStream dos = new DataOutputStream(os);   
        dos.writeLong(fileLength.length);
        dos.write(fileLength, 0, fileLength.length);     
        dos.flush();  

        socket.close();  

Answer 1

It gives you OutOfMemoryError because you are trying to read the entire file into memory before sending it. This is 100% completely and utterly unnecessary. Just read and write chunks, much as you are doing in the receiving code.

Answer 2

The problem is that you are trying to load the entire file at once into memory (via readFully ), which exceeds your heap space (which by default is 256mb I think).

You have two options:

1. Good option: load the file in chunks (eg 1024 bytes at a time) and send it like that. Takes a bit more effort to implement.
2. Bad option: increase your heap space via -Xmx option. Not recommended, I mostly mentioned this just in case you will need a bit more heap space for a program, but in this case it's a really bad idea.

For option one you can try something like:

DataInputStream in = new DataInputStream(new FileInputStream("file.txt"));
byte[] arr = new byte[1024];
try {
    int len = 0;
    while((len = in.read(arr)) != -1)
    {
        // send the first len bytes of arr over socket.
    } 
} catch(IOException ex) {
    ex.printStackTrace();
}

Answer 3

OutOfMemoryError is thrown when your program reaches the heap size limit. You loads entire file to your RAM. The default heap size is 1/4 of physical memory size or 1GB, so your program reaches the limit (you probably have got 2GB RAM, so the heap size is 512MB).

You should read your file in chunks (eg 10MB), so you won't reach the heap size limit, and you can resend chunks in the case of some error, instead of resending whole file. You can even read chunks in different thread, so when you're sending 1 chunk, you can load the second chunk, and when you'll sent the first one, you can start sending the second immediately.

Answer 4

The problem is that in your client you create a byte array for the entire file.

byte[] fileLength = new byte[(int) file.length()];  //potentially huge buffer allocated here

You should do the same thing you do on the server side and read the file chunk by chunk into a fixed size buffer.

Answer 5

Just like you're using a buffer on the server side the send the data, use a buffer in your applet/client to read it. If you're transferring a large file, you will clearly run out of memory when you use readFully() . This method is only useful for cases where you know your data will be really, really small.