How to download large sized Files (size > 50MB) in java

Question

I'm downloading files from a remote location, and the download is complete for smaller sized files and in-complete for large sized files (>10 MB). Here is my code that i have used for downloading files from remote server .

    File dstFile = null;
    // check the directory for existence.
    String dstFolder = LOCAL_FILE.substring(0,LOCAL_FILE.lastIndexOf(File.separator));
    if(!(dstFolder.endsWith(File.separator) || dstFolder.endsWith("/")))
        dstFolder += File.separator;

    // Creates the destination folder if doesn't not exists
    dstFile = new File(dstFolder);
    if (!dstFile.exists()) {
        dstFile.mkdirs();
    }
    try {
        URL url = new URL(URL_LOCATION);
        HttpURLConnection connection = (HttpURLConnection) url.openConnection(); 
        connection.addRequestProperty("User-Agent", "Mozilla/4.76"); 
        //URLConnection connection = url.openConnection();
        BufferedInputStream stream = new BufferedInputStream(connection.getInputStream());
        int available = stream.available();
        byte b[]= new byte[available];
        stream.read(b);
        File file = new File(LOCAL_FILE);
        OutputStream out  = new FileOutputStream(file);
        out.write(b);
    } catch (Exception e) {
        System.err.println(e);
        VeBLogger.getInstance().log( e.getMessage());
    }

Answer 1

Firstly, I'd suggest you use:

FileInputStream in = new FileInputStream(file);  

instead of:

BufferedInputStream in = new BufferedInputStream(new FileInputStream(file));

(To avoid building up memory usage)

try
{
    FileInputStream fileInputStream  = new FileInputStream(file);
    byte[] buf=new byte[8192];
    int bytesread = 0, bytesBuffered = 0;
    while( (bytesread = fileInputStream.read( buf )) > -1 ) {
        out.write( buf, 0, bytesread );
        bytesBuffered += bytesread;
        if (bytesBuffered > 1024 * 1024) { //flush after 1MB
            bytesBuffered = 0;
            out.flush();
        }
    }
}
finally {
    if (out != null) {
        out.flush();
    }
}

Answer 2

You can use apache commons IO library . It's easy. I have used it in many projects.

File dstFile = null;
// check the directory for existence.
String dstFolder = LOCAL_FILE.substring(0,LOCAL_FILE.lastIndexOf(File.separator));
if(!(dstFolder.endsWith(File.separator) || dstFolder.endsWith("/")))
    dstFolder += File.separator;

// Creates the destination folder if doesn't not exists
dstFile = new File(dstFolder);
if (!dstFile.exists()) {
    dstFile.mkdirs();
}
try {
    URL url = new URL(URL_LOCATION);
    FileUtils.copyURLToFile(url, dstFile);
} catch (Exception e) {
    System.err.println(e);
    VeBLogger.getInstance().log( e.getMessage());
}

Answer 3

Please read BufferedInputStream's method .available() in the API.

It returns the number of available bytes already downloaded (ie. the number of bytes you can read out of the stream without accessing/waiting for the network).

You should create a fixed size byte array, fx. 2048 bytes, and use the read() methods until it returns -1.