Apache FTPClient fails to retrieve a file when it's almost downloaded

Question

Apache's FTPClient fails to download a file which is perfectly downloaded by FileZilla.

Basically, what I'm trying to do after a successfull login and listing is to download one specific file:

FTPClient client = new FTPClient();
client.setDataTimeout(20000);
client.setConnectTimeout(20000);
client.setBufferSize(65536);
//...
client.connect(host);
client.login(user, pswd);
// response validation
client.enterLocalPassiveMode();
// some listings with validations

InputStream in = new BufferedInputStream(client.retrieveFileStream(ftpFilePath), 16384);
// ...
byte[] buffer = new byte[8192];
while ((rd = in.read(buffer)) > 0) {
// .. reading the file and updating download progress

The last lines could be easily replaced with FTPClient 's file download with virtually the same result, but then we can't track the download progress:

client.setControlKeepAliveTimeout(30);
client.retrieveFile(ftpFilePath, new org.apache.commons.io.output.NullOutputStream());

As a result of all those actions I can see the file is downloading until something very close to 100% and then the exception occurs:

java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.net.SocketInputStream.read(Unknown Source)
        at java.io.FilterInputStream.read(Unknown Source)
        at java.io.BufferedInputStream.fill(Unknown Source)
        at java.io.BufferedInputStream.read1(Unknown Source)
        at java.io.BufferedInputStream.read(Unknown Source)
        at java.io.FilterInputStream.read(Unknown Source)
        at <my code from here on>

There seems to be no firewall, but when the internet connection speed is better the download succedes (probably, some sort of timeout is hit). I would think that the problem is with the connection, but the thing is FileZilla succedes to download the same file.

So, I can reformulate my question like this: how do I make my FTPClient behave like FileZilla when downloading a file. Probably, there's some complicated retry on ping logic that I'm not aware of.

Commons Net: commons-net-3.6

FTP Server: proftpd-1.3.3g-6.el5 on CentOS 5.8 with a default configuration, does not support FTP over TLS.

Answer 1

Seem like it's caused by the data timeout defined at the following line:

client.setDataTimeout(20000);

According to JavaDoc:

Sets the timeout in milliseconds to use when reading from the data connection. This timeout will be set immediately after opening the data connection, provided that the value is ≥ 0.

Note: the timeout will also be applied when calling accept() whilst establishing an active local data connection.

Parameters: timeout - The default timeout in milliseconds that is used when opening a data connection socket. The value 0 means an infinite timeout.

May you try to set this value to 0 (which means infinite in this context)?

Answer 2

I don't know what is the actual reason for this phenomena (time out on the last chunk of a file), but I've checked with Wireshark what it is that FileZilla does to download a file and found that it suffers from the very same problem with the same timeout and it is reconnecting to a server and sending a REST FTP query to restart the download of this specific file from when it was aborted that is to download only the last chunk.

So, the solution would be to add some sort of retry logic to the download process, so that this chunk of code:

InputStream in = new BufferedInputStream(client.retrieveFileStream(ftpFilePath), 16384);
// ...
byte[] buffer = new byte[8192];
while ((rd = in.read(buffer)) > 0) {

Becomes this:

InputStream in = new BufferedInputStream(client.retrieveFileStream(ftpFilePath), 16384);
// ...
byte[] buffer = new byte[8192];
long totalRead = 0;
for (int resumeTry = 0; resumeTry <= RESUME_TRIES; ++resumeTry) {
    try {
        while ((rd = in.read(buffer)) > 0) {
            //...
            totalRead += rd;
        }
        break;
    } catch (SocketTimeoutException ex) {
        // omitting exception handling
        in.close();
        client.abort();
        client.connect(...);
        client.login(...);
        client.setFileType(FTPClient.BINARY_FILE_TYPE);
        client.enterLocalPassiveMode();
        client.setRestartOffset(totalRead);
        in = client.retrieveFileStream(...);
        if (in == null) {
            // the FTP server doesn't support REST FTP query
            throw ex;
        }
        in = new BufferedInputStream(in, 16384);
    }
}