File Download from a FTP Server using apache FTPClient

Question

I am using apache's FTPClient to download files from a FTP server. My scenario is - the FTP server can loose network connectivity and may remain disconnected for a max of 1 day. When it connects back, download of the files should start from where it was left. I am using the following code to connect to the server and then download the file from the server

public void retrieve(String server, int port, String username,
        String password, String remote, String local, int fileType,
        ProgressHandler progressHandler) throws Exception {     
    final FTPClient ftp = new FTPClient();
    Date date = new Date();
    long startTime_ms = date.getTime();
    if (progressHandler != null) {
        ftp.setCopyStreamListener(new FtpCopyStreamListener(progressHandler));
    }

    ftpConnect(server,ftp, port,startTime_ms);


    if(ftp.getReplyCode()==0 || !String.valueOf(ftp.getReplyCode()).startsWith("2")){
        cleanup(ftp, "Could not log into server: " + server, null);
        return;
    }

    boolean loggedIn = false;
    try {
        if (username == null || username.isEmpty()) {
            username = "anonymous";
            password = System.getProperty("user.name") + "@"
                    + InetAddress.getLocalHost().getHostName();
        }
        if (!ftp.login(username, password)) {
            ftp.logout();
            cleanup(ftp, "Could not log into server: " + server, null);
        }
        loggedIn = true;

        ftp.setFileType(fileType);
        ftp.enterLocalPassiveMode();
        OutputStream output = null;
        try {
            output = new FileOutputStream(local);
            LOGGER.info("About to download " + remote + " from " + server
                    + " on " + (port > 0 ? port : ftp.getDefaultPort())
                    + " to " + local);              
            ftp.setControlKeepAliveTimeout(300);
            boolean isFileDownloaded = false;
            try{
                isFileDownloaded = ftp.retrieveFile(remote, output);

            }
            catch(IOException ex){
                if(ftp.isConnected()){

                    try{
                        int retryCode = 0;
                        while (!String.valueOf(retryCode).startsWith("2") && (new Date().getTime() < (startTime_ms + TOTAL_DURATION_MSECS))){
                            try{

                                 retryCode = ftp.retr(local);
                            }catch(Exception e1){
                                Thread.sleep(1000);
                                System.out.println("File not downloaded !! " + e1.toString());
                                ftpConnect(server, ftp, port, startTime_ms);
                            }

                        }
                    }catch(Exception e){
                        //ftp.getReplyCode()
                        //throw e;
                        //disconnect(ftp);
                        //ftpConnect(server, ftp, port, startTime_ms);

                        System.out.println("File not downloaded !! " + e.toString());
                    }                   
                }
            }
            if(isFileDownloaded){
                LOGGER.info("Finished downloading " + remote + " from "
                        + server + " on "
                        + (port > 0 ? port : ftp.getDefaultPort()) + " to "
                        + local);
            }
            else{
                System.out.println("File not downloaded !! ");
            }           

        }catch(IOException io){
            io.printStackTrace();

            throw io;
        }           
        finally {
            if (output != null) {
                try {
                    output.close();
                } catch (IOException f) {
                    LOGGER.severe("output.close() error: "
                            + ServiceUtils.stackToString(f));
                }
            }
        }
    }       
    catch (FTPConnectionClosedException e) {
        LOGGER.severe("Server closed connection: " + server + " "
                + ServiceUtils.stackToString(e));
        throw e;
    } catch (IOException e) {
        LOGGER.severe("IOException, server: " + server + " "
                + ServiceUtils.stackToString(e));
        throw e;
    } finally {
        if (loggedIn) {
            try {
                ftp.logout();
            } catch (IOException f) {
                LOGGER.severe("ftp.logout() error: "
                        + ServiceUtils.stackToString(f));
            }
        }
        disconnect(ftp);
    }
}

My question is basically if the retrieveFile() method fails is there any anyway I can reconnect and start downloading from the point it was disconnected.

right now I am using a Microsoft IIS server as my FTP server but in the production environment it will be a FileZilla FTP server.

Any help is appreciated. Thanks

Answer 1

Sorry for very late response. Here what I did to solve the problem:

1. Setup a FileZilla server
2. Use retrieveFileStream instead of retrieveFile.

3. Keep a count of bytes written and use the same output stream to write.

     BufferedInputStream inputStream = null; BufferedOutputStream outputStream = null; InputStream in = null; int osCount = 0; in = ftp.retrieveFileStream(remote); inputStream = new BufferedInputStream(in); File localFile = new File(local, fileName); outputStream = new BufferedOutputStream(new FileOutputStream(localFile)); for (int read = inputStream.read(); read != -1; read = inputStream.read()) { outputStream.write(read); osCount++; } outputStream.flush(); if (osCount < fileSize) { System.out.println(fileName + ": Errored out !!!"); BufferedInputStream inputStream1 = null; InputStream in1 = null; if (ftp.isConnected()) { try { while (osCount < fileSize && new Date().getTime() < (startTime_ms + TOTAL_DURATION_MSECS)) { try { disconnect(ftp); ftpConnect(server, ftp, port, startTime_ms); ftp.login(username, password); ftp.setRestartOffset(osCount); System.out.println(fileName + ": Re reading from position " + osCount); in1 = ftp.retrieveFileStream(remote); inputStream1 = new BufferedInputStream(in1); for (int read = inputStream1.read(); read != -1; read = inputStream1.read()) { outputStream.write(read); osCount++; } outputStream.flush(); if(FTPReply.isPositiveCompletion(ftp.getReplyCode())){ isErrored = true; break; } } catch (Exception e1) { Thread.sleep(1000); LOGGER.severe(fileName + ": File not downloaded " + server + " " + ServiceUtils.stackToString(e1)); } } } catch (Exception e) { LOGGER.severe(fileName + ": File not downloaded " + server + " " + ServiceUtils.stackToString(e)); } } } 

Answer 2

FTPCLient.restart(...) is protected . I think setRestartOffset(...) and retrieveFile(...) is the way to go: "Note: if you have used setRestartOffset(long), the file data will start from the selected offset." .

I'd also catch CopyStreamException instead of just IOException : "The CopyStreamException allows you to determine the number of bytes transferred" '.

You check for ftp.isConnected() before entering the retry block but you write "FTP server loose network connectivity and may remain disconnected" . I guess that if you enter this catch block connectivity has been lost already, which is one of the reasons for entering this block at all.

What is ftp.retr(...) doing? Retrying, yes, but does it work as intended?

What do you get if you debug your code by:

• setting a breakpoint at ftp.isConnected()
• retrieving a file that is large enough to take a while to download
• pulling out your network cable while the transfer is in progress
• looking at the value of ftp.isConnected()
• if it is true (unlikely, but who knows) what does ftp.retr(...) return?