Multiple file downloads from SFTP using JSch failing with “Request queue: Unknown request issue”

Question

I am trying to download multiple files from FTP server. It works fine when i simply put it through a for loop . I don't know the file size which may vary , might be too small or too big. So, i am downloading them using executor service. When i put the download function through executor, i get Request queue: Unknown request issue . Please let me know where i'm going wrong.

System.out.println("Connecting to FTP Server");
Session session = null;
ChannelSftp channelSftp = null;

JSch jsch = new JSch();
session = jsch.getSession(sftpUser,sftpHost,22);        
session.setPassword(sftpPassword);
java.util.Properties config = new java.util.Properties();
config.put("StrictHostKeyChecking", "no");
session.setConfig(config);
session.connect();
System.out.println("Session Connected Status: "+session.isConnected());
Channel channel = session.openChannel("sftp");
channel.connect();
channelSftp = (ChannelSftp)channel; 
System.out.println("Channel Connected Status: "+channelSftp.isConnected());
System.out.println("File: "+sourceFilePath);

if(channelSftp.isConnected())
{
    String path = this.propertyManager.getValue("targetDirectoryPath");
    channelSftp.cd(path);
    Vector filelist = channelSftp.ls(path);

    for(int i=0; i<filelist.size();i++)
    {
        LsEntry entry = (LsEntry) filelist.get(i);
        String fileName     = sourceFilePath + entry.getFilename();
        destinationFilePath =  ShipmentUtils.getDownloadPath(entry.getFilename());

        System.err.println(entry.getFilename());
        exec.execute(new FileDownloader(destinationFilePath, channelSftp.get(fileName)));
    }
}
channelSftp.disconnect();
session.disconnect();

Here is the downloader class

private class FileDownloader implements Runnable
{
    String destinationFilePath;
    InputStream stream;

    public  FileDownloader(String destinationFilePath , InputStream stream) {
        this.destinationFilePath = destinationFilePath;
        this.stream = stream;
    }

    @Override
    public void run() {
        byte[] buffer = new byte[1024];
        BufferedOutputStream bos = null;
        BufferedInputStream   bis = null ;

        try
        {
            bis = new BufferedInputStream(stream);
            File newFile = new File(destinationFilePath);
            OutputStream os = new FileOutputStream(newFile);

            bos = new BufferedOutputStream(os);
            int readCount=0;
            while( (readCount = bis.read(buffer)) > 0) {
                bos.write(buffer, 0, readCount);
            }
        }
        catch (Exception e) {
            e.printStackTrace();
        }
        finally {
            try {
                if(bis != null) bis.close();
                if(bos != null) bos.close();    
            } catch (IOException e) {
                e.printStackTrace();
            }
        }

    }

}

This is the error

java.io.IOException: error: 4: RequestQueue: unknown request id 1399811183
    at com.jcraft.jsch.ChannelSftp$2.read(ChannelSftp.java:1406)
    at java.io.BufferedInputStream.fill(Unknown Source)
    at java.io.BufferedInputStream.read1(Unknown Source)
    at java.io.BufferedInputStream.read(Unknown Source)
    at java.io.FilterInputStream.read(Unknown Source)
    at com.shipmentprocessing.network.FTPConnector$FileDownloader.run(FTPConnector.java:141)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
java.io.IOException: error
    at com.jcraft.jsch.ChannelSftp$2.close(ChannelSftp.java:1505)
    at java.io.BufferedInputStream.close(Unknown Source)
    at com.shipmentprocessing.network.FTPConnector$FileDownloader.run(FTPConnector.java:150)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)

Answer 1

While you can use one SFTP connection/JSch session for multiple downloads, the JSch session is definitely not thread safe.

So if you need to run the downloads is separate threads, you have to open new connection for (in) each thread. Your download performance will be much better this way too.