Apache Camel downloads some files incompletely from SFTP

Question

I've been struggling to get to the bottom of why it is that some files are not correctly downloaded. It seems like certain files just won't be downloaded fully, even when testing locally and restarting my application. To make matters more difficult it is not always consistent.

Info:

• Apache Camel version: 2.20.0
• Integrated into Spring-Boot application using the camel-spring-boot-starter
• Files are about 190M
• Files download ok using standalone Jsch and Linux sftp client
• Heap size set to 1G and memory usage doesn't even get close to the max
• Camel doesn't detect anything wrong with the download, even if number of bytes written is tens of megabytes less than the length of the file according to camel headers (camel headers have correct file length)
• I've observed the issue with org.apache.camel logging set to TRACE without seeing anything strange in the logs.
• Idemoptent repo is updated as if the file was processed correctly
• I see the same issue on Linux and Windows

Any advise on what the issue might be or suggestions for how to troubleshoot would be awesome!

Route config (a bit artificially created since values come from spring-boot config):

public class FileRouteBuilder extends RouteBuilder {
 // Cut

    @Override
    public void configure() throws Exception {

        errorHandler(deadLetterChannel("seda:"+ROUTE_ID_ERROR_EMAIL));      

        from("sftp://username@hostname/OUT?noop=true&streamDownload=true&password=password&include=Data_file.*csv&idempotentRepository=#keyRepo&greedy=true&delay=5m&maxMessagesPerPoll=10&readLock=changed")
        .id(routeConfig.getRouteId())
        .routeDescription(routeConfig.getRouteId())
        .setHeader(HEADER_FILE_SOURCE, constant(routeConfig.getRouteId()))
        .to("log:feeds." + routeConfig.getRouteId() + "?level=INFO&showAll=true")
        // Exclude all files oder than the specified number of hours
        .filter(new FileModifiedSincePredicate(24))
        .to(file:rootDir/DATA)
        .to("seda:" + ROUTE_ID_ACTIVITY_EMAIL_NOTIFICATION)
        .end();
       }
    }
}

Update1

Observations after adding binary=true .

First two files are downloaded correctly but the 3rd and final file on the server is not.

193255587 Data_File_12.csv
191072548 Data_File_15.csv
139929360 Data_File_16.csv

The correct file size of teh Data_FIle_16.csv file is 192867682 bytes, which is captured correctly in the the CamelFileLength header.

Update 2

Removed all the log and seda email components above, and re-ran. The third file still doesn't get completely written.

Adding the relevant DEBUG level log output in the hope that it sheds some light on what is going on or perhaps rules out certain things.

From what I can tell the log doesn't show anything suspicious and there is not hint that the _16 file is incompletely written.

Is there anything which could be happening on the SFTP server that anyone is aware of that it is worth checking with the provider?

o.a.c.c.file.remote.SftpConsumer         : Took 0.194 seconds to poll: OUT
o.a.c.c.file.remote.SftpConsumer         : Total 3 files to consume
o.a.c.c.file.remote.SftpConsumer         : About to process file: RemoteFile[Data_File_12.csv] using exchange: Exchange[]
o.apache.camel.processor.SendProcessor   : >>>> file://target/file-dest/MISA Exchange[ID-LON-2016-1516204084378-0-1]
o.a.camel.component.file.FileOperations  : Using InputStream to write file: target\file-dest\MISA\Data_File_12.csv
o.a.camel.converter.jaxp.XmlConverter    : Created TransformerFactory: com.sun.org.apache.xalan.internal.xsltc.trax.TransformerFactoryImpl@d9dfe93
o.a.c.c.file.GenericFileProducer         : Wrote [target\file-dest\MISA\Data_File_12.csv] to [file://target/file-dest/MISA]
o.a.c.c.file.GenericFileOnCompletion     : Done processing file: RemoteFile[Data_File_12.csv] using exchange: Exchange[ID-LON-2016-1516204084378-0-1]
o.a.c.p.i.FileIdempotentRepository       : Appending Data_File_12.csv-193255587 to idempotent filestore: target\file-dest\.file-key-repo\repo
o.a.c.c.file.remote.SftpConsumer         : About to process file: RemoteFile[Data_File_15.csv] using exchange: Exchange[]
o.apache.camel.processor.SendProcessor   : >>>> file://target/file-dest/MISA Exchange[ID-LON-2016-1516204084378-0-2]
o.a.camel.component.file.FileOperations  : Using InputStream to write file: target\file-dest\MISA\Data_File_15.csv
o.a.c.c.file.GenericFileProducer         : Wrote [target\file-dest\MISA\Data_File_15.csv] to [file://target/file-dest/MISA]
o.a.c.c.file.GenericFileOnCompletion     : Done processing file: RemoteFile[Data_File_15.csv] using exchange: Exchange[ID-LON-2016-1516204084378-0-2]
o.a.c.p.i.FileIdempotentRepository       : Appending Data_File_15.csv-191072548 to idempotent filestore: target\file-dest\.file-key-repo\repo
o.a.c.c.file.remote.SftpConsumer         : About to process file: RemoteFile[Data_File_16.csv] using exchange: Exchange[]
o.apache.camel.processor.SendProcessor   : >>>> file://target/file-dest/MISA Exchange[ID-LON-2016-1516204084378-0-3]
o.a.camel.component.file.FileOperations  : Using InputStream to write file: target\file-dest\MISA\Data_File_16.csv
o.a.c.c.file.GenericFileProducer         : Wrote [target\file-dest\MISA\Data_File_16.csv] to [file://target/file-dest/MISA]
o.a.c.c.file.GenericFileOnCompletion     : Done processing file: RemoteFile[Data_File_16.csv] using exchange: Exchange[ID-LON-2016-1516204084378-0-3]
o.a.c.p.i.FileIdempotentRepository       : Appending Data_File_16.csv-192867682 to idempotent filestore: target\file-dest\.file-key-repo\repo

Answer 1

Ah you log the message after you download it, and you use streamDownload=true .
See this FAQ-why-is-my-message-body-empty and how you need to use stream caching if doing so.

Because the message is streaming based, then either do NOT log the message body (you can log headers etc) and then route it to the file endpoint so its saved directly as a file.