WSO2ESB file adapter - transport.vfs

Question

I use WSO2ESB 4.8.1. At the moment I try to create rather simple service which will read files from one folder and write them to another one. All folder are on the same physical partition.

I followed instructions at VFS+Transport and at Sample File Processing

this is my simple file processing proxy configuration:

<?xml version="1.0" encoding="UTF-8"?>
<proxy xmlns="http://ws.apache.org/ns/synapse" name="FileProcessorProxy" transports="vfs" startOnLoad="true" trace="disable">
        <target>
            <inSequence>
                <log level="full"/>
                <property xmlns:ns2="http://org.apache.synapse/xsd" name="transport.vfs.ReplyFileName" expression="fn:concat(fn:substring-after(get-property('MessageID'), 'urn:uuid:'), '.txt')" scope="transport" type="STRING"/>
                <property name="OUT_ONLY" value="true" scope="default" type="STRING"/>
                <send>
                    <endpoint>
                        <address uri="vfs:file:///opt/wso2/wso2data/esboverviewtest/out"/>
                    </endpoint>
                </send>
            </inSequence>
        </target>
        <parameter name="transport.vfs.ActionAfterProcess">MOVE</parameter>
        <parameter name="transport.PollInterval">5</parameter>
        <parameter name="transport.vfs.MoveAfterProcess">file:///opt/wso2/wso2data/esboverviewtest/original</parameter>
        <parameter name="transport.vfs.FileURI">file:///opt/wso2/wso2data/esboverviewtest/in</parameter>
        <parameter name="transport.vfs.MoveAfterFailure">file:///opt/wso2/wso2data/esboverviewtest/failure</parameter>
        <parameter name="transport.vfs.FileNamePattern">.*.txt</parameter>
        <parameter name="transport.vfs.ContentType">text/plain</parameter>
        <parameter name="transport.vfs.ActionAfterFailure">MOVE</parameter>
        <parameter name="transport.vfs.MoveTimestampFormat">yyyy-MM-dd'T'HH:mm:ss.SSSZ_</parameter>
        <parameter name="transport.vfs.FailedRecordsFileDestination">file:///opt/wso2/wso2data/esboverviewtest</parameter>
        <parameter name="transport.vfs.FailedRecordsFileName">transferFails.log</parameter>
        <parameter name="transport.vfs.MoveFailedRecordTimestampFormat">yyyy-MM-dd'T'HH:mm:ss.SSSZ_</parameter>
</proxy>

All in all, it works. When I put file into "in" directory it is moved to "out" and "original" directories.

The issue I began when I started to test what would happen if "out" or "original" directories are not available... ie there is an error while moving a new incoming file.

It appears that despite documentation on wso2 website such parameters as:

transport.vfs.MoveAfterFailure
transport.vfs.ActionAfterFailure
transport.vfs.FailedRecordsFileDestination
transport.vfs.FailedRecordsFileName
transport.vfs.MoveFailedRecordTimestampFormat

are just absolutely IGNORED... in case of any failure during a file transfer to "out" directory the file is never put in "failure" folder.

this the logs:

TID: [0] [ESB] [2015-09-03 21:43:41,120]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: , WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: urn:uuid:86C197EE849A6CA9071441298621105, Direction: request, Envelope: <?xml version="1.0" encoding="utf-8"?><soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"><soapenv:Body><text xmlns="http://ws.apache.org/commons/ns/payload">test message 1
</text></soapenv:Body></soapenv:Envelope> {org.apache.synapse.mediators.builtin.LogMediator}
TID: [0] [ESB] [2015-09-03 21:43:41,128]  INFO {org.apache.synapse.core.axis2.TimeoutHandler} -  This engine will expire all callbacks after : 120 seconds, irrespective of the timeout action, after the specified or optional timeout {org.apache.synapse.core.axis2.TimeoutHandler}
TID: [0] [ESB] [2015-09-03 21:43:41,134] ERROR {org.apache.synapse.transport.vfs.VFSUtils} -  Cannot get the lock for the file : file:///XXXopt/wso2/wso2data/esboverviewtest/out before processing {org.apache.synapse.transport.vfs.VFSUtils}
TID: [0] [ESB] [2015-09-03 21:43:41,135]  WARN {org.apache.synapse.transport.vfs.VFSTransportSender} -  Couldn't get the lock for the file : file:///XXXopt/wso2/wso2data/esboverviewtest/out, retry : 1 scheduled after : 30000 {org.apache.synapse.transport.vfs.VFSTransportSender}
TID: [0] [ESB] [2015-09-03 21:44:11,136] ERROR {org.apache.synapse.transport.vfs.VFSUtils} -  Cannot get the lock for the file : file:///XXXopt/wso2/wso2data/esboverviewtest/out before processing {org.apache.synapse.transport.vfs.VFSUtils}
TID: [0] [ESB] [2015-09-03 21:44:11,136]  WARN {org.apache.synapse.transport.vfs.VFSTransportSender} -  Couldn't get the lock for the file : file:///XXXopt/wso2/wso2data/esboverviewtest/out, retry : 2 scheduled after : 30000 {org.apache.synapse.transport.vfs.VFSTransportSender}
TID: [0] [ESB] [2015-09-03 21:44:41,137] ERROR {org.apache.synapse.transport.vfs.VFSUtils} -  Cannot get the lock for the file : file:///XXXopt/wso2/wso2data/esboverviewtest/out before processing {org.apache.synapse.transport.vfs.VFSUtils}
TID: [0] [ESB] [2015-09-03 21:44:41,137]  WARN {org.apache.synapse.transport.vfs.VFSTransportSender} -  Couldn't get the lock for the file : file:///XXXopt/wso2/wso2data/esboverviewtest/out, retry : 3 scheduled after : 30000 {org.apache.synapse.transport.vfs.VFSTransportSender}
TID: [0] [ESB] [2015-09-03 21:45:11,138] ERROR {org.apache.synapse.transport.vfs.VFSUtils} -  Cannot get the lock for the file : file:///XXXopt/wso2/wso2data/esboverviewtest/out before processing {org.apache.synapse.transport.vfs.VFSUtils}
TID: [0] [ESB] [2015-09-03 21:45:11,139] ERROR {org.apache.synapse.transport.vfs.VFSTransportSender} -  Couldn't send the message to file : file:///XXXopt/wso2/wso2data/esboverviewtest/out, unable to acquire the lock even after 4 retries {org.apache.synapse.transport.vfs.VFSTransportSender}
TID: [0] [ESB] [2015-09-03 21:45:11,140]  INFO {org.apache.axis2.engine.AxisEngine} -  [MessageContext: logID=9ffc75049fa389747d1fe3eac64c356ce001a7d103ba20fd] Couldn't send the message to file : file:///XXXopt/wso2/wso2data/esboverviewtest/out, unable to acquire the lock even after 4 retries {org.apache.axis2.engine.AxisEngine}

The file moved to "original" (what is ok) and but after 4 attempts the file was simply discarded, folder "failure" was left empty...

More of it, properties transport.vfs.FailedRecordsFileDestination and transport.vfs.FailedRecordsFileName were also ignored and no any file were generated, which should contain the list of problem transfers...

The question is that how to configure reliable file transfer by means of WSO2ESB so that it could be possible to track what files were successfully transfered and what files should be reprocessed.

Answer 1

In case of failure, your input file will be moved to transport.vfs.MoveAfterFailure. But here, "failure" stands for a case where the ESB can't read the file or build it's content.

For exemple, you configure transport.vfs.ContentType to be application/xml and your file's content is flat text : it will be moved to transport.vfs.MoveAfterFailure.

As soon as the content of your input file has been sent into your inSequence, whatever the issue of your mediation can be, the file will be moved to transport.vfs.MoveAfterProcess

transport.vfs.FailedRecordsFileName is the file name to maintain the list of failure files. Default is repository/conf/vfs-move-failed-records.properties

There is a second chance with transport.vfs.MoveAfterFailedMove : New destination to move the failed file.

If you want to choose inside your mediation (in the faultSequence for exemple) an other directory into which the input file must be moved, you have to extend org.apache.synapse.transport.vfs.VFSTransportListener, put your .jar into repository/components/lib and configure your specific class into axis2.xml