春季集成-从FTP读取文件并进行处理。 我们如何实现元存储并在不进行轮询的情况下对其进行处理

Question

In spring integration - Im trying to read file from FTP and process it. How can we achieve metastore and process it without polling. In the below configuration, in order to avoid reading the same file, if server restart happen ,I have introduced message-store in the ftpChannel. Now, the processor of the file is service activator, which needs polling. How can i avoid polling in service activator and read the file from ftpChannel queue immediately. If i use int:dispatcher then, I couldnot use message-store.

How can we resolve this?

<int:channel id="ftpChannel">
    <int:queue message-store="mongoDbMessageStore" />
    <!-- <int:dispatcher task-executor="taskExecutor"/> -->
</int:channel>

<bean id="mongoDbMessageStore"
    class="org.springframework.integration.mongodb.store.MongoDbMessageStore">
    <constructor-arg name="mongoDbFactory" ref="mongoDbFactory" />
    <constructor-arg name="collectionName" value="ftpInfo" />
</bean>

<int-ftp:inbound-channel-adapter id="ftpInbound"
    channel="ftpChannel" session-factory="ftpClientFactory" charset="UTF-8"
    auto-create-local-directory="true" delete-remote-files="false"
    filename-pattern="*.gz" remote-directory="/myfilerepo/#{istDate.getISTDate()}"
    remote-file-separator="/" local-filename-generator-expression="#this.toUpperCase()"
    temporary-file-suffix=".writing" preserve-timestamp="true"
    local-directory="/temp/spring/#{istDate.getISTDate()}">

     <int:poller cron="0-5 0/5 * * * ?" max-messages-per-poll="-1"/>
</int-ftp:inbound-channel-adapter>

<int:service-activator id="jobServiceActivator"
    input-channel="ftpChannel" ref="triggerJobLauncher" method="launch">
    <int:poller fixed-delay="10" />
</int:service-activator>

<!-- job context -->
<bean id="jobRepository"
    class="org.springframework.batch.core.repository.support.MapJobRepositoryFactoryBean">
    <property name="transactionManager" ref="transactionManager" />
</bean>

<bean id="transactionManager"
    class="org.springframework.batch.support.transaction.ResourcelessTransactionManager" />

<bean id="jobLauncher"
    class="org.springframework.batch.core.launch.support.SimpleJobLauncher">
    <property name="jobRepository" ref="jobRepository" />
</bean>
<!-- job context -->

Answer 1

You do not need a message-store on the channel; you need to use an FtpPersistentAcceptOnceFileListFilter in the filter and/or a FileSystemPersistentAcceptOnceFileListFilter in the local-filter to avoid reprocessing files after a system restart.

They need a MetadataStore ; if you want to use mongo you'll need to implement one; the framework currently doesn't have a mongo implementation.

EDIT:

As of version 4.2, the framework now has a mongo MetadataStore .

Answer 2

According to your concern to avoid reading the same file from the QueueChannel , I'd say that you worry that Spring Application always process the same message after application startup. But it isn't true. The message are removed from the queue (and, of course, from the MessageStore ) when it is polled from there. If you MessageStore is transactional resource (eg JDBC), the record for messages is marked to remove until TX commit or rollback. It isn't case for MongoDB though, but any message from the queue are polled only once, even if you have a cluster of such an application.

So, I don't understand why are you sure that your <service-activator> accepts the same file (as payload ) because you restart the app.

From other side, if you want to access to the MessageGroup backed by your ftpChannel , you can do this:

mongoDbMessageStore.getMessageGroup("mongoDbMessageStore:ftpChannel");

From other side you always can purge QueueChannel manually, injection ftpChannel to some service as QueueChannelOperations .